[Feature]: update to DataFusion 51.0.0

[lonless9]

Is your feature request related to a problem?

• Release DataFusion 51.0.0 (Nov 2025) apache/datafusion#17558
• feat!: arrow 57, MSRV 1.85+ delta-kernel-rs#1424

Describe the solution you'd like

1. DataFusion 51
2. arrow 57
3. delta_kernel 0.17.0
4. pyo3-arrow 0.14.0

Describe alternatives you've considered

No response

Priority

None

Additional context

No response

Contribution

• I'm willing to submit a pull request for this feature
• I can help with testing this feature
• I can help with documentation for this feature

[lonless9]

take

[alamb]

Sorry @lonless9 ! I actually just saw this ticket.

I actually started working on this item here: #3921

If you would like to take it over that would be fine with me

[lonless9]

Thanks @alamb ! I was simply resolving the blocker and your help quiet counts!
I won't take over this PR #3921 since it looks pretty good!
Ideally you can directly merge or pick this PR to see if there is still something left undone.

• chore: datafusion 51 lakehq/delta-rs#2

We could cooperate to reduce the reduntant work next time.

[lonless9]

@rtyler Please assign this issue to @alamb

[alamb]

Hi @lonless9 -- I would be more than happy for you to take over the upgrade. I have been doing it in the past as a way to do pre-release testing of DataFusion releases but it would be amazing if you could take over

I just closed #3921 (as you say it is basically the same as lakehq#2)

[alamb]

We could cooperate to reduce the reduntant work next time.

Yes I very much apologize -- I didn't realize you were also doing this work.

[hntd187]

I've already done a lot of the work for this, I'll take it over @alamb, I'm also exploring variant enablement for delta-rs so this all lines up there.

[hntd187]

I think looking at this more too, @lonless9 if we get a released version of DF 51, just merge yours and I can update mine, I've already done the changes you have, but I wanted to push them with variant.

[lonless9]

Thank you for your efforts! @alamb @hntd187

I suggest we divide this task into two parts:

The first part involves upgrading delta-kernel-rs 0.17.0 and dependencies related to arrow 57. This part should be fine to merge after

• release 0.17.0 delta-kernel-rs#1459.

The second part is about upgrading DataFusion

• (Release DataFusion 51.0.0 (Nov 2025) apache/datafusion#17558 (comment)).

This part can be pinned to DataFusion branch-51 which is expected to be cut to tomorrow, but it can only be merged after DataFusion 51 is released.

Since most work has got done, @hntd187 feel free to take over and directly open the related PRs!