add deletion vector APIs to transaction #1430

emkornfield · 2025-10-28T05:32:21Z

🥞 Stacked PR

Use this link to review incremental changes.

stack/remove_files [Files changed]
- stack/dv_transaction [Files changed]

This PR adds the update_deletion_vectors API to the Transaction type, enabling engines to update deletion vectors for existing files without rewriting data. This completes the write-side support for Delta Lake's deletion vector feature, allowing efficient row-level deletes and updates.

codecov · 2025-11-05T07:43:04Z

Codecov Report

❌ Patch coverage is 90.96774% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.13%. Comparing base (cfb3320) to head (33e1f56).
⚠️ Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
kernel/src/transaction/mod.rs	90.80%	7 Missing and 25 partials ⚠️
test-utils/src/lib.rs	89.23%	1 Missing and 6 partials ⚠️
...src/engine/arrow_expression/evaluate_expression.rs	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1430      +/-   ##
==========================================
+ Coverage   84.99%   85.13%   +0.14%     
==========================================
  Files         120      120              
  Lines       31662    32259     +597     
  Branches    31662    32259     +597     
==========================================
+ Hits        26911    27465     +554     
- Misses       3441     3450       +9     
- Partials     1310     1344      +34

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

kernel/src/transaction/mod.rs

emkornfield · 2025-11-05T19:03:17Z

kernel/src/transaction/mod.rs

-                        .with_dropped_field("modificationTime"),
-                );
-                let expr = Expression::struct_from([transform]);
+        Ok(remove_files_metadata.map(


Note to reviewer, this is a formatting change because of not reference self.remove_file_metadata)

emkornfield · 2025-11-05T19:04:36Z

kernel/src/transaction/mod.rs

+                    )
+                    .with_dropped_field(FILE_CONSTANT_VALUES_NAME)
+                    .with_dropped_field("modificationTime");
+                for column_to_drop in columns_to_drop {


Note to reviewer this is an actual change.

kernel/src/transaction/mod.rs

kernel/tests/dv.rs

emkornfield · 2025-11-05T19:22:40Z

kernel/tests/dv.rs

+
+    // Step 4: Create deletion vectors marking rows 2, 5, and 7 as deleted
+    let mut dv = KernelDeletionVector::new();
+    dv.add_deleted_row_indexes([2, 5, 7]);


updates for two files:

Apply this to only the first file (with updated indices).

Verify deleted rows

Delete one more row from the same file, and delete some rows from the second file.

verify deleted rows again.

emkornfield · 2025-11-05T19:23:09Z

kernel/tests/dv.rs

+        .transaction(Box::new(FileSystemCommitter::new()))?
+        .with_dv_update();
+    let write_context = temp_txn.get_write_context();
+    let dv_path = write_context.new_deletion_vector_path(String::from(""));


add a parameter comment that the prefix is empty.

kernel/tests/dv.rs

This was referenced Oct 28, 2025

feat!: add remove_files API #1353

Open

feat!: Add optional stats field to remove action #1390

Merged

github-actions bot assigned emkornfield Oct 28, 2025

github-actions bot added the breaking-change Change that require a major version bump label Oct 28, 2025

emkornfield changed the title ~~add deletion vector APIs to transaction~~ WIP NOT READY FOR REVIEW add deletion vector APIs to transaction Oct 28, 2025

emkornfield force-pushed the stack/dv_transaction branch 4 times, most recently from 6a65734 to 688fabc Compare November 4, 2025 01:28

emkornfield force-pushed the stack/dv_transaction branch from 688fabc to 5ab2c58 Compare November 5, 2025 07:41