feat: Iceberg v3 deletion vectors #2561
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces FormatVersionManager to detect and manage Iceberg table format versions. This follows the pattern used by Spark/Flink/Kafka Connect integrations for gating V3-specific features like deletion vectors. Key changes: - Add FormatVersionManager with detectFormatVersion() and supportsDeletionVectors() - Use TableMetadata.formatVersion() for accurate detection (format-version is a reserved property not stored in table.properties()) - Add comprehensive unit tests for format version detection
Introduces V3DeltaTaskWriter that uses Iceberg's built-in DVFileWriter for position deletes when the table format version supports deletion vectors (V3+). Key changes: - V3DeltaTaskWriter extends BaseTaskWriter with DV support detection - Uses BaseDVFileWriter for position deletes in V3 tables - Falls back to traditional position delete files for V2 tables - Add comprehensive unit tests for V3 writer functionality This enables efficient row-level deletes using Puffin-based deletion vectors for Iceberg V3 tables, which provide better performance than traditional position delete files through compact bitmap representation.
Shekharrajak
force-pushed
the
feature/iceberg-v3-deletion-vectors
branch
from
b64fac6 to
34f0091
Compare
- V3DVWriter tracks positions and uses DVFileWriter for position deletes - V2EqualityDeltaWriter uses base class position delete handling - Add TDD tests verifying Puffin format and referencedDataFile
Shekharrajak
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: #2536
This PR adds foundational support for Iceberg V3 deletion vectors in Fluss. Deletion vectors (DVs) are a V3 feature that provides more efficient row-level deletes by storing deleted row
positions in a compact bitmap format (Puffin files) rather than separate position delete files.
Brief change log
Add FormatVersionManager: A utility class for detecting Iceberg table format versions and gating V3-specific features.
Add V3DeltaTaskWriter: A V3-aware delta writer that uses Iceberg's built-in
BaseDVFileWriterfor position deletes when the table format version supports deletion vectors.BaseTaskWriterwith DV support detectionAPI and Format
No breaking API changes. This adds new internal classes:
FormatVersionManager- New utility class for format version detectionV3DeltaTaskWriter- New writer classWhen V3 tables are used, delete files will use Puffin format (
.puffin) instead of traditional position delete files, which is the standard Iceberg V3 format.Tests
Unit tests