Skip to content

Conversation

@ImDoubD-datazip
Copy link
Collaborator

@ImDoubD-datazip ImDoubD-datazip commented Sep 5, 2025

Description

Clear destination feature in Iceberg (clears state of selected streams for clearance and drops those streams from destination side) and difference find between stream catalogs.

Fixes # (issue)

Type of change

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Sync was done when sync mode was full_refresh. It cleared the already present table and only synced the records that were introduced in the full_refresh. Tested on JDBC and GLUE catalog

build.sh commands for local test:

  • generates difference_streams.json:
./build.sh driver-mongodb discover --streams /path/old_streams.json --difference /path/new_streams.json
  • does clear destination:
./build.sh driver-mongodb clear-destination --streams /path/difference_streams.json --destination /path/examples/destination.json --state /path/examples/state.json

Documentation

  • Documentation Link: [link to README, olake.io/docs, or olake-docs]
  • N/A (bug fix, refactor, or test changes only)

@hash-data hash-data linked an issue Sep 19, 2025 that may be closed by this pull request
hash-data
hash-data previously approved these changes Oct 22, 2025
@hash-data hash-data merged commit 7175a50 into staging Oct 24, 2025
10 of 11 checks passed
@hash-data hash-data deleted the chore/clear-dest-ice branch October 24, 2025 11:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Clear Destination And Sync

4 participants