CREATE OR REPLACE in Iceberg Rest Catalog Doesn't Preserve Snapshot History

[ofirNakdai]

Hi Trino team,

We're experiencing an issue with the Iceberg Rest Catalog where using the CREATE OR REPLACE TABLE statement seems to replace the entire snapshot history instead of preserving it. Here's a detailed summary of our setup and issue:

Setup:

• Trino Version: 440 (also tested on 464)
• Catalog Type: Iceberg Rest Catalog
• Configuration: Docker setup with MinIO for storage, catalog properties set with iceberg.catalog.type=rest and pointing to the Iceberg Rest Catalog via http://rest:8183

Steps to Reproduce:

1. Create Table: We create a table using CREATE OR REPLACE TABLE rest_catalog.test_schema.test1 AS (SELECT * FROM tpch.tiny.orders).
2. Make Update: We update this table (e.g., changing the totalprice column), creating a new snapshot in the history.
3. Re-run CREATE OR REPLACE: We re-run the CREATE OR REPLACE TABLE statement with updated data, expecting this to keep the history intact.
4. Time Travel Query Fails: Attempting a SELECT * FROM rest_catalog.test_schema.test1 FOR TIMESTAMP AS OF returns an error:
   SQL Error [76]: No version history table "test_schema".test1 at or before

Looking at the metadata table, it appears the previous snapshots are not linked to the new snapshot, suggesting CREATE OR REPLACE might be invalidating the history. However, according to the Iceberg documentation, CREATE OR REPLACE should create a new snapshot while retaining the old ones.

Any insights or suggestions on why this might be occurring would be greatly appreciated.

Thank you!

[mayankvadariya]

I was also able to reproduce issue with IcebergPolarisQueryRunnerMain but same queries work with IcebergMinioHiveMetastoreQueryRunnerMain

CREATE OR REPLACE TABLE test1 AS SELECT 'value' col;
UPDATE test1 SET col = 'new value';
SELECT * FROM "test1$snapshots";
CREATE OR REPLACE TABLE test1 AS SELECT 'value' col;
SELECT * FROM test1 FOR TIMESTAMP AS OF TIMESTAMP '...'; -- fails as stated above
SELECT * FROM test1 FOR VERSION AS OF 2786598815656921987; -- works

I believe it has to do with how table is loaded(TrinoCatalog#loadTable) in rest catalog vs other catalog implementations.
Rest catalog implementation relies on Iceberg rest catalog vs other catalog implementation reads metadata file and prepares table from it.

Table table = catalog.loadTable(...);
List<HistoryEntry> history = table.history(); // list has only one entry of latest snapshot for time travel with timestamp query in Rest vs HMS has three entries.

table.history() is used in time travel query with timestamp flow.

trino/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

Line 932 in d89e230

return table.history().stream()

Time travel with snapshot id works as it doesn't use table.history()

trino/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

Lines 557 to 558 in d89e230

	case TEMPORAL -> getTemporalSnapshotIdFromVersion(session, table, version, versionType);
	case TARGET_ID -> getTargetSnapshotIdFromVersion(table, version, versionType);

[ofirNakdai]

Do you have any suggestions to how we can fix the Trino iceberg rest catalog? so we can make time traveling work with the rest catalog when using create or replace