REST catalog doesn't return old history if we execute CREATE OR REPLACE TABLE statement

[ebyhr]

Apache Iceberg version

1.7.1 (latest release)

Query engine

Trino

Please describe the bug 🐞

trinodb/trino#24312 is trying to use org.apache.iceberg.MetadataTableUtils#createMetadataTableInstance method for history metadata table. Only REST catalog doesn't return the past history if we execute CREATE OR REPLACE TABLE statement.

trino:tpch> TABLE "region$history";
          made_current_at           |     snapshot_id     | parent_id | is_current_ancestor
------------------------------------+---------------------+-----------+---------------------
 2024-12-13 16:26:02.387 Asia/Tokyo | 4753270524621689028 |      NULL | true
(1 row)

trino:tpch> CREATE OR REPLACE TABLE region AS SELECT * FROM orders;
CREATE TABLE: 15000 rows

trino:tpch> TABLE "region$history";
          made_current_at           |     snapshot_id     | parent_id | is_current_ancestor
------------------------------------+---------------------+-----------+---------------------
 2024-12-13 16:27:51.756 Asia/Tokyo | 7793069171159305791 |      NULL | true
(1 row)

trino:tpch> SHOW CREATE TABLE region;
                                                                          Create Table
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
 CREATE TABLE iceberg.tpch.region (
    orderkey bigint,
    custkey bigint,
    orderstatus varchar,
    totalprice double,
    orderdate date,
    orderpriority varchar,
    clerk varchar,
    shippriority integer,
    comment varchar
 )
 WITH (
    format = 'PARQUET',
    format_version = 2,
    location = '/var/folders/9s/_zwn4r_n2_9bp0krllp1pl3c0000gp/T/a6a26fac-b6be-4c9c-b05b-cbe7ec894b2f/iceberg_data/tpch/region-4f41a42ee7f84d3093d85649bc557e46'
 )
(1 row)

region-4f41a42ee7f84d3093d85649bc557e46.zip

The following request contains RemoveSnapshotRef that clears existing snapshot logs.

iceberg/core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java

Lines 434 to 437 in 540d6a6

	TableMetadata.Builder metadataBuilder = TableMetadata.buildFrom(base);
	request.updates().forEach(update -> update.applyTo(metadataBuilder));
	TableMetadata updated = metadataBuilder.build();

https://apache-iceberg.slack.com/archives/C03LG1D563F/p1733195549342209

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[ebyhr]

I think we need to modify RemoveSnapshotRef for providing an option to clear snapshot-logs or not, or add a new MetadataUpdate class (e.g. ResetMainBranch).