REST encryption integration #13225

smaheshwar-pltr · 2025-06-03T21:08:13Z

This PR implements encryption for the REST catalog.

It's very similar to #13066 but for the REST catalog, not Hive.

smaheshwar-pltr · 2025-06-03T21:22:42Z

spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestTableEncryption.java

+// TODO(smaheshwar): This test is taken from https://github.com/apache/iceberg/pull/13066, with the
+// exception of testCtas, but adapted for the REST catalog. When that merges, we can directly use
+// those tests for the REST catalog as well by  adding to the parameters method there, to have a
+// single test class for table encryption.


Highlighting this - this file is largely taken from #13066

smaheshwar-pltr · 2025-06-03T21:26:48Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+              : Integer.parseInt(dekLength);
+    }
+
+    // Force re-creation of encryptingFileIO and encryptionManager


See #13066 (comment)

github-actions · 2025-07-07T00:19:57Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

smaheshwar-pltr · 2025-07-07T12:37:19Z

(Bump to remove staleness)

github-actions · 2025-08-07T00:19:41Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

smaheshwar-pltr · 2025-08-07T13:06:57Z

(Bump to remove staleness)

github-actions · 2025-09-08T00:17:56Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

smaheshwar-pltr · 2025-09-11T17:35:59Z

(Bump to remove staleness)

smaheshwar-pltr · 2025-09-23T15:29:31Z

core/src/main/java/org/apache/iceberg/encryption/StandardEncryptionManager.java

+   * @param dataKeyLength length of data encryption key (16/24/32 bytes)
+   * @param kmsClient Client of KMS used to wrap/unwrap keys in envelope encryption
+   */
+  public StandardEncryptionManager(


Could deprecate the old constructor

smaheshwar-pltr · 2025-09-23T15:32:52Z

spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestTableEncryption.java

+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.TestTemplate;
+
+// TODO(smaheshwar-pltr): This test is taken from https://github.com/apache/iceberg/pull/13066, with


I added a CTAS test to the ones authored by @ggershinsky here. Perhaps there's a world in which this PR merges first in which case can undo this comment. There are also review comments on these tests in #13066 that are potentially relevant

smaheshwar-pltr · 2025-09-26T13:48:06Z

Given #7770 is merged, curious for thoughts on this PR.
REST integration sounds on the cards. Also happy for this PR to be superseded if other folks have worked on it.

cc @huaxingao @RussellSpitzer @ggershinsky @rdblue

RussellSpitzer · 2025-09-27T00:04:50Z

Could you elaborate on the api additions? I think it would help to have some more description on the general direction of this or

smaheshwar-pltr · 2025-10-27T21:13:18Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

      @Override
      public FileIO io() {
+        // TODO(smaheshwar-pltr): Hive fetches encryption props here from uncommitted metadata
+        // RESTTableOperations.this.encryptionPropsFromMetadata(uncommittedMetadata);


I think the Hive table operations are a bit different in how they handle things. Commenting this out on the Hive side causes the new CTAS to fail for Hive, and leaving this in for REST causes that test to fail.

I believe the temporary operations using the same IO should not be problematic and the constructor that initialises the encryption props with the LoadTableResponse metadata means that things work here.

Happy to remove this TODO but @ggershinsky maybe you'd like to take a look? 🙏

Well, as long as the CTAS test is passing ok.. Encryption properties are delivered somewhat differently in this catalog.

smaheshwar-pltr · 2025-10-27T21:22:48Z

spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java

  }

+  @TestTemplate
+  public void testCtas() {


Adding this test, I think it's useful (https://github.com/apache/iceberg/pull/13225/files#r2467103538)

Maybe this can be done by extending the https://github.com/apache/iceberg/blob/main/spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestCTASEncryption.java ? This way, we won't have an unused table created before the test, and an undeleted table after the test.

also, the encryption is verified by a direct read without keys

Thank you! I somehow missed this new test class, must've been added after I started working on this - updated to use that, looks better now 🙌

smaheshwar-pltr · 2025-10-27T21:25:49Z

@huaxingao @smaheshwar-pltr Our team has a person who works on encryption with the REST catalog. If @smaheshwar-pltr does not object, we can follow up on this patch.

Sorry for lack of responses, was out of office for a bit.

@ggershinsky, I'm happy to continue with this PR unless the approaches significantly differ, in which case happy to also hand it off or accept a review here! 🙏

smaheshwar-pltr · 2025-10-27T21:27:58Z

Also, it would be good to refactor (if possible) a code common to this PR and to #13066 , so that other catalogs will be able to re-use it.

@ggershinsky, makes sense - though I struggled slightly; I think your comment here #13066 (comment) sums it up well

smaheshwar-pltr · 2025-10-27T21:29:07Z

@smaheshwar-pltr Could you please resolve the conflicts?

@huaxingao thanks for reminding, done! I think this PR is ready for an initial pass 🙏

smaheshwar-pltr · 2025-10-27T21:53:53Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+      TableMetadata.Builder builder = TableMetadata.buildFrom(metadata);
+      for (Map.Entry<String, EncryptedKey> entry :
+          EncryptionUtil.encryptionKeys(encryption()).entrySet()) {
+        builder.addEncryptionKey(entry.getValue());


Pointing out that this adds the required REST updates to the metadata to be committed. I think that additional REST requirements for these keys are not needed

ggershinsky · 2025-10-28T09:11:43Z

@huaxingao @smaheshwar-pltr Our team has a person who works on encryption with the REST catalog. If @smaheshwar-pltr does not object, we can follow up on this patch.

Sorry for lack of responses, was out of office for a bit.

@ggershinsky, I'm happy to continue with this PR unless the approaches significantly differ, in which case happy to also hand it off or accept a review here! 🙏

Sure, no problem. I think the approach is similar.

ggershinsky · 2025-10-28T09:13:07Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+  private EncryptingFileIO encryptingFileIO;
+  private String tableKeyId;
+  private int encryptionDekLength;
+  private List<EncryptedKey> encryptedKeysFromMetadata;


Please also factor in the #14427 changes

Thanks, I'll need to think about whether those changes are needed here actually, given the metadata file equality check in updateCurrentMetadata (test passes).

I suspect it's best to have such changes so when there's new metadata, the old keys are kept around. Will have to think if there's a case for this though

ggershinsky · 2025-10-29T06:32:32Z

spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java

    append.appendFile(dataFiles.get(0));
+
+    // append to the table in the meantime. use a separate load to avoid shared operations
+    validationCatalog.loadTable(tableIdent).newFastAppend().appendFile(dataFiles.get(0)).commit();


@szlta , could you have a quick look at this addition?

Sure, this does look good to me

huaxingao · 2025-11-01T22:48:32Z

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java

            RESTCatalogProperties.METRICS_REPORTING_ENABLED,
            RESTCatalogProperties.METRICS_REPORTING_ENABLED_DEFAULT);
+    if (mergedProps.containsKey(CatalogProperties.ENCRYPTION_KMS_IMPL)) {
+      this.kmsClient = EncryptionUtil.createKmsClient(mergedProps);


Should we add kmsClient to closeables so it’s closed with the catalog?

Great point, thanks! 0ea78c1

huaxingao · 2025-11-01T23:11:04Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+    encryptedKeysFromMetadata = metadata.encryptionKeys();
+
+    Map<String, String> tableProperties = metadata.properties();
+    if (tableKeyId == null) {


Shall we always get tableKeyId in case there is a key rotation?

I suppose we can always get it from the properties here though I suspect automatic key rotation often keeps key ID the same so perhaps reasonable to require that (otherwise would imagine you'd need tasks to commit new metadata with changed properties on rotation), cc @ggershinsky curious for thoughts / if there've been discussions on this?

if the tableKeyId == null (here and below), it just means the table is not encrypted.

singhpk234 · 2025-11-02T02:10:54Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

    List<MetadataUpdate> updates;
+
+    TableMetadata metadataToCommit = metadata;
+    if (encryption() instanceof StandardEncryptionManager) {


is it possible to add this update to the metadata before we call commit ?
per API contract we should try commit metadata

iceberg/core/src/main/java/org/apache/iceberg/TableOperations.java

Lines 45 to 63 in aa14aae

* Replace the base table metadata with a new version.

*

* <p>This method should implement and document atomicity guarantees.

*

* <p>Implementations must check that the base metadata is current to avoid overwriting updates.

* Once the atomic commit operation succeeds, implementations must not perform any operations that

* may fail because failure in this method cannot be distinguished from commit failure.

*

* <p>Implementations must throw a {@link

* org.apache.iceberg.exceptions.CommitStateUnknownException} in cases where it cannot be

* determined if the commit succeeded or failed. For example if a network partition causes the

* confirmation of the commit to be lost, the implementation should throw a

* CommitStateUnknownException. This is important because downstream users of this API need to

* know whether they can clean up the commit or not, if the state is unknown then it is not safe

* to remove any files. All other exceptions will be treated as if the commit has failed.

*

* @param base table metadata on which changes were based

* @param metadata new table metadata with updates

*/

Thanks for pointing this out, makes sense 🙏

I've mirrored the HiveTableOperations in this PR (I will try to reduce the duplication).

@ggershinsky I think this review comment along with others apply to the Hive side too that the REST-side should ideally have consistency with, WDYT here? Maybe makes sense to patch for Hive if relevant then I can rebase and reflect?

A couple of points,

The stated requirement is to "check that the base metadata is current to avoid overwriting updates". This is met. We add a missing field to the new metadata after the check.

There are lots of calls to commit from various classes. Moving this outside will be technically hard (if it works at all).

singhpk234 · 2025-11-02T02:11:59Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+    if (base != null) {
+      Set<String> removedProps =
+          base.properties().keySet().stream()
+              .filter(key -> !metadata.properties().containsKey(key))


nit: for consistency ?

Suggested change

.filter(key -> !metadata.properties().containsKey(key))

.filter(key -> !metadataToCommit.properties().containsKey(key))

Thanks - eacbfe7

singhpk234 · 2025-11-02T02:12:29Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+      if (removedProps.contains(TableProperties.ENCRYPTION_TABLE_KEY)) {
+        throw new RuntimeException("Cannot remove key in encrypted table");
+      }


shouldn't this be an IllegalState ? instead of RTE ?

cc @ggershinsky perhaps similar situation to #13225 (comment).

Thoughts on IllegalArgumentException? (If you alter this property and try to commit, IllegalArgumentException maybe makes sense?)

I'll be ok with any decision on this; will add to the HiveTO

singhpk234 · 2025-11-02T04:17:16Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+      String dekLength = tableProperties.get(TableProperties.ENCRYPTION_DEK_LENGTH);
+      encryptionDekLength =
+          (dekLength == null)
+              ? TableProperties.ENCRYPTION_DEK_LENGTH_DEFAULT
+              : Integer.parseInt(dekLength);


can use PropertyUtils ?

Agree, cc @ggershinsky maybe similar situation to #13225 (comment).

(Happy to think / put up these patches if it'd help, but the original code is @ggershinsky's so will wait 🙏)

Sure, I'm working on an "encryption clean up" patch, will add this to the todo list.

singhpk234 · 2025-11-02T04:23:51Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+    return encryptionManager;
+  }
+
+  private void encryptionPropsFromMetadata(TableMetadata metadata) {


Thank you everyone for great discussions !

There is one more perspective I wanna share, which IMHO we should consider

That metadata.json should be consistent to what we have done in other catalogs, one of the reasons to have a metadata.json is to have portability between different catalogs for example one can migrate from hive to REST and REST to hive, but if we store part of metadata in different place and plan to return in /config, imho it will become a concern to portability

Also please consider there are REST server implementations out there who cache the whole metadata.json in memory / db to avoid lookup to remote store.

In rest we can add requirement for an update that we can't drop it / meanwhile server can also double assert, if there is a concern for malicious clients and the key update or drop, then an implicit trust relation between catalog and client is required which means a catalog can reject the whole update if it doesn't trust the client ? if the concern is server tampering stuff, then a lot of things go at toss ? we requires similar trust relation for other works as wells :

DEFINER views

Access Decision Exchange

singhpk234 · 2025-11-02T04:34:59Z

core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

+      encryptionProperties.put(TableProperties.ENCRYPTION_TABLE_KEY, tableKeyId);
+      encryptionProperties.put(
+          TableProperties.ENCRYPTION_DEK_LENGTH, String.valueOf(encryptionDekLength));


any reason why we don't list these properties here : https://iceberg.apache.org/docs/1.10.0/docs/configuration/?h=write.m#table-properties

Good point, I'll send a patch to add them.

ggershinsky · 2025-11-06T09:11:48Z

I wonder if #14465 could affect safety of encrypted tables, and if yes, how this can be handled..

XJDKC · 2025-11-06T19:07:04Z

I wonder if #14465 could affect safety of encrypted tables, and if yes, how this can be handled..

IIUC, the encryption key must be accessible to the query engine in order for it to decrypt and encrypt the data. Could you please help me understand why allowing a custom builder would raise security concerns?

From my perspective, even without that PR, users could still copy the code and replace the default TableOperations implementation. The PR just simply introduces a generic interface to make such customization cleaner and more maintainable.

ggershinsky · 2025-11-06T20:50:41Z

Thanks @XJDKC , it's likely just a matter of documenting the new interface to make sure the users are aware of the security aspects of the REST TO (if they plan to use table encryption).

why allowing a custom builder would raise security concerns?

Maybe its ok, but we need to check the risk for metadata integrity (if broken, can be used for data leaks and other attacks), as discussed in this PR comments - making sure the client gets the metadata from the REST server, and not from the metadata.json file.
I'll have a look at the 14465 details to see if there are other security implications.

smaheshwar-pltr · 2025-11-07T23:48:29Z

Thanks @XJDKC , it's likely just a matter of documenting the new interface to make sure the users are aware of the security aspects of the REST TO (if they plan to use table encryption).

Maybe its ok, but we need to check the risk for metadata integrity (if broken, can be used for data leaks and other attacks), as discussed in this PR comments - making sure the client gets the metadata from the REST server, and not from the metadata.json file.

Thanks both for the discussions here.

I don't see anything concerning with the motivation behind the other PR. If a a REST client wanted to read from the metadata JSON in storage, they can do so regardless after calling loadTable - so enabling custom operations builders doesn't enable but facilitates custom client behaviour.

As such, I suspect the documentation should be at the spec level to advise clients if required, possibly depending on the corresponding REST spec discussions / conclusions (#14486 / https://lists.apache.org/thread/0nn11o4xf1nmw68d4px33sxw5tzzmgbo).

ggershinsky · 2025-11-09T09:21:34Z

I'm also fine with the motivation. Regarding the support for encrypted tables, I have two concerns,

Unrelated to REST. What if the encryption.key-id table property is set (https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/TableProperties.java#L391) but a custom TO implementation ignores it. Do the users expect a table to be encrypted if the encryption.key-id is set? Should the implementors of custom TOs validate them by running the Iceberg unitests (inc TestTableEncryption)?
Related to REST. The standard RESTableIOperations, built in Iceberg, is verified (in this PR) to be safe wrt metadata access. A custom TO replacement can behave differently of course.

I think both concerns can be addressed by adding a few lines to the RESTOperationsBuilder javadoc API comments (not to the REST spec, because the first point is unrelated, and the second is related only to client impl, not the server impl)

github-actions bot added spark core labels Jun 3, 2025

smaheshwar-pltr commented Jun 3, 2025

View reviewed changes

smaheshwar-pltr marked this pull request as ready for review June 3, 2025 21:24

smaheshwar-pltr mentioned this pull request Jun 3, 2025

Encryption integration and test #13066

Merged

smaheshwar-pltr commented Jun 3, 2025

View reviewed changes

smaheshwar-pltr mentioned this pull request Jun 8, 2025

AWS: KeyManagementClient implementation that works with AWS KMS #13136

Merged

github-actions bot added the stale label Jul 7, 2025

github-actions bot removed the stale label Jul 8, 2025

github-actions bot added the stale label Aug 7, 2025

github-actions bot removed the stale label Aug 8, 2025

github-actions bot added the stale label Sep 8, 2025

github-actions bot removed the stale label Sep 12, 2025

Sreesh Maheshwar added 3 commits September 23, 2025 16:08

REST encryption integration

43cb728

fix typo

00a6ffc

Rebase properly

889cbbf

smaheshwar-pltr force-pushed the rest-encrypt-on-main-cherry-pick branch from d007aa2 to 889cbbf Compare September 23, 2025 15:23

smaheshwar-pltr mentioned this pull request Sep 23, 2025

Encryption for REST catalog #13224

Closed

Nit

1faac8b

smaheshwar-pltr commented Sep 23, 2025

View reviewed changes

Trigger CI

e4839a6

Sreesh Maheshwar added 6 commits October 27, 2025 18:05

Merge branch 'main' into rest-encrypt-on-main-cherry-pick

a7627c4

format fixes

f44ce34

format fixes

d77a301

throw if removed properties

82fdbb5

encryption

a2c0277

spotless

186aadd

smaheshwar-pltr commented Oct 27, 2025

View reviewed changes

Name fixes

16a22b2

smaheshwar-pltr commented Oct 27, 2025

View reviewed changes

ggershinsky reviewed Oct 28, 2025

View reviewed changes

Sreesh Maheshwar added 3 commits October 29, 2025 00:11

Use existing CTAS test instead

7c6da18

Merge branch 'main' into rest-encrypt-on-main-cherry-pick

2f3c939

Merge

f251f42

ggershinsky reviewed Oct 29, 2025

View reviewed changes

huaxingao reviewed Nov 1, 2025

View reviewed changes

singhpk234 reviewed Nov 2, 2025

View reviewed changes

ggershinsky mentioned this pull request Nov 5, 2025

REST spec: Metadata integrity for encrypted tables #14486

Open

Sreesh Maheshwar added 2 commits November 8, 2025 00:01

Add KMS client to closeables

0ea78c1

Prefer metadataToCommit over metadata in commit method

eacbfe7

	* Replace the base table metadata with a new version.
	*
	* <p>This method should implement and document atomicity guarantees.
	*
	* <p>Implementations must check that the base metadata is current to avoid overwriting updates.
	* Once the atomic commit operation succeeds, implementations must not perform any operations that
	* may fail because failure in this method cannot be distinguished from commit failure.
	*
	* <p>Implementations must throw a {@link
	* org.apache.iceberg.exceptions.CommitStateUnknownException} in cases where it cannot be
	* determined if the commit succeeded or failed. For example if a network partition causes the
	* confirmation of the commit to be lost, the implementation should throw a
	* CommitStateUnknownException. This is important because downstream users of this API need to
	* know whether they can clean up the commit or not, if the state is unknown then it is not safe
	* to remove any files. All other exceptions will be treated as if the commit has failed.
	*
	* @param base table metadata on which changes were based
	* @param metadata new table metadata with updates
	*/

	.filter(key -> !metadata.properties().containsKey(key))
	.filter(key -> !metadataToCommit.properties().containsKey(key))

REST encryption integration #13225

Are you sure you want to change the base?

REST encryption integration #13225

Conversation

smaheshwar-pltr commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

smaheshwar-pltr commented Jul 7, 2025

Uh oh!

github-actions bot commented Aug 7, 2025

Uh oh!

smaheshwar-pltr commented Aug 7, 2025

Uh oh!

github-actions bot commented Sep 8, 2025

Uh oh!

smaheshwar-pltr commented Sep 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smaheshwar-pltr commented Sep 26, 2025

Uh oh!

RussellSpitzer commented Sep 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smaheshwar-pltr commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smaheshwar-pltr commented Oct 27, 2025

Uh oh!

smaheshwar-pltr commented Oct 27, 2025

Uh oh!

smaheshwar-pltr Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggershinsky commented Oct 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smaheshwar-pltr Nov 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

smaheshwar-pltr commented Jun 3, 2025 •

edited

Loading

smaheshwar-pltr commented Oct 27, 2025 •

edited

Loading

smaheshwar-pltr Oct 27, 2025 •

edited

Loading

smaheshwar-pltr Nov 8, 2025 •

edited

Loading

ggershinsky Nov 9, 2025 •

edited

Loading

smaheshwar-pltr Nov 8, 2025 •

edited

Loading