Skip to content

Conversation

@singhpk234
Copy link
Contributor

@singhpk234 singhpk234 commented Nov 4, 2025

About the change

Introduces a transformSnapshots API in the TableMetadata, one use case which will be helpful is snapshot summary contains information like total files / total-records etc and when partition summary is enabled it contains partition summary which is a sensitive information incase the table is protected against FGAC specially Row Access Policy, this essentially will help in dropping the snapshot summary by the RESTCatlaog before the table metadata is being sent to the being sent to the client (untrusted) as part of LoadTable response.
Never the less dropping the summary from snapshot obj can reduce the size transfer cost.

Testing

unit tests for w/wo lazy loading

@github-actions github-actions bot added the core label Nov 4, 2025
@sfc-gh-prsingh sfc-gh-prsingh force-pushed the feature/snapshot-transformer branch 2 times, most recently from 728959e to 76306db Compare November 4, 2025 19:15
@singhpk234 singhpk234 closed this Nov 4, 2025
@singhpk234 singhpk234 reopened this Nov 4, 2025
@singhpk234 singhpk234 closed this Nov 4, 2025
@singhpk234 singhpk234 reopened this Nov 5, 2025
@singhpk234
Copy link
Contributor Author

singhpk234 commented Nov 5, 2025

I was talking to @stevenzwu, he suggested a nice alternative here inspired by StructProjection adding it here for broder forum, techinically indeed its a projection

class TableMetadataProjection {

  TableMetadata create(TableMetadata metadata, Function<Snapshot, Snapshot> transformer) {
    new SnapshotTransformer(metadata.())
  }

  private static class SnapshotTransformer extends TableMetadata {
    @Override
    public List<Snapshot> snapshots() {
      List<Snapshot> s = super.snapshot()
      // apply the tranform on s ?
    }
  }

In this approach we will need to make all the members getters, they can be package private for sure ! we additionally would want to serialize the tranformer too if we want to send this back as part of loadTableResponse ?

@singhpk234 singhpk234 requested a review from stevenzwu November 5, 2025 02:48
@sfc-gh-prsingh sfc-gh-prsingh force-pushed the feature/snapshot-transformer branch from 76306db to af17886 Compare November 5, 2025 14:49
@sfc-gh-prsingh sfc-gh-prsingh force-pushed the feature/snapshot-transformer branch from af17886 to be138e7 Compare November 5, 2025 15:29
@singhpk234 singhpk234 marked this pull request as ready for review November 6, 2025 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants