Skip to content

Conversation

shubhampatel28
Copy link

What is the purpose of the pull request

  • This PR introduces a new Spark catalog plugin that provides unified access to both Hudi and Iceberg tables using AWS Glue as the metastore. The plugin automatically detects table formats and delegates operations to the appropriate catalog implementation (Iceberg's SparkCatalog or Hudi's HoodieCatalog), enabling seamless querying of mixed table formats through a single catalog interface.

Brief change log

  • Added new module xtable-spark-plugin to parent POM
  • Implemented XTableSparkCatalog class with unified catalog interface for Hudi and Iceberg tables
  • Added automatic table format detection using multiple fallback strategies:
  • Integrated AWS Glue client for table metadata lookup with cross-account support

Verify this pull request

Added TestXTableSparkCatalog for unit tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant