Integrate Unity catalog for federated governance and data sharing

[caldeirav]

As a data product owner, I want to be able to define data access and sharing policies for my product, ideally at a granular level (column / row based filter).

Looking at a potential partnership with Databricks to engineer this as part of our FINOS engagement. Reference material:

https://github.com/unitycatalog/unitycatalog
https://medium.com/@kywe665/unity-catalog-oss-with-hudi-delta-iceberg-and-emr-duckdb-710ab8f8a7dc

[caldeirav]

As discussed in our weekly sprint today, the Unity catalog can be used jointly with X-Table for interoperability and governance.

High level overview: https://www.linkedin.com/pulse/getting-started-x-table-unity-catalog-universal-datalakes-soumil-shah-l3rpe/

[jpaulrajredhat]

@caldeirav just unity catalog deploy to our openshift and also deployed unity catalog web UI. Just build the image from the source and deploy it to OCP .. Still more work needs to be done. Right now the catalog is configured to use the default database which is read only . so, You can’t create any schema. I need to figure out what Database should be the best option . Let me know if you know any database we should use .

https://unitycatalog-ui-datamesh-demo.apps.rosa-8grhg.ssnp.p1.openshiftapps.com

[caldeirav]

Great progress - I am not sure the next step is to replace the database, because from the documentation it looks like Write is possible out of the box if we had Delta tables (https://docs.unitycatalog.io/usage/cli/). My take is we need to:

• Setup a new Iceberg REST catalogue with Apache Spark, leveraging UNIFORM tables (https://docs.unitycatalog.io/usage/tables/uniform/). If we can use the work we have done so far on Spark to ingest into uniform tables via Spark, we will be great.
• Setup the Trino integration (https://docs.unitycatalog.io/integrations/unity-catalog-trino/)

Optionally we can have a look at Daft as an alternative to Spark (https://docs.unitycatalog.io/integrations/unity-catalog-daft/).

If any issue I think the best will be to leverage the community on Slack, which seems pretty responsive.