How does Apache Gravitino relate to Apache Atlas?

[ottlukas]

Are these two datacatalogs with the same approach? What are the different architecture and goals in comparision?

Apache Atlas

[shaofengshi]

Gravitino and Atlas are all metadata management related projects, but they have many differences:

1. Atlas mainly works on hadoop, such as it supports Hive, HBase, Kafka, Storm, Impala, Sqoop as the source, and it's metadata is stored in HBase as well; Gravitino is new, which supports more data lake formats and lakehouse systems as the data source, such as Iceberg, Paimon, Hudi, Doris, etc (it supports Hive, Kafka as well); Gravitino doesn't have dependency on Hadoop, which uses RocksDB and RDBMS to store the metadata.
2. Atlas has a hook mechanism to sync and capture source system's metadata change to Atlas, in an async way; Gravitino connects to the source system in realtime, so its metadata is always the latest version.
3. Atlas's REST API is read-only (mostly I think, especially for the metadata); Gravitino provides a unified API for user not only read the metadata, but also can update the source systems' metadata; Some users are using it to centrally manage and control their metadata changes.
4. Atlas doesn't provide engine supports; Gravitino provides Trino, Apache Spark and Apache Flink connectors, which allows the engines to consume the catalogs directly.
5. Atlas doesn't have much support for unstructured data like fileset/volume; Gravitino has fileset for user to centrally manage unstructured data, which can be used in many AI/ML scenarios.
6. Atlas has many advanced features like metadata index and search, lineage, classification, notification which are not ready in Gravitino now, but they are on Gravitino's roadmap.

There is a blog comparing the product and technical data catalog, which is suitable for Atlas and Gravitino, worth to read:

https://medium.com/@lisancao/technical-vs-product-data-catalogs-which-one-is-best-for-you-6a54337e1ded

[ottlukas]

Thank you very much! :-)