[Ideas] The Way to Apache Iceberg

[Xuanwo]

Description

Hi, thank you for your efforts to revive GP and donate Cloudberry to the ASF.

I've read your incubator proposal and am very interested in this item:

Alignment

...

• Additionally, Cloudberry will integrate with Apache Iceberg, Apache Hudi, or Hadoop to help build a robust lakehouse data platform.

For context, I'm from the Iceberg community and currently working on iceberg-rust: the Rust implementation of Apache Iceberg™.

Given that CloudberryDB is written in C and C++, I'm wondering if it's possible for our communities to collaborate so that CloudberryDB can integrate with Iceberg-Rust to provide native Iceberg support.

Use case/motivation

Build a robust lakehouse data platform using CloudberryDB. As the first step, add native support for Apache Iceberg (via iceberg-rust).

Related references

• Iceberg Mailing List: mailto:[email protected]
• Iceberg Slack: iceberg#rust

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[my-ship-it]

It's an interesting topic. In fact, we already support IceBerg in Lightning(Cloudberry commercial version).

[jianlirong]

We appreciate you bringing up this topic, @Xuanwo . As we move towards the lakehouse architecture, the Cloudberry community is committed to increasing our investment in Apache Iceberg integration. As @my-ship-it pointed out, a commercial version of Cloudberry already supports Apache Iceberg, and we plan to open-source this feature soon. However, our current implementation has two limitations: (1) it only supports reading from Apache Iceberg tables, and (2) it's written in both Java and C++ for efficiency reasons. The Java part handles metadata, while C++ handles data. This hybrid approach adds complexity and introduces potential performance overhead due to the JVM.

We are very interested in leveraging iceberg-rust to address the challenges we're facing with our current implementation. Furthermore, given the growing popularity of Rust in system programming, we are actively exploring the possibility of developing PostgreSQL/Cloudberry extensions in Rust. I personally believe that iceberg-rust is a great starting point.

So, how should we start? Any suggestion? Thanks.

[Xuanwo]

Hi, iceberg-rust is currently under active development. At the time of writing, iceberg-rust now fully supports reading. For instance, one of Iceberg-rust's users, Databend, is able to run the full TPC-H workload over iceberg-rust. Additionally, the community is diligently working on writing support and Rust-native compaction.

As a starting point, I think we should explore ways to integrate iceberg-rust inside and figure out how to effectively utilize it. This definitely involves a new approach and requires setting up various aspects such as contributing to documentation, establishing CI, and so forth.

[jianlirong]

Cool. We will do some investigations first, and then get back to you. Thanks.