Cloudberry Database Roadmap 2024

[tuhaihe]

This roadmap outlines the key milestones and focuses on the Cloudberry Database project in 2024. It aims to enhance functionality to strengthen further the Cloudberry Database's position as the next-gen unified open-source database for Analytics and AI.

Note

Some plans may change depending on how the Cloudberry Database evolves in reality.

Features and Enhancements

• Implement federated queries across multiple Cloudberry Database clusters. (Use FDW to query multiple servers as shards #320 in review)
• Optimize query performance by leveraging materialized views.
• Refactor the dispatch logic for improved efficiency.
• Enhance the access method interface. Add new callback 'scan_flags' for table access method #391
• Support directory table to manage files. Implement of Directory table #390
• Enable deployment on a single node. SingleNode deployment #206
• Support hybrid Row-Column storage for Cloudberry Database. (to be open sourced)
• Support vectorization to optimize the query performance. (to be open sourced)

Kernel Update

• Upgrade Cloudberry Database kernel from PostgreSQL 14.4 to PostgreSQL 15.x stable release.

Streaming

• Support streaming data loading.
• Support the materialized views refresh mechanism, triggered by manual or automatic tasks, to enable near real-time analytics. (WIP)
• Support Dynamic Table.

Lakehouse

• Integration with Apache Iceberg
• Integration with Apache Hudi
• Integration with Delta Lake, Hadoop

AI/ML

• MADlib for Cloudberry Database: @cloudberrydb/madlib
• Support the distributed Python compute engine via Ray for AI/ML workloads.

Utilities, Tools, and Ecosystem

• PostGIS for Cloudberry Database: @cloudberrydb/postgis
• PXF for Cloudberry Database: @cloudberrydb/pxf
• PL/R for Cloudberry Database: @cloudberrydb/plr
• PL/Java for Cloudberry Database: @cloudberrydb/pljava
• JDBC/ODBC for Cloudberry Database
• Container Service for Cloudberry Database
• K8S deployment support for Cloudberry Database
• Cloudberry Database Manager, one graphical tool with the deployment, SQL editor and monitoring dashboard, etc. (to be open sourced)
• Migration tool to help users migrate from Greenplum to Cloudberry Database.

Release

We plan to release the Cloudberry Database at the following pace:

	v1.5.x	v1.6.x	v1.7.x
Jan	1.5.0
Feb	1.5.1 / 1.5.0
Mar	1.5.2 / 1.5.1
Apr	1.5.2	1.6.0 rc1
May	1.5.3pre-release	1.6.0 rc2
Jun		1.6.0
Jul	1.5.4	1.6.1
Aug		1.6.2
Sept			1.7.0 rc1
Oct			1.7.0 rc2
Nov			1.7.0
Dec			1.7.1

In addition, we will support more CPUs and operating systems for the release packages, including Linux for x86_64/aarch64, and KylinOS for x86_64/aarch64.

Ask for feedbacks

We want to invite you to join us in shaping the future of Cloudberry Database. We'd love to hear your thoughts and ideas, so please don't hesitate to share them with us. We're committed to making our platform as open, user-friendly, and innovative as possible, and your feedback will help us achieve that goal. Thank you for your support, and we can't wait to hear from you!

---

_{Want to try it out quickly? Follow our Bootcamp program to get started with the Cloudberry Database Sandbox.}

[francksidi-edb]

Hi can you share what are the changes regarding compute/storage separation in the architecture and how we can dynamically increase/decrease number of segments ?

[tuhaihe]

For open-source Cloudberry, it is still in MPP architecture. The Cloudberry vendor, HashData, has implemented compute/storage separation in its commercial product.

I am wondering where you are seeing the information. We can update it to avoid confusing anyone.

[tuhaihe]

We now support gpshrink and gpexpand to scale out/in segments.

for gpshrink, you can check the PR #393

[pjvldb]

Iceberg, PostGIS & GUI are all very smart moves.

[tuhaihe]

Thanks @pjvldb ❤️

[robozor]

Hello everyone,

I'm currently working with the Cloudberry Database and I'm interested in using the pgvector extension for vector similarity searches. Before proceeding, I wanted to ask if anyone has experience with installing or using pgvector within Cloudberry Database.

Is it compatible with Cloudberry, given its PostgreSQL 14.4 kernel? Are there any specific steps or considerations I should be aware of when setting it up?

Any insights or guidance would be greatly appreciated.

Thank you!

Josef Szylar

[francksidi-edb]

Hi, I successfully installed pgvector extension 0.7 on Cloudberry on a 1 node system.

[roseduan]

We have already integrated pgvector extension into Cloudberry.
https://github.com/cloudberrydb/pgvector

[reshke]

Is greenplum6->cloudberry migration native support in this roadmap?
That is, upgrading existing greenplum6 deployment to cloudberry via pg_upgrade (gp_upgrade)

[tuhaihe]

yes, thanks for your reminder. will open source one business migration tool to help users migrate from GP to Cloudberry. I can udpate this to the roadmap later.

[tuhaihe]

Oh, you mean the native migration support, right?
Any thoughts? @jianlirong @my-ship-it

[reshke]

Oh, you mean the native migration support, right? Any thoughts? @jianlirong @my-ship-it

I mean something similar to pg_upgrade.

[jianlirong]

It seems like we need to do some investigations on the differences in system catalogs and data table formats between GP 6 and Cloudberry. Could you help us with this, @reshke ? Once we have a clear understanding of the differences, we can discuss the implementation options in more detail.

[reshke]

It seems like we need to do some investigations on the differences in system catalogs and data table formats between GP 6 and Cloudberry. Could you help us with this, @reshke ? Once we have a clear understanding of the differences, we can discuss the implementation options in more detail.

Ok, i will do my best
as a starting point i propose #599

[techkuz]

does airflow.providers.postgres.operators.postgres import PostgresOperator support cloudberry?

[my-ship-it]

does airflow.providers.postgres.operators.postgres import PostgresOperator support cloudberry?

Cloudberry keeps interface compatibility with postgres. So integration with airflow seems not hard. If you found any issues when integrating with airflow, please file issues here: https://github.com/apache/cloudberry/issues