Roadmap v0.2

[skyzh]

In v0.2, we focus on sorting everything -- from the composite sort key and primary key indexing in storage layer, to sort merge executors in executor layer, to composite key support in catalog.

Storage

• storage: support composite sort key in MergeIterator #588
• storage: support range-filter scan by sort key #589
• refactor storage to support snapshot isolation
• refactor VersionManager to include catalog

Optimizer

• filter push-down optimizer rule for all operators
• support column pruning for some operators #654
• add property to plan nodes optimizer: add property sorted to plan nodes #601

Executor

• optimize NestedLoopJoin to pull fewer data in memory executor: optimize NestedLoopJoin's memory usage #592
• executor: implement SortAgg #591
• executor: implement SortMergeJoin #597
• new expression framework to vectorize any scalar function and support type inference

SQL

• support subquery in binder binder: bind subqueries #578
• add function call support feat(function): add function definition #581
• make catalog consistent catalog: use a single lock for catalog #590

Tutorial

• basic query engine (expected April)
• query execution
• storage

Test Framework

• test: introduce planner and optimizer test #593

We're still drafting detailed design for each item, stay tuned! If you are interested in any of the item, please ping us in community channel for instructions, or create a tracking issue for that, thanks!

[MingjiHan99]

I will work on the binder for subquery.
It is time to update it for subqueries :)

[skyzh]

I will work on the binder for subquery.
It is time to update it for subqueries :)

I've created a new issue for you :) By the way, you may ping @xxchan, who is recently working on subqueries support on RisingWave.

[evanxg852000]

The reason I started following this repo was because of the prosed learning via the tutorial. This roadmap feels more like the focus is shifting to building a production-grade DB. We certainly have many of them. I think if the tutorial section is taken as a priority this will end up with more contributors (including me). Has been expecting the tutorials since February 😞

[xxchan]

@evanxg852000 I'd like to defend risinglight a little bit. Although the tutorial stood us up for a long while, risinglight is still very educationally valuable for me.

As a database newbee and a current risinglight member, I'd like to share my personal experences and feelings. Here's the history of my PRs:

I first browsed the list of good-first-issues, and found #434 a good starting point for me. Because I'm interested in how CLI works and displaying good message in CLI is also meaningful work. Another reason is that CLI is the entry point of the program, so it's a good starting point to learn how the whole system works. (And of course, also because the issue seems not hard to resolve.)

After it, I began to seek for bigger issues to work on (and leave the good-first-issue chances for others). And I choose #404, because implementing a new executor and optimization rules seems a good chance to learn the execution framework and optimization framework. And the task seems to be something I can do, because I can refer a lot to existing executors (and actually executors are not hard to understand).

At that time, I already got familiar with some components (executor/optimizer) and wanted to learn other components, e.g., storage, so I began to work on #428.

During this period, risinglight feels rather friendly to me:

• There are several good-first-issues, some of which even seems intentionally not fixed. They are not as trivial as typo fix. Instead, some do require a little bit investigation and can help understand how things work indeed.
• The structure is clear and the code is very neat. There are almost no dirty hacks or distracting non-critical details.
• The reviewers are quite responsive (including myself now 😄). PRs will be reviewed and if luckily, merged within one or two days. This is very precious as an open-source project.

So, what I want to convey is that, even without tutorial, risinglight is still a good learning material, and has a good community.

To clarify things, the maintainers seem absolutely not intented to make it a production-level DB. Instead, features (even in v0.2) are mostly basic ones instead of production-level killer features. Also, many chances are left to the community, instead of actively and quickly developed by the most powerful developers (yeah that's @skyzh).

But, last but not least, I totally agree with you and must say that, the delaying of the tutorial is frustrating😕, even though I enjoyed a lot the learn-by-doing experience. Maybe writing good tutorials is just harder than writing good code...

[evanxg852000]

Thanks @xxchan, I think I will just forget about the tutorials and start picking some tasks. The reason I am so inclined about the tuts is that I would expect design docs or at least an architecture of the whole thing. I think this will make life easier before proper tutorial land. Like, have a document describing all the current components at high level.

[xxchan]

@evanxg852000 Yes, I totally agree! I still don't understand some components like storage, and I'm also looking forward to reading the tutorial to gain more insights.

Actually, before I got started, I had kind of similar feeling with you and wanted to wait until the toturial come out. But I just had quite a lot free time then and thought why not give it a try😂. Then it turned out everything goes rather smooth. I feel very lucky now.

[skyzh]

I think we can have a storage doc soon if I have time to work on it this week 🤪

[evanxg852000]

@xxchan thanks, I will make the jump this coming weekend. most likely asking too many questions while working on a real task will put more pressure on @skyzh to make the doc ready 😄
I can already sense how responsive the community is here 🥇 .

[wangrunji0408]

Hi @evanxg852000! Thanks for your interest in our project! I'm the tutorial writer (and maybe the only one for a long time...). I must say sorry for the continued delay of the tutorial, to you and all community followers. 🙏

The tutorial was drafted from the end of last year. It was designed to be a series of posts and code to show how to build RisingLight step-by-step from scratch. I had finished splitting the code for each step, but when I started writing the post, I found it's much slower than I expected. It's true that writing is much harder than coding for me. Another bad news is that I have less time to contribute over the past few months, which made progress little. 😭

One last thing to note is that the tutorial is written in Chinese, since at the beginning we expected our main target would be Chinese college students (just like ourselves). However, there are increasing number of contributors from all over the world. So maybe we will switch to English in a near future. Currently you may have to use Google translate to read the tutorial...

Anyway, I also feel sad about the current progress. I know it's very important to have such a tutorial for newcomers, after all I was one of them 6 month ago. Please just give me more time and I'll try my best to finish it. Thank you all (

[evanxg852000]

@wangrunji0408 No need to be sorry open source work is hard and thanks for giving more details. What I definitely would suggest is that do not wait until all is clean. we can read and give feedback as you write. You can be assured we will participate in helping (feedback and translation). Thanks for the hard work.
I did not mean to put pressure on anyone. I just wanted to have the docs given more love ❤️ . Sorry again if it sounded like something else. You guys are doing amazing work on this repo. That's even why I am commenting & discussing.

[lokax]

I want to support column pruning for some operators.
Can I have a try 😇 😇

[skyzh]

I want to support column pruning for some operators.
Can I have a try 😇 😇

Of course! Just create a tracking issue on which operator you're working on, thanks!

[sfsf9797]

Hi, love this project!! May I know if there is any timeline for the tutorials mentioned in the roadmap? Thanks

[skyzh]

Hopefully by the end of this year.

[skyzh]

Unluckily some of the tasks might be postponed to v0.3. Stay tuned and thanks for your interest!

[sfsf9797]

hi, thanks, I think there is a tutorial about risinglight in Chinese? Could you share me the link?

[skyzh]

See risinglight-tutorial in this org.