feat: support altering the target table’s columns of the sink.

[shanicky]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Title: Enhance Sink Behavior to Handle Table Alterations and Add Associated Tests

Summary:

This pull request implements enhancements to the system's behavior concerning sinks when table alterations occur. The changes span across several areas of the codebase, including test scripts, protobuf definitions, and Rust source code. Here's a breakdown of the key additions and modifications:

• A new end-to-end test script alter_column.slt has been added under e2e_test/sink/sink_into_table. This script validates the system's ability to manage sinks during various table alteration operations, including creating tables and sinks, inserting values, and adding or attempting to drop columns. It further checks if the sink tables reflect the correct state after these operations.

• Removal of a specific test case from basic.slt indicates an important change in the behavior of the system. The removed test case previously checked for an error when altering a table by adding a column with active incoming sinks. This change implies that the system may now support adding columns to a populated sink table.

• The protobuf definition for a Sink within catalog.proto has been updated to incorporate a repeated field original_target_columns. This new field stores the initial structure of a sink's target table, facilitating the management of table alterations in relation to sinks.

• The main body of Rust source code changes revolves around handling table alterations when sinks are involved:

  • The modules sink/catalog/desc.rs and sink/catalog/mod.rs have been updated to manage the new original_target_columns data.
  • A new function called default_column_exprs has been added to frontend/src/catalog/table_catalog.rs. This function generates default expressions for columns, which is essential for populating default values in rows added to altered tables.
  • The code within frontend/src/handler/alter_table_column.rs has been augmented to enable table alterations in the presence of sinks, highlighting improved support for adding columns to tables with active sinks.

• The pull request also notes the use of Rust HashMap and Vec collections to maintain the newly introduced data points, signaling improvements in data handling, particularly for append-only tables engaged in sink operations.

This pull request presents a comprehensive enhancement to the system, focusing on improved sink interaction with table alterations. Reviewers can delve into the individual elements of this update for a more detailed understanding, and further discussions are encouraged to fine-tune these changes.

Please review these changes and provide feedback on any aspects that may benefit from further clarification

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• All checks passed in ./risedev check (or alias, ./risedev c)

[gitguardian]

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
Once a secret has been leaked into a git repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

---

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}}

[st1page]

original_target_columns compatibility code is scattered across the meta and frontend, and only performs compatibility after changing the schema of the table. Seems a bit easy to get wrong. Can we execute the compatibility code on meta startup to fill all sink values?

[st1page]

LGTM

[st1page]

It might be clear to explicitly populate these defaults value and add an todo comment here? #17658

[shanicky]

It seems like there's no way to resolve the single node test stackoverflow issue. 🥵

[st1page]

It seems like there's no way to resolve the single node test stackoverflow issue. 🥵

:lark_cry

[fuyufjh]

Shall we try rust_min_stack