Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cdc] paimon cdc database sync support computed columns #4193

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

MOBIN-F
Copy link
Contributor

@MOBIN-F MOBIN-F commented Sep 14, 2024

Purpose

Currently, there are still many paimon users who need to support computed columns when synchronizing multiple tables in the entire database. This PR will implement this feature.

If the referenced columned column exists in the table, the columned column function will be triggered. For example,

Table A column:
col1 col2 col3 create_sys_tm

Table B column:
col1 col4 col4 create_sys_tm

Define the calculated function:
--computed_column="_substring=substring(col3,2), pt=date_format(create_sys_tm,yyyyMMdd)"

The final table structure is as follows:
Table A column:
col1 col2 col3 _substring pt

Table B column:
col1 col4 col4 pt

The main changes of PR:
Move the implementation of xxxRecordParser#evalComputedColumns down to RichCdcMultiplexRecordEventParser#evalComputedColumns

Tests

MySqlSyncDatabaseTableListITCase#testComputedColumn

MongoDBSyncDatabaseActionITCase#testComputedColumn

KafkaOggSyncDatabaseActionITCase#testComputedColumn

KafkaMaxwellSyncDatabaseActionITCase#testComputedColumn

KafkaCanalSyncDatabaseActionITCase#testComputedColumn

API and Format

Documentation

@MOBIN-F
Copy link
Contributor Author

MOBIN-F commented Sep 23, 2024

@JingsongLi @yuzelin @zhuangchong can you help review this pr? tks~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant