Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Whole database synchronization] Dinky whole database synchronizes mysql to doris, how to automatically create doris tables based on the configured table name suffix and suffix to achieve data synchronization #3807

Closed
2 of 3 tasks
NCUZK opened this issue Sep 13, 2024 · 2 comments
Assignees
Labels
FAQ Frequently Asked Questions
Milestone

Comments

@NCUZK
Copy link

NCUZK commented Sep 13, 2024

Search before asking

  • I had searched in the issues and found no similar feature requirement.

Description

当前现状:

  1. 测试发现需要doris侧先手动建表,然后才能同步,如果不创建会出现类似错误
    org.apache.doris.flink.sink.DorisSink(594): Doris FE's response cannot map to schema. res: "errCode=7,detailMessage=unknowntable,tableName=test" org.apache.doris.shaded.com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type int from String "errCode=7,detailMessage=unknowntable,tableName=test": not a valid int value #3667

Use case

实现类似flink-doris-connector整库同步效果
https://doris.apache.org/zh-CN/docs/2.0/ecosystem/flink-doris-connector/#mysql-%E5%A4%9A%E8%A1%A8%E5%90%8C%E6%AD%A5%E7%A4%BA%E4%BE%8B

<FLINK_HOME>bin/flink run
-Dexecution.checkpointing.interval=10s
-Dparallelism.default=1
-c org.apache.doris.flink.tools.cdc.CdcTools
lib/flink-doris-connector-1.16-1.6.1.jar
mysql-sync-database
--database test_db
--mysql-conf hostname=127.0.0.1
--mysql-conf port=3306
--mysql-conf username=root
--mysql-conf password=123456
--mysql-conf database-name=mysql_db
--including-tables "tbl1|test.*"
--sink-conf fenodes=127.0.0.1:8030
--sink-conf username=root
--sink-conf password=123456
--sink-conf jdbc-url=jdbc:mysql://127.0.0.1:9030
--sink-conf sink.label-prefix=label
--table-conf replication_num=1

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@NCUZK NCUZK added New Feature New feature Waiting for reply Waiting for reply labels Sep 13, 2024
@github-actions github-actions bot changed the title [Feature][整库同步] dinky整库同步mysql到doris,如何根据配置的表名前后缀自动创建doris表,实现数据同步 [Feature][Whole database synchronization] Dinky whole database synchronizes mysql to doris, how to automatically create doris tables based on the configured table name suffix and suffix to achieve data synchronization Sep 13, 2024
@Zzm0809
Copy link
Contributor

Zzm0809 commented Sep 14, 2024

直接使用 execute jar 语法即可
https://www.dinky.org.cn/docs/next/extend/expand_statements/execute_jar

@Zzm0809 Zzm0809 added FAQ Frequently Asked Questions and removed Waiting for reply Waiting for reply New Feature New feature labels Sep 14, 2024
@aiwenmo
Copy link
Contributor

aiwenmo commented Sep 18, 2024

Auto create table only supports mysql.
Other database need to extend.

@aiwenmo aiwenmo self-assigned this Sep 18, 2024
@aiwenmo aiwenmo added this to the 1.2.0 milestone Sep 18, 2024
@aiwenmo aiwenmo moved this to ToDo in Dinky Roadmap Sep 18, 2024
@Zzm0809 Zzm0809 closed this as completed Sep 23, 2024
@aiwenmo aiwenmo moved this from ToDo to Done in Dinky Roadmap Nov 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FAQ Frequently Asked Questions
Projects
Status: Done
Development

No branches or pull requests

3 participants