Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ticdc(sink) Write DML events to pulsar #9622

Merged
merged 20 commits into from
Aug 25, 2023
Merged

ticdc(sink) Write DML events to pulsar #9622

merged 20 commits into from
Aug 25, 2023

Conversation

yumchina
Copy link
Contributor

What problem does this PR solve?

add DML event to pulsar
Issue Number: ref #9413

What is changed and how it works?

Check List

Tests

  • Unit test

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

None

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Aug 22, 2023

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot bot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. contribution This PR is from a community contributor. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Aug 22, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Aug 22, 2023

Hi @yumchina. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Aug 22, 2023
@yumchina yumchina changed the title Dev merge replica config dml ticdc(sink) Add write DML event to pulsar Aug 22, 2023
@asddongmen asddongmen removed the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Aug 22, 2023
@asddongmen
Copy link
Contributor

/ok-to-test

@ti-chi-bot ti-chi-bot bot added ok-to-test Indicates a PR is ready to be tested. and removed needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Aug 22, 2023
@yumchina yumchina changed the title ticdc(sink) Add write DML event to pulsar ticdc(sink) Write DML event to pulsar Aug 22, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Aug 22, 2023

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot bot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Aug 22, 2023
@yumchina yumchina changed the title ticdc(sink) Write DML event to pulsar ticdc(sink) Write DML events to pulsar Aug 22, 2023
asddongmen
asddongmen reviewed Aug 22, 2023
View reviewed changes
cdc/sink/dmlsink/mq/pulsar_dml_sink.go
return nil, cerror.WrapError(cerror.ErrPulsarNewProducer, err)
}

topicManager, err := pulsarTopicManagerCreator(pConfig, client)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
topicManager, err := pulsarTopicManagerCreator(pConfig, client)
// The topicManager is not actually used; it is only used to create dmlSink.
// TODO: Find a way to remove it.
topicManager, err := pulsarTopicManagerCreator(pConfig, client)

@asddongmen asddongmen removed the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Aug 23, 2023
@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Aug 24, 2023
3AceShowHand
3AceShowHand reviewed Aug 24, 2023
View reviewed changes
cdc/sink/dmlsink/mq/dmlproducer/pulsar_dml_producer.go
zap.String("changefeed", p.id.ID), zap.String("topic", topicName))
}

p.client.Close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also close the client asynchronously ?

@ti-chi-bot ti-chi-bot bot added the lgtm label Aug 24, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Aug 24, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, asddongmen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [3AceShowHand,asddongmen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Aug 24, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Aug 24, 2023

[LGTM Timeline notifier]

Timeline:

  • 2023-08-24 08:48:37.468759943 +0000 UTC m=+1398482.017775929: ☑️ agreed by asddongmen.
  • 2023-08-24 09:00:39.838556529 +0000 UTC m=+1399204.387572517: ☑️ agreed by 3AceShowHand.

@asddongmen
Copy link
Contributor

/retest

@ti-chi-bot ti-chi-bot bot merged commit c703296 into pingcap:master Aug 25, 2023
3 checks passed
3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this pull request Aug 29, 2023
@yumchina @3AceShowHand
ticdc(sink) Write DML events to pulsar (pingcap#9622)
d098c73 
ref pingcap#9413
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@asddongmen asddongmen asddongmen approved these changes

@3AceShowHand 3AceShowHand 3AceShowHand approved these changes

@nongfushanquan nongfushanquan Awaiting requested review from nongfushanquan

Assignees
No one assigned
Labels
approved contribution This PR is from a community contributor. lgtm ok-to-test Indicates a PR is ready to be tested. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@yumchina @asddongmen @3AceShowHand