Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cdc: add description about checksum v2 #18273

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

3AceShowHand
Copy link
Contributor

@3AceShowHand 3AceShowHand commented Aug 5, 2024

First-time contributors' checklist

What is changed, added or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions (in Chinese).

  • master (the latest development version)
  • v8.3 (TiDB 8.3 versions)
  • v8.2 (TiDB 8.2 versions)
  • v8.1 (TiDB 8.1 versions)
  • v8.0 (TiDB 8.0 versions)
  • v7.5 (TiDB 7.5 versions)
  • v7.1 (TiDB 7.1 versions)
  • v6.5 (TiDB 6.5 versions)
  • v6.1 (TiDB 6.1 versions)
  • v5.4 (TiDB 5.4 versions)
  • v5.3 (TiDB 5.3 versions)
  • v5.2 (TiDB 5.2 versions)

What is the related PR or file link(s)?

  • This PR is translated from:
  • Other reference link(s):

Do your changes match any of the following descriptions?

  • Delete files
  • Change aliases
  • Need modification after applied to another branch
  • Might cause conflicts after applied to another branch

@ti-chi-bot ti-chi-bot bot added the missing-translation-status This PR does not have translation status info. label Aug 5, 2024
@ti-chi-bot ti-chi-bot bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Aug 5, 2024
@3AceShowHand
Copy link
Contributor Author

This is planned to be release since v8.4.0

@Oreoxmt Oreoxmt added v8.4 This PR/issue applies to TiDB v8.4. translation/doing This PR’s assignee is translating this PR. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Aug 5, 2024
@ti-chi-bot ti-chi-bot bot removed the missing-translation-status This PR does not have translation status info. label Aug 5, 2024
@Oreoxmt Oreoxmt self-assigned this Aug 5, 2024
Copy link

ti-chi-bot bot commented Sep 14, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from oreoxmt, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link

ti-chi-bot bot commented Sep 14, 2024

@flowbehappy: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 14, 2024
@Oreoxmt Oreoxmt added translation/done This PR has been translated from English into Chinese and updated to pingcap/docs-cn in a PR. and removed translation/doing This PR’s assignee is translating this PR. labels Sep 19, 2024
@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Sep 23, 2024
Copy link
Collaborator

@qiancai qiancai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM


### Checksum V1

在 v8.4.0 之前,TiDB 和 TiCDC 采用 Checksum v1 进行 Checksum 计算和校验。
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
在 v8.4.0 之前,TiDB 和 TiCDC 采用 Checksum v1 进行 Checksum 计算和校验。
在 v8.4.0 之前,TiDB 和 TiCDC 采用 Checksum v1 算法进行 Checksum 计算和校验。


在 v8.4.0 之前,TiDB 和 TiCDC 采用 Checksum v1 进行 Checksum 计算和校验。

在启用单行数据 Checksum 正确性校验功能后,TiDB 使用 CRC32 算法计算该行数据的 Checksum 值,并将其一并写入 TiKV。TiCDC 从 TiKV 读取数据,根据相同的算法重新计算 Checksum,如果该值与 TiDB 写入的值相同,则可以证明数据在 TiDB 至 TiCDC 的传输过程中是正确的。
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
在启用单行数据 Checksum 正确性校验功能后,TiDB 使用 CRC32 算法计算该行数据的 Checksum 值,并将其一并写入 TiKVTiCDC 从 TiKV 读取数据,根据相同的算法重新计算 Checksum,如果该值与 TiDB 写入的值相同,则可以证明数据在 TiDB TiCDC 的传输过程中是正确的。
在启用单行数据 Checksum 正确性校验功能后,TiDB 会使用 CRC32 算法计算该行数据的 Checksum 值,并将这个值与该行数据一并存储在 TiKV 中。随后,TiCDC 从 TiKV 读取这些数据,并使用相同的算法重新计算 Checksum,如果得到的 Checksum 值与 TiDB 写入的 Checksum 值相同,则表明数据在从 TiDB TiCDC 的传输过程中是正确的。


在启用单行数据 Checksum 正确性校验功能后,TiDB 使用 CRC32 算法计算该行数据的 Checksum 值,并将其一并写入 TiKV。TiCDC 从 TiKV 读取数据,根据相同的算法重新计算 Checksum,如果该值与 TiDB 写入的值相同,则可以证明数据在 TiDB 至 TiCDC 的传输过程中是正确的。

TiCDC 将数据编码成特定格式并发送至 Kafka。Kafka Consumer 读取数据后,可以使用与 TiDB 相同的算法计算得到新的 Checksum,将此值与数据中携带的 Checksum 值进行比较,若二者一致,则可证明从 TiCDC 至 Kafka Consumer 的传输链路上的数据是正确的。
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
TiCDC 将数据编码成特定格式并发送至 Kafka。Kafka Consumer 读取数据后,可以使用与 TiDB 相同的算法计算得到新的 Checksum,将此值与数据中携带的 Checksum 值进行比较,若二者一致,则可证明从 TiCDC Kafka Consumer 的传输链路上的数据是正确的。
TiCDC 将数据编码成特定格式并发送至 Kafka。Kafka Consumer 读取数据后,可以使用与 TiDB 相同的 CRC32 算法计算得到新的 Checksum,将此值与数据中携带的 Checksum 值进行比较,若二者一致,则表明从 TiCDC Kafka Consumer 的传输链路上的数据是正确的。


### Checksum V2

从 v8.4.0 开始,TiDB 和 TiCDC 引入 Checksum V2 算法,解决了 Checksum V1 在执行 Add Column 或 Drop Column 后无法正确校验 Update 或 Delete 事件中 Old Value 数据的问题。
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
从 v8.4.0 开始,TiDB 和 TiCDC 引入 Checksum V2 算法,解决了 Checksum V1 在执行 Add ColumnDrop Column 后无法正确校验 Update 或 Delete 事件中 Old Value 数据的问题。
从 v8.4.0 开始,TiDB 和 TiCDC 引入 Checksum V2 算法,解决了 Checksum V1 在执行 `ADD COLUMN``DROP COLUMN` 后无法正确校验 Update 或 Delete 事件中 Old Value 数据的问题。

Copy link
Collaborator

@qiancai qiancai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

Copy link
Collaborator

@qiancai qiancai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 24, 2024
Copy link

ti-chi-bot bot commented Sep 24, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-09-23 08:01:17.390506097 +0000 UTC m=+1466547.130930037: ☑️ agreed by Oreoxmt.
  • 2024-09-24 01:33:46.801885739 +0000 UTC m=+1529696.542309678: ☑️ agreed by qiancai.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm size/S Denotes a PR that changes 10-29 lines, ignoring generated files. translation/done This PR has been translated from English into Chinese and updated to pingcap/docs-cn in a PR. v8.4 This PR/issue applies to TiDB v8.4.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants