ddl_puller.go(ticdc): fix DDLs are ignored when schema versions are out of order (#11733)

[ti-chi-bot]

This is an automated cherry-pick of #11733

What problem does this PR solve?

Issue Number: close #11714

What is changed and how it works?

Description:
This pull request addresses a bug in TiCDC where a DDL job could be inadvertently dropped if a subsequent DDL job with a higher SchemaVersion but lower CommitTs is processed first. This behavior occurs because the ddlJobPuller processes DDL jobs based on their CommitTs in ascending order and uses a check against SchemaVersion that can prevent older DDL jobs from being processed, leading to data inconsistencies.

Background:
The issue manifests when two DDL jobs are processed:

• Job 60: ALTER TABLE a ADD COLUMN x, y, z with CommitTs 400
• Job 62: ALTER TABLE b ADD COLUMN y with CommitTs 300

In this scenario, Job 62 is processed first due to its lower CommitTs. The ddlJobPuller then updates its SchemaVersion to that of Job 62. When Job 60 is subsequently processed, the current logic discards it because its SchemaVersion is deemed older, even though its CommitTs is higher.

This issue occurs because the SchemaVersion increment and job metadata write to TiKV are separate transactions. During a TiDB owner change, different instances might write these transactions without synchronization, leading to potential out-of-order CommitTs relative to SchemaVersion.

Solution:
To resolve this, the check on SchemaVersion is removed, ensuring that only the CommitTs is verified:

Updated Code:

if job.BinlogInfo.FinishedTS <= p.getResolvedTs() {
    log.Info("ddl job finishedTs less than puller resolvedTs," +
        "discard the ddl job",
        zap.String("namespace", p.changefeedID.Namespace),
        zap.String("changefeed", p.changefeedID.ID),
        zap.String("schema", job.SchemaName),
        zap.String("table", job.TableName),
        zap.Uint64("startTs", job.StartTS),
        zap.Uint64("finishedTs", job.BinlogInfo.FinishedTS),
        zap.String("query", job.Query),
        zap.Uint64("pullerResolvedTs", p.getResolvedTs()))
    return true, nil
}

Reasoning:
The ResolvedTs check ensures that only DDL jobs with FinishedTS greater than the current ResolvedTs are processed. Since the ddlJobPuller receives DDLs sorted by CommitTs, any new DDL received with a FinishedTS greater than ResolvedTs must be handled, making the SchemaVersion check redundant.

Check List

Tests

• Unit test
• Integration test

Questions

Will it cause performance regression or break compatibility?

No.

Do you need to update user documentation, design documentation or monitoring documentation?

No.

Release note

Fix the issue that DDL jobs could be incorrectly discarded in TiCDC when their schema versions were not in a strictly linear order with commit timestamps due to TiDB owner changes.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (release-8.1@0c49d2a). Learn more about missing BASE report.

Additional details and impacted files

Components	Coverage Δ
cdc	61.7007% <0.0000%> (?)
dm	51.0844% <0.0000%> (?)
engine	63.4585% <0.0000%> (?)

Flag	Coverage Δ
unit	57.3872% <100.0000%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             release-8.1     #11759   +/-   ##
================================================
  Coverage               ?   57.3872%           
================================================
  Files                  ?        854           
  Lines                  ?     126171           
  Branches               ?          0           
================================================
  Hits                   ?      72406           
  Misses                 ?      48368           
  Partials               ?       5397           

[wlwilliamx]

/cc @sdojjy @asddongmen

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lidezhu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [lidezhu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[wlwilliamx]

/test cdc-integration-mysql-test

[wlwilliamx]

/test verify

[wlwilliamx]

/test verify