ddl puller(ticdc): Fix memory retention issue in ddlPullerImpl's PopFrontDDL method

[wlwilliamx]

What problem does this PR solve?

Issue Number: close #11793

What is changed and how it works?

Problem

In the PopFrontDDL method of ddlPullerImpl, the operation h.pendingDDLJobs = h.pendingDDLJobs[1:] creates a new slice by slicing the existing one. However, this leaves the first element (pendingDDLJobs[0]) still referenced in the underlying array, potentially causing memory retention issues.

If the removed element (a Job object) occupies significant memory and is no longer needed, this could lead to increased memory usage since it cannot be garbage collected.

Solution

The fix ensures that the reference to the removed element is explicitly set to nil before updating the slice. This helps the garbage collector to reclaim memory associated with the removed element.

Code Changes

• Explicitly set h.pendingDDLJobs[0] = nil before updating the slice in PopFrontDDL.

Updated code snippet:

func (h *ddlPullerImpl) PopFrontDDL() (uint64, *timodel.Job) {
    h.mu.Lock()
    defer h.mu.Unlock()
    if len(h.pendingDDLJobs) == 0 {
        return atomic.LoadUint64(&h.resolvedTS), nil
    }
    job := h.pendingDDLJobs[0]
    h.pendingDDLJobs[0] = nil // Explicitly clear reference to allow GC
    h.pendingDDLJobs = h.pendingDDLJobs[1:]
    return job.BinlogInfo.FinishedTS, job
}

Check List

Tests

• Unit test
• Integration test

Questions

Will it cause performance regression or break compatibility?

No.

Do you need to update user documentation, design documentation or monitoring documentation?

No.

Release note

Fix a memory retention issue in TiCDC’s DDL job processing to enhance memory efficiency in long-running tasks.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign gmhdbjd for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[wlwilliamx]

/test cdc-integration-kafka-test

[wlwilliamx]

/test cdc-integration-kafka-test

[wlwilliamx]

/test cdc-integration-pulsar-test

[wlwilliamx]

CC @CharlesCheung96 @asddongmen

[wlwilliamx]

/cc @CharlesCheung96 @asddongmen