[source-mysql] missing data during replicate big table, from changed cdc initial position after multi attempts

[amelia-ay]

Connector Name

source-mysql

Connector Version

3.7.3

What step the error happened?

During the sync

Relevant information

we used airbyte to sync a big database which contains 200+ tables and over TB's data from mysql to databricks. After initial sync, we found missing a few data, and refresh data changed nothing. we checked info in log, and updated the '_ab_cdc_log_file' in final table and refresh data again for test, it worked.
so, maybe this is a bug?

Relevant log output

2024-11-22 10:05:30 source > INFO main i.a.i.s.m.c.MySqlCdcTargetPosition(reachedTargetPosition):86 Signalling close because record's binlog file : mysql-bin-changelog.912510 , position : 92861302 is after target file : mysql-binlog.912510 , target position : 46293735
2024-11-22 10:05:31 source > INFO  i.d.c.b.BinlogStreamingChangeEventSource$ReaderThreadLifecycleListener(lambda$onDisconnect$0):1206 Stopped reading binlog after 0 events, last recorded offset: {ts_sec=1732269172, file=mysql-binlog.912511, pos=11846779, server_id=1448378605, event=607}

Contribute

• Yes, I want to contribute

[amelia-ay]

we also notice that " CDC events queue poll(): returned a heartbeat event: progressing to FileName: mysql-binlog.\d+, Position : \d+.", while we have not found yet where it came from ?

update: this error came from destination, ignore it.

[amelia-ay]

we found state changed in one attempt of snapshot, next, we updated source-mysql to [3.9.0-rc.21], that's great , it worked correcttly.