Kafka Connect Sporadic Commit Delay #11796
Labels
bug
Something isn't working
Comments
|
Also added some debug logging inside the class @Override
public void save(Collection<SinkRecord> sinkRecords) {
if (sinkRecords != null && !sinkRecords.isEmpty()) {
worker.save(sinkRecords);
LOG.info(
"Saving {} records. Last offset {}",
sinkRecords.size(),
sinkRecords.stream()
.reduce((first, second) -> second)
.map(SinkRecord::kafkaOffset)
.orElse(-1L));
}
processControlEvents();
}Which produced the following logs for the above problem: So the message is read from the source kafka topic when it should be. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Apache Iceberg version
1.7.1 (latest release)
Query engine
Kafka Connect
Please describe the bug 🐞
We experience unexplainable large delays on the commits from time to time.
The setup is using the kafka-connect connector. Tested both with the latest release (1.7.1) and with main. (1.8.0-SNAPSHOT)
These are the important settings from the connector config:
So we commit every 10 seconds and timeout after 20 seconds. This work well approximately 75% of the time. But occasionally (every 4-5 commits or so) we experience a delay that cannot be explained.
When this happens, a message is received. The connector goes through the commit logic and times out as if there is no message received. This happens a couple of times until it is finally committed (no other messages are received in this time period.
Here's an example:
These messages are from the input topic (The timestamp is in GMT+1)
This is the kafka offset and consumer offset for the source topic, showing the delay in offset commit (matches the iceberg commit)
This is the same for the iceberg connector control topic, which does not have any delays (so that's not the issue)
At 07:40:15 two messages are received. One of them is committed immediately after, but the other one is not. No more messages are received and the commit loop goes through several cyles (4 commits and timeouts), before it is actually committed at 07:41:44.
The logs can be seen below, as well as a filtered part of the log, which only shows when a commit interval starts, times out and completes. Log files are in GMT
Any idea what could be causing this? Another thing I notice, is that when no messages have been received since the last commit, it still has to wait for the timeout. This does not seem like the most feasible solution. Wouldn't it be better to track that and then simply return?
Filtered log:
Complete log:
Willingness to contribute
The text was updated successfully, but these errors were encountered: