Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace sync Kafka consumers with confluent_kafka one #767

Merged
merged 1 commit into from
Dec 14, 2023

Conversation

matyaskuti
Copy link
Contributor

@matyaskuti matyaskuti commented Nov 30, 2023

About this change - What it does

Why this way

  • Continuing the previous patterns towards removing kafka-python (errors and KarapaceKafkaClient still remain)
  • Keeping confluent-kafka references in karapace.kafka.* this was agreed to be simplified and not be a goal after all

@matyaskuti matyaskuti force-pushed the matyaskuti/confluent_kafka_sync_consumer branch 18 times, most recently from 90714d8 to 26ea01d Compare December 4, 2023 12:45
@matyaskuti matyaskuti changed the title Confluent-kafka consumer Replace sync Kafka consumers with confluent_kafka one Dec 4, 2023
@matyaskuti matyaskuti force-pushed the matyaskuti/confluent_kafka_sync_consumer branch 4 times, most recently from 78ce72e to 4740bab Compare December 4, 2023 13:42
@matyaskuti matyaskuti marked this pull request as ready for review December 4, 2023 14:32
@matyaskuti matyaskuti requested review from a team as code owners December 4, 2023 14:32
@matyaskuti matyaskuti force-pushed the matyaskuti/confluent_kafka_sync_consumer branch 3 times, most recently from 981fa69 to 0039127 Compare December 6, 2023 15:05
Copy link
Contributor

@eliax1996 eliax1996 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost convinced its fine, just want to discuss a little bit on the doubt I've asked 😁, when @aiven-anton take a look I'm fine with merging

karapace/schema_reader.py Outdated Show resolved Hide resolved
karapace/schema_reader.py Outdated Show resolved Hide resolved
@matyaskuti matyaskuti force-pushed the matyaskuti/confluent_kafka_sync_consumer branch 2 times, most recently from ea02eb7 to 7a31284 Compare December 12, 2023 11:02
Copy link
Contributor

@aiven-anton aiven-anton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partial review so far, leaving some comments.

karapace/backup/api.py Show resolved Hide resolved
karapace/backup/backends/v3/backend.py Show resolved Hide resolved
karapace/backup/backends/v3/backend.py Show resolved Hide resolved
karapace/kafka/consumer.py Show resolved Hide resolved
karapace/kafka/types.py Outdated Show resolved Hide resolved
karapace/kafka/types.py Outdated Show resolved Hide resolved
karapace/schema_reader.py Show resolved Hide resolved
@matyaskuti matyaskuti force-pushed the matyaskuti/confluent_kafka_sync_consumer branch from 7a31284 to d3e000f Compare December 12, 2023 12:39
stubs/confluent_kafka/cimpl.pyi Outdated Show resolved Hide resolved
stubs/confluent_kafka/cimpl.pyi Show resolved Hide resolved
karapace/backup/api.py Show resolved Hide resolved
@matyaskuti matyaskuti force-pushed the matyaskuti/confluent_kafka_sync_consumer branch from d3e000f to 770ff3d Compare December 12, 2023 14:32
@matyaskuti matyaskuti force-pushed the matyaskuti/confluent_kafka_sync_consumer branch 6 times, most recently from 640a9d0 to fda504b Compare December 13, 2023 10:52
Copy link
Contributor

@aiven-anton aiven-anton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, fully reviewed now, I think looks good as far as I can tell. The outstanding thing is the questions about LEO/HWM off by one ... Can we find some way to build higher confidence in those changes?

tests/unit/backup/backends/v3/test_backend.py Outdated Show resolved Hide resolved
tests/utils.py Outdated Show resolved Hide resolved
karapace/schema_reader.py Show resolved Hide resolved
@matyaskuti matyaskuti force-pushed the matyaskuti/confluent_kafka_sync_consumer branch 2 times, most recently from 8e0f0d4 to e91aafa Compare December 13, 2023 12:51
start_offset: int = consumer.beginning_offsets([topic_partition])[topic_partition]
end_offset: int = consumer.end_offsets([topic_partition])[topic_partition]
) -> Iterator[Message]:
start_offset, end_offset = consumer.get_watermark_offsets(topic_partition)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had another doubt this was correct due to mix of terminology, but have now verified it's the same underlying Protocol message involved (ListOffsets):

So looks correct 👍

aiven-anton
aiven-anton previously approved these changes Dec 13, 2023
Copy link
Contributor

@aiven-anton aiven-anton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, solid work! 👍

This change replaces all synchronous Kafka consumers (from the
kafka-python library) with a new implementation based on
confluent-kafka-python's `Consumer`, keeping the same interface as much
as possible.

The PyTest timeout is raised from 60s to 90s to accomodate for the
default poll timeout for backups consumers (otherwise the tests would
time out while still waiting for messages to arrive).o

Since the `conluent_kafka.Consumer` implementation does not allow for
consumers to be without a group ID, if the new `KafkaConsumer` client is
not given one, we'll generate one on the fly to mimic a groupless
behaviour.

Resources:
* confluent-kafka-python documentation:
  https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#
* librdkafka configuration documentation:
  https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md
@matyaskuti matyaskuti force-pushed the matyaskuti/confluent_kafka_sync_consumer branch from e91aafa to dd0ec0f Compare December 14, 2023 09:19
@eliax1996 eliax1996 merged commit 376a6b6 into main Dec 14, 2023
8 checks passed
@eliax1996 eliax1996 deleted the matyaskuti/confluent_kafka_sync_consumer branch December 14, 2023 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants