Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consolidate KafkaEventSubscriber and KafkaOutboxSubscriber into the former (GSI-1405) #155

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

TheByronHimes
Copy link
Member

@TheByronHimes TheByronHimes commented Feb 18, 2025

Context:

DLQ retry topic subscription in select services

Problem description:

The services listed below run both event subscribers and outbox subscribers simultaneously.
This has not been a problem until now because the subscribed topics have been mutually exclusive.
The DLQ process introduced retry topics, of which there exists 1 per service.
Both retried outbox events and retried normal events both get funneled into the same retry topic for a service.
Both the outbox consumer and event consumer try to subscribe to the service's retry topic, but only one gets the events (barring a rebalance).

Affected Services:

  • DINS
  • NOS
  • IRS
  • DCS
  • UCS

Notes:

  1. In at least the NOS, the outbox subscriber is used for its intended purpose (cannot simply convert to event subscriber).
  2. It is not practical to convert all events in a service to one type or the other as a way to use only one type of subscriber class, since that would require a cascade of changes in other services.

Preliminary Conclusion:

Given the problem stems from the fact that there are potentially two running consumers that listen to the same topic,
there are two obvious approaches to fix the problem:

  1. Create another retry topic
  2. Consolidate the consumers

@TheByronHimes TheByronHimes changed the title Augment TranslatorConverter and deprecate KafkaOutboxSubscriber (GSI-1405) Consolidate KafkaEventSubscriber and KafkaOutboxSubscriber into the former (GSI-1405) Feb 19, 2025
@coveralls
Copy link

Pull Request Test Coverage Report for Build 13410390804

Details

  • 25 of 28 (89.29%) changed or added relevant lines in 2 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage decreased (-0.04%) to 92.148%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/hexkit/providers/akafka/provider/eventsub.py 24 27 88.89%
Totals Coverage Status
Change from base Build 13073068960: -0.04%
Covered Lines: 1995
Relevant Lines: 2165

💛 - Coveralls

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants