edX Event Bus - Kafka

Kafka implementation for Open edX event bus.

Overview

This package implements an event bus for Open edX using Kafka.

The event bus acts as a broker between services publishing events and other services that consume these events. Implementing the event bus will allow for asynchronous messaging across services which greatly improves efficiency as we don't have to wait for scheduled batch synchronizations. Additionally, since the services all speak to the event bus, they are independent of one another and can still function if one service crashes.

This package contains both the publishing code, which processes events into messages to send to the broker, and the consumer code, which polls the broker using a while True loop in order to turn messages back into event data to be emitted. The actual Kafka host will be configurable.

The goal for event-bus-kafka is to eventually have a flexible event bus that can be easily brought into other apps and repositories to produce and consume arbitrary topics. Ideally, the event bus itself will also be an abstraction behind which platform maintainers can use non-Kafka implementations (Redis, Pulsar, etc.). The documentation/ADRs may also be moved to more appropriate places as the process matures.

The repository works together with the openedx/openedx-events repository to make the fully functional event bus.

Documentation

To use this implementation of the Event Bus with openedx-events, you'll need to ensure that you include the dependency confluent_kafka[avro,schema-registry] (see ADR 5 for an explanation) and set the following Django settings:

EVENT_BUS_KAFKA_BOOTSTRAP_SERVERS: ...
EVENT_BUS_KAFKA_SCHEMA_REGISTRY_URL: ...
EVENT_BUS_TOPIC_PREFIX: ...

# On the producing side:
EVENT_BUS_PRODUCER: edx_event_bus_kafka.create_producer

# On the consuming side:
EVENT_BUS_CONSUMER: edx_event_bus_kafka.KafkaEventConsumer

Optional settings that are worth considering:

• EVENT_BUS_KAFKA_CONSUMER_CONSECUTIVE_ERRORS_LIMIT

For manual testing, see docs/how_tos/manual_testing.rst.

Django management commands:

• If you need to test event production, use the produce_event management command
• To actually run the consumer, use openedx-events' consume_events management command

OEP-52 documentation: https://open-edx-proposals.readthedocs.io/en/latest/architectural-decisions/oep-0052-arch-event-bus-architecture.html (TODO: Set up documentation)

Development Workflow

One Time Setup

# Clone the repository
git clone [email protected]:openedx/event-bus-kafka.git
cd event-bus-kafka

# Set up a virtualenv using virtualenvwrapper with the same name as the repo and activate it
mkvirtualenv -p python3.8 event-bus-kafka

Every time you develop something in this repo

# Activate the virtualenv
workon event-bus-kafka

# Grab the latest code
git checkout main
git pull

# Install/update the dev requirements
make requirements

# Run the tests and quality checks (to verify the status before you make any changes)
make validate

# Make a new branch for your changes
git checkout -b <your_github_username>/<short_description>

# Using your favorite editor, edit the code to make your change.
vim …

# Run your new tests
pytest ./path/to/new/tests

# Run all the tests and quality checks
make validate

# Commit all your changes
git commit …
git push

# Open a PR and ask for review.

Installation

This library is not intended to be a direct dependency of any service. Instead, it will likely be installed by some private mechanism. Unless the platform gains new processes for installing private dependencies, upgrades will need to be manually managed via a pin.

License

The code in this repository is licensed under the AGPL 3.0 unless otherwise noted.

Please see LICENSE.txt for details.

How To Contribute

Contributions are very welcome. Please read How To Contribute for details.

The pull request description template should be automatically applied if you are creating a pull request from GitHub. Otherwise you can find it at PULL_REQUEST_TEMPLATE.md.

The issue report template should be automatically applied if you are creating an issue on GitHub as well. Otherwise you can find it at ISSUE_TEMPLATE.md.

Reporting Security Issues

Please do not report security issues in public. Please email [email protected].

Getting Help

If you're having trouble, we have discussion forums at https://discuss.openedx.org where you can connect with others in the community.

Our real-time conversations are on Slack. You can request a Slack invitation, then join our community Slack workspace.

For more information about these options, see the Getting Help page.