Rework the Kafka Guide #124
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
matkuliak
force-pushed
the
mm/kafka-connect-update
branch
from
66b387f
to
09c21ee
Compare
amotl
approved these changes
Sep 17, 2024
Comment on lines
-149
to
-222
| For this example, this Python script will simulate the creation | ||
| of random sensor data and push it into the ``metrics`` topic: | ||
|
|
||
| .. code-block:: python | ||
|
|
||
| import time | ||
| import random | ||
|
|
||
| from confluent_kafka import avro | ||
| from confluent_kafka.avro import AvroProducer | ||
|
|
||
| # Define the Avro schema we want our produced records to conform to. | ||
| VALUE_SCHEMA_STR = """ | ||
| { | ||
| "namespace": "cratedb.metrics", | ||
| "name": "value", | ||
| "type": "record", | ||
| "fields": [ | ||
| {"name": "id", "type": "string"}, | ||
| {"name": "timestamp", "type": "float"}, | ||
| {"name": "payload", "type": { | ||
| "type": "record", | ||
| "name": "payload", | ||
| "fields": [ | ||
| {"name": "temperature", "type": "float"}, | ||
| {"name": "humidity", "type": "float"}, | ||
| {"name": "pressure", "type": "float"}, | ||
| {"name": "luminosity", "type": "float"} | ||
| ] | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| """ | ||
|
|
||
| # Load the Avro schema. | ||
| VALUE_SCHEMA = avro.loads(VALUE_SCHEMA_STR) | ||
|
|
||
| # Create an Avro producer using the defined schema, assuming that our | ||
| # Kafka servers are running at localhost:9092 and the Schema Registry | ||
| # server is running at localhost:8081. | ||
| AVRO_PRODUCER = AvroProducer( | ||
| { | ||
| "bootstrap.servers": "localhost:9092", | ||
| "schema.registry.url": "http://localhost:8081", | ||
| }, | ||
| default_value_schema=VALUE_SCHEMA, | ||
| ) | ||
|
|
||
| # Create a metric payload from a simulated sensor device. | ||
| def create_metric(): | ||
| return { | ||
| "id": "sensor-" + str(random.choice(list(range(1, 21)))), | ||
| "timestamp": int(time.time()), | ||
| "payload": { | ||
| "temperature": random.uniform(-5, 35), | ||
| "humidity": random.uniform(0, 100), | ||
| "pressure": random.uniform(1000, 1030), | ||
| "luminosity": random.uniform(0, 65000), | ||
| }, | ||
| } | ||
|
|
||
| # Create a new metric every 0.25 seconds and push it to the metrics topic. | ||
| while True: | ||
| AVRO_PRODUCER.produce(topic="metrics", value=create_metric()) | ||
| AVRO_PRODUCER.flush() | ||
| time.sleep(0.25) | ||
|
|
||
| This script can be run by installing the following dependencies and running it: | ||
|
|
||
| .. code-block:: console | ||
|
|
||
| $ pip install "confluent-kafka[avro]" "avro-python3" | ||
| $ python simulator.py |
There was a problem hiding this comment.
I am sad to see this Python snippet removed. Maybe we can bring it back in one way or another, if not within this tutorial, maybe at another spot.
There was a problem hiding this comment.
Absolutely. The current setup is pretty crude, I'm sure we can improve it with a similar script
Comment on lines
+4
to
+6
| This guide describes a dockerized procedure for integrating CrateDB with Kafka | ||
| Connect. By following these steps, you will set up a pipeline to ingest data | ||
| from Kafka topics into CrateDB seamlessly. |
Co-authored-by: Andreas Motl <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary of the changes / Why this is an improvement
Original guide wasn't working anymore. It's similar, just dockerized and updated.
Preview
https://cratedb-guide--124.org.readthedocs.build/integrate/etl/kafka-connect.html
Checklist