A CLI for consuming protobuf messages over Kakfa.
Serializing Kafka messages to protobuf has many benefits, but it can make debugging and peeking at messages can be difficult. Milena aims to make it easy to produce and consume protobuf messages over Kafka using JSON.
- Dynamically decodes any protobuf message type in the provided file descriptors
- Outputs messages as newline delimited JSON with an envelope containing common metadata
- Produces protobuf messages from JSON
- Deals with pesky magic bytes
-T
- Server mode that proxies Kafka/protobuf messages to HTTP/JSON
- Download the latest release archive for your platform from the latest release
- Extract the archive
- If you're on MacOS, remove the binary from quarantine
cargo install --git https://github.com/VanceLongwill/milena
docker pull vancelongwill/milena:latest
Usage: milena [OPTIONS] <COMMAND>
Commands:
consume Consume & decode protobuf messages to JSON
produce Encode JSON messages as protobuf and produce them to a topic
decode Decode an arbitrary protobuf message
encode Encode an arbitrary protobuf message
serve Start a HTTP server that can proxy kafka/protobuf to http/json
help Print this message or the help of the given subcommand(s)
Options:
-F, --config <FILE>
Sets a custom config file
-f, --file-descriptors <FILE_DESCRIPTORS>
The path to the protobuf file descriptors [default: ./descriptors.binpb]
-X, --rdkafka-options <RDKAFKA_OPTIONS>
A catchall for specifying additional librdkafka options
-v, --verbose...
Enable verbose logging, can be repeated for more verbosity up to 5 times
-h, --help
Print help
-V, --version
Print version
Based on the docker example.
milena produce \
-X bootstrap.servers=localhost:9092 \
--topic sometopic \
--file-descriptors descriptors.binpb \
--message-name example.v1.UserUpdated \
--key 123 \
--header 'message-name=example.v1.UserUpdated' \
--data '{"name": "John Smith", "age": 40}'
milena consume \
-X bootstrap.servers=localhost:9092 \
-t sometopic \
-f descriptors.binpb \
-H "message-name" \
-o=beginning
milena
can also proxy kafka/protobuf to HTTP/json using the serve
command. Each request starts a new consumer stream.
This provides a number of advantages:
- Developers can read protobuf encoded kakfa messages without having to install another tool (just curl it)
- Consume only access to Kafka
- Credentials need only reside in a single server (this could live within your network for devs to access).
- Develops don't have to compile protobuf file descriptors. It's all just done once and can be automated via CI/CD.
milena serve -p 29999 \
-X bootstrap.servers=localhost:9092 \
-f descriptors.binpb
The JSON request body can contain any option provided to the
consume
command. Seemilena consume -h
for more details.
curl localhost:29999 -H 'Content-type: application/json' -d '{
"topic": "sometopic",
"message_name_from_header": "message-name",
"exit_on_last": true
}'
(while also simultaneously piping to jq
for pretty printing)
curl localhost:29999 -H 'Content-type: application/json' -d '{
"topic": "sometopic",
"message_name_from_header": "message-name",
"offsets": ["end"]
}' -N -s | jq -rc
-N
prevents curl buffering the output so we can can pipe it to jq as soon as we receive something.-s
hides other output from curl
See milena encode -h
or milena decode -h
for more options.
milena encode -N example.v1.UserUpdated '{"name": "alice", "age": 90}' |\
milena decode -N example.v1.UserUpdated
{"name":"alice","age":90}%
Set the verbose flag up to 5 times -vvvvv
to get increasingly more
detailed logging.
Ensure that [email protected]
installed with brew & set the following environment variables.
export OPENSSL_ROOT_DIR=$(brew --prefix [email protected])
export OPENSSL_LIBRARIES="${OPENSSL_ROOT_DIR}/lib"
Protobuf file descriptors can be generated using the buf CLI or with protoc.
buf build ./ -o descriptors.binpb
protoc example.proto -o protocdescriptors.binpb