Releases: streamdal/plumber
Releases · streamdal/plumber
Redis relay
This release adds support for relaying Redis pubsub channels to Batch.
To run plumber relay with redis in Docker:
docker run -d --name plumber-redis -p 8080:8080 \
-e PLUMBER_RELAY_REDIS_ADDRESS=localhost:6379 \
-e PLUMBER_RELAY_REDIS_CHANNELS=channel1,channel2 \
-e PLUMBER_RELAY_TYPE=redis \
-e PLUMBER_RELAY_TOKEN=$YOUR-BATCHSH-TOKEN-HERE \
batchcorp/plumber
Dockerhub: docker pull batchcorp/plumber:a934947
Relay mode fixes and cli flag improvements
Breaking changes:
- remove
--output-type
for reading; remove--input-type
for writing. Both are unnecessary and the intent can be gathered from other flags.
Features:
- Kafka: Ability to specify an offset to begin reading from
- GCP: Environment variable
PUBSUB_EMULATOR_HOST
can now be specified to connect to a pubsub emulator instead of GCP - Retries for GRPC calls in relay mode
Bug fixes:
--queue-declare
now actually declares the queue for rabbit- Increasing max record size for relay payloads
Batching for all relays, stats, improved kafka
This is a fairly large (and awesome) release.
In this release:
- New
--stats
flag (andPLUMBER_STATS
env var) which will cause plumber to periodically dump consume/produce speed stats. - Lots of kafka improvements to improve read and relay throughput
- Several new env vars that can be tweaked to further improve throughput and/or reliability
- Batching support for relay
- Pretty config printing - all relay modes will now print out the config they're using on console
plumber
now has a lot of options - some messaging platforms (such as kafka) have 20+ flags/options - it is nice to be able to see what options are actually being used at runtime.
Enjoy!
Fixing quiet flag
v0.14.1 Fixing quiet flag
Support for relaying GCP PubSub messages to Batch collections
v0.14.0 Support for GCP PubSub relaying
Support for Azure relay, improved Kafka read and relay
- Support for relaying Azure Service Bus to Batch.sh
- Improved Kafka read and relay speeds; works with Confluent cloud!
- Kafka read and relay is highly configurable for improved throughput
- Improved docs (including env var docs!)
Updating GRPC token name
v0.12.1 Modifying grpc token name
Support for NATS and Redis PubSub
Merge pull request #65 from batchcorp/blinktag/redis_pubsub Support for Redis PubSub
Support for Azure Service Bus and AWS SNS
Merge pull request #61 from batchcorp/blinktag/aws-sns Write support for AWS-SNS
Kafka relay support
v0.10.0 Kafka relay support