Skip to content
This repository has been archived by the owner on Jan 8, 2024. It is now read-only.

Enable Kafka batch processing #60

Open
wagmarcel opened this issue Mar 7, 2021 · 0 comments
Open

Enable Kafka batch processing #60

wagmarcel opened this issue Mar 7, 2021 · 0 comments
Assignees
@wagmarcel

Comments

@wagmarcel
Copy link
Member

Kafka batch processing is not enabled in the current backend.
Problem:
This is reducing the performance of the backend since communicating with KairosDB creates a 3-5ms latency. Since Kafka mandates sequential processing this was reducing the backend to ~200Hz processing.
Enabling batching is removing this bottleneck since this allows to processing up to 1000 samples per KairosDB call.
The implementaion will be similar to https://memorynotfound.com/spring-kafka-batch-listener-example/
@wagmarcel wagmarcel self-assigned this Mar 8, 2021
wagmarcel added a commit to wagmarcel/oisp-backend that referenced this issue Mar 8, 2021
@wagmarcel
Kafka-Consumer: Enabled batch processing of incoming metrics. The Max…
0afcb9d 
…PayloadSize paramter has to be added to the KAFKA_CONFIG

Closing issue Open-IoT-Service-Platform#60.

Signed-off-by: Marcel Wagner <[email protected]>
@wagmarcel wagmarcel mentioned this issue Mar 8, 2021
Merged
oguzcankirmemis pushed a commit that referenced this issue Mar 8, 2021
@wagmarcel @oguzcankirmemis
Kafka-Consumer: Enabled batch processing of incoming metrics. The Max…
77cd693 
…PayloadSize paramter has to be added to the KAFKA_CONFIG

Closing issue #60.

Signed-off-by: Marcel Wagner <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@wagmarcel wagmarcel

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@wagmarcel