rewrite kafka connector

[ekneg54]

this PR rewrites the confluent kafka input and output processors with focus on the input side.
please take a look at the CHANGELOG.md for changes.

[codecov-commenter]

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (9458a24) 91.97% compared to head (6a036bd) 92.01%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #444      +/-   ##
==========================================
+ Coverage   91.97%   92.01%   +0.03%     
==========================================
  Files         132      132              
  Lines        9389     9476      +87     
==========================================
+ Hits         8636     8719      +83     
- Misses        753      757       +4     

Files	Coverage Δ
logprep/abc/component.py	100.00% <100.00%> (ø)
logprep/abc/input.py	99.30% <100.00%> (ø)
logprep/connector/confluent_kafka/output.py	100.00% <100.00%> (ø)
logprep/connector/elasticsearch/output.py	98.24% <100.00%> (ø)
logprep/framework/pipeline.py	97.89% <ø> (-0.04%)	⬇️
logprep/util/prometheus_exporter.py	95.45% <100.00%> (+0.10%)	⬆️
logprep/util/validators.py	96.36% <66.66%> (-3.64%)	⬇️
logprep/connector/confluent_kafka/input.py	92.59% <90.62%> (-0.82%)	⬇️

... and 3 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ekneg54]

rebased from main.

@ppcad now it`s ready to rumble 👍

[ppcad]

Whenever group.id is missing from the kafka_config the error message is Required option is missing: input instead of telling what is missing.

Whenever bootstrap.servers is missing from the kafka_config Logprep crashes with a stacktrace.

group: "cgroup" in the confluent_kafka input example should be updated to group.id: "cgroup".

I have ran a couple of tests to compare it with the old version. I used the same config parameters. I activated only the labeler with no rules. In that case, the new version was about 25% slower than the old version. Do you have an idea why that could be?

[ekneg54]

it is possible, that the statistics.interval.ms default was too low. It was set to 1s. I increased the default to 30 seconds.

[ekneg54]

I added a validation for mandatory keys in kafka_config object. This prevents the Exceptions during start and logs the missing keys.

[ppcad]

I added a validation for mandatory keys in kafka_config object. This prevents the Exceptions during start and logs the missing keys.

Thanks for the changes. This solves the problem.

[ppcad]

it is possible, that the statistics.interval.ms default was too low. It was set to 1s. I increased the default to 30 seconds.

This does not seem to be the reason for the performance loss.

It seems that the new version is faster than the old one with 1 process, but slower once I increase the amount up to the partition count.