Feat: Add Kafka integration for Parseable server #936 .

[hippalus]

Fixes #936

Description

This pull request implements the Kafka connector for Parseable by introducing better stream management, modularizing the code, and integrating Prometheus metrics. It also adds groundwork for dynamic configurations.

How It Works

Partition Management

• The connector dynamically manages streams for each partition using StreamState.
• Each Kafka message is routed to the corresponding PartitionStreamReceiver for processing.

Rebalance Handling

• On partition assignment:
  • New streams are created for untracked partitions.
• On partition revocation:
  • Streams for revoked partitions are terminated and cleaned up.

Stream Processing

• The StreamWorker processes messages in batches, ensuring backpressure handling with configurable buffer_size and buffer_timeout.

Metrics

• Exposed metrics via Prometheus include: Partition lag, consumer throughput, etc provided by rd-kafka statistics.

TODO

1. Integration Tests:
   • Add tests using test containers to simulate Kafka.
2. Docker Improvements:
   • Add build dependencies to the Dockerfile for rd-kafka
   • Add docker-compose to test distributed env.
3. Metrics Collector:
   • Complete KafkaConsumerMetricsCollector

---

This PR has:

• been tested to ensure log ingestion and log query works.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added documentation for new or modified features or behaviors.

[github-actions]

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

[hippalus]

I have read the CLA Document and I hereby sign the CLA

[de-sh]

I wouldn't want to serialize this back into a file, but I would love to extract it directly from the env vars using clap, can we do that? refer: S3Config

[de-sh]

Same for all other config types that have to be extracted from env vars

[nitisht]

Unfortunately we just completed this feature in #1021 @hippalus

[nitisht]

Would you want to contribute something else?

[nitisht]

I see useful stuff like backpressure handling which we can augment to the changes added in #1021

[hippalus]

Unfortunately we just completed this feature in #1021 @hippalus

@nitisht As someone with production-ready experience working with Kafka on both the client and broker sides, I was surprised to see the previous PR merged, as it overlaps with the work I was pursuing. While I may have missed the chance to open a draft PR earlier due to time constraints, I’d still love the opportunity to enhance and refine this feature further.

My goal is to contribute to a more robust and efficient solution. I believe there’s significant room for improvement, particularly in areas like backpressure, error handling, retrying, parallel processing, consumer rebalance, protection from data loss, etc. I’d be happy to collaborate on the existing implementation to make it more solid. Let me know how I can best support these efforts.

[nitisht]

My goal is to contribute to a more robust and efficient solution. I believe there’s significant room for improvement, particularly in areas like backpressure, error handling, retrying, parallel processing, consumer rebalance, protection from data loss, etc. I’d be happy to collaborate on the existing implementation to make it more solid. Let me know how I can best support these efforts.

This is great to hear @hippalus . We're more than happy to support you in this. Would you change this PR to add these features instead?

[hippalus]

My goal is to contribute to a more robust and efficient solution. I believe there’s significant room for improvement, particularly in areas like backpressure, error handling, retrying, parallel processing, consumer rebalance, protection from data loss, etc. I’d be happy to collaborate on the existing implementation to make it more solid. Let me know how I can best support these efforts.

This is great to hear @hippalus . We're more than happy to support you in this. Would you change this PR to add these features instead?

Of course! @nitisht As I mentioned in the PR description, I will implement the current TODOs and PR comments that @de-sh made. Then let's evaluate it.