Kafka integration

[hippalus]

Fixes parseablehq#936

Description

This pull request implements the Kafka connector for Parseable by introducing better stream management, modularizing the code, and integrating Prometheus metrics. It also adds groundwork for dynamic configurations.

How It Works

Partition Management

• The connector dynamically manages streams for each partition using StreamState.
• Each Kafka message is routed to the corresponding PartitionStreamReceiver for processing.

Rebalance Handling

• On partition assignment:
  • New streams are created for untracked partitions.
• On partition revocation:
  • Streams for revoked partitions are terminated and cleaned up.

Stream Processing

• The StreamWorker processes messages in batches, ensuring backpressure handling with configurable buffer_size and buffer_timeout.

Metrics

• Exposed metrics via Prometheus include: Partition lag, consumer throughput, etc provided by rd-kafka statistics.

TODO

1. Integration Tests:
   • Add tests using test containers to simulate Kafka.
2. Docker Improvements:
   • Add build dependencies to the Dockerfile for rd-kafka
   • Add docker-compose to test distributed env.
3. Metrics Collector:
   • Complete KafkaConsumerMetricsCollector

---

This PR has:

• been tested to ensure log ingestion and log query works.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added documentation for new or modified features or behaviors.