[FEATURE] Add getTopics API to StreamMetadataProvider for Topic Enumeration in Streaming Connectors

[suvodeep-pyne]

Introduces a new getTopics() API to the StreamMetadataProvider interface, enabling the ability to enumerate topics for streaming connectors in Apache Pinot.

Key Changes

1. New API in StreamMetadataProvider:

• Added getTopics() method to fetch the list of topics
• Introduced TopicMetadata as a representation of topic metadata, currently including the topic name but extensible for future metadata fields.

2. Implementation for Kafka Connectors (2.0 & 3.0)

• Implemented the listTopics method in KafkaStreamMetadataProvider for both Kafka 2.0 and Kafka 3.0 plugins.
• Utilized Kafka’s Consumer#listTopics() API to retrieve the topic list and mapped the results to the TopicMetadata structure.

3. Unit Test Coverage:

• Added unit test in KafkaPartitionLevelConsumerTest to validate the listTopics functionality.

4. Backward Compatibility:

• Default implementation of listTopics in StreamMetadataProvider throws UnsupportedOperationException to maintain backward compatibility with other connectors that do not override the method.

Future Work:

• Extend TopicMetadata to include additional metadata like partition count, replication factor, or topic configurations.
• Add similar support for other streaming connectors in Apache Pinot.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 39.13043% with 14 lines in your changes missing coverage. Please review.

Project coverage is 64.04%. Comparing base (59551e4) to head (34952e0).
Report is 1464 commits behind head on master.

Files with missing lines	Patch %	Lines
...in/stream/kafka20/KafkaStreamMetadataProvider.java	0.00%	11 Missing ⚠️
...in/stream/kafka30/KafkaStreamMetadataProvider.java	81.81%	1 Missing and 1 partial ⚠️
...pache/pinot/spi/stream/StreamMetadataProvider.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #14654      +/-   ##
============================================
+ Coverage     61.75%   64.04%   +2.28%     
- Complexity      207     1602    +1395     
============================================
  Files          2436     2700     +264     
  Lines        133233   148726   +15493     
  Branches      20636    22783    +2147     
============================================
+ Hits          82274    95245   +12971     
- Misses        44911    46494    +1583     
- Partials       6048     6987     +939     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (+99.99%)	⬆️
integration	100.00% <ø> (+99.99%)	⬆️
integration1	100.00% <ø> (+99.99%)	⬆️
integration2	0.00% <ø> (ø)
java-11	63.98% <39.13%> (+2.27%)	⬆️
java-21	63.92% <39.13%> (+2.30%)	⬆️
skip-bytebuffers-false	64.00% <39.13%> (+2.25%)	⬆️
skip-bytebuffers-true	63.90% <39.13%> (+36.17%)	⬆️
temurin	64.04% <39.13%> (+2.28%)	⬆️
unittests	64.03% <39.13%> (+2.28%)	⬆️
unittests1	56.30% <0.00%> (+9.41%)	⬆️
unittests2	34.48% <39.13%> (+6.75%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.