Improve catalog information synchronisation with GraphQL

[ThomasCAI-mlv]

1. Context

• Currently, we have the possibility to add tags and description to a topic and ns4kafka synchronizes these information with Confluent Cloud catalog information (tags & description). This synchronisation is performed with the Stream Catalog API (Stream Catalog documentation).

• However, this API is not suited for the synchronisation because of the 500 topics limit per call, and the fact that there is no filter allowing to query only topics with a description (though it is possible to query only topics with at least one tag). These drawbacks lead to having to query all the cluster topics 500 per 500 in order to perform the synchronization, which is quite bad for performance.

2. Proposed implementation
This PR adds GraphQL API calls to query the topic lists with their tags and description. This API overcomes the two previous problems: it is possible to query the list of topics with a description, and query the list of topics with at least one tag, without pagination.

This API still has drawbacks:

• it is only usable with the advanced Stream Governance Package on Confluent Cloud, which is the non-free package at 1$/hour: so we first try to use GraphQL API, and if it fails, we use the old Stream Catalog API to get the topics.
• In order to query topics with at least one tag, all the existing tags need to be filled in the GraphQL query, so another API call to Stream Catalog shall be made to get the tags list.
• I didn't find a way to query the list topics with a description OR with at least one tag. I had to split the call: so one GraphQL API call to get the topics with a description, another one to get the topics with at least one tag. If you want to challenge it, you can try the GraphQL API (doc: https://docs.confluent.io/cloud/current/stream-governance/graphql-apis.html#definition-kafka_topic)

3. Other

• I added a sync-catalog property which must be set to true in order to allow the synchronization of catalog information in Confluent Cloud. It is set at false by default.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
80.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud