[Feature]: Implement persistent storage and caching layer for cluster metadata

[vladoatanasov]

Contact Details

No response

Description

The broker needs to report metadata about the Kafka cluster via the Metadata API. The information will be accessed regularly and updated occasionally. The proposed design is to use a flat file, like json, protobuf, etc, as a persistent layer and to cache the data in memory. When the process starts, it will load the flat file in memory and will occasionally flush to disk.
When the process detects a graceful termination signal, it should flush the metadata to disk and then exit.

See Kafka documentation for required metadata https://kafka.apache.org/protocol.html. Currently OpenTalaria supports Metadata API Version 8.

Code of Conduct

• I agree to follow this project's Code of Conduct

[vladoatanasov]

This issue is a prerequisite