Add cluster info and stats API

[lispyclouds]

Now that we are using the management plugin enabled RabbitMQ, check out its API and see what could be used to make a useful cluster stats endpoint. This could be used by the UI.

[aldosolorzano]

@lispyclouds what is the current state of bob stats? Can you provide a little more context on what you have in mind for this issue?

[lispyclouds]

Essentially the intention here is to come up with a comprehensive map of what the state the cluster is in currently. Could be similar to what Elasticsearch provides for instance. Gather as much useful info as possible like:

• Runners connected to the queue and their metadata
• APIServers and their metadata
• The declared queues and exchanges in RabbitMQ
• Versions of things like RabbitMQ, Postgres, Podman etc
• something more useful...

Hence I thought of using the REST API of RabbitMQ as a source of a lot of this. This needs a bit of discovery too as I am not fully sure what all can be obtained and shown as well. 😄

Primary intention is for cluster debugging needs.

[lispyclouds]

Maybe the stats can be avoided now that the metrics API is there.

[aldosolorzano]

Maybe the stats can be avoided now that the metrics API is there.

Partially, this metrics provide informations about the output of the behaviour of bob which is to process pipelines. If you see a lot of pipeline failures for example, is there a way to ask for a health endpoint? If not, then it would be hard to know why pipelines are failing right? as you said, having a landscape of the cluster can be very handy.

I looking into what would be more useful for bob, there are a couple of issues but I don't know what to prioritize.

[lispyclouds]

is there a way to ask for a health endpoint?

We do have a health check endpoint. In addition to that Bob runs a periodic heartbeat that can warn about health issues too.

This is more about debugging or verifying cluster state. For instance nodes not joining the cluster, jobs not being picked up, wrong routing etc.