[Recovery] Metrics

[lovromazgon]

Add metrics regarding pipeline recovery:

• conduit_pipeline_status should be a gauge that's set every time a pipeline is created and every time its status changes. The gauge value should always be set to 1. When the pipeline gets deleted, the gauge with the corresponding pipeline_name should be deleted. It should contain two labels:
  • status holds the status of the pipeline (e.g. running, stopped, degraded, recovering)
  • pipeline_name holds the pipeline name (note that we use the pipeline name, because the API generates a random pipeline ID, the user is only in charge of the pipeline name, this might change in the future)
• pipeline_recovering_count should be a counter which is created and set to 0 when a pipeline is created. The counter should increase every time the recovery mechanism is triggered (before the backoff retry starts waiting). It should contain one label:
  • pipeline_name holds the pipeline name