Improve serving of NiFi 2 metrics

[NickLarsenNZ]

Part of #809.

Tip

Prometheus/OpenMetrics endpoints are an operational concern and should not be treated as part of the application's API.

Current state

The metrics endpoint for NiFi 2 cluster members is part of the API server/endpoints, and therefore inherits the authentication/authorization configuration of the API.

It is unusual to use Bearer auth for serving metrics, not to mention the current Oauth2 implementation restricts tokens to each cluster member they were obtained from (see: NIFI-7246). This makes regular metrics scraping rather difficult to near impossible.

Result from Refinement

NiFi 2 metrics, in the current state, can be scraped using mTLS. The client scraping the metrics needs a client certificate, where NiFi trusts the signing CA, which can be, for example, generated by using the secret-operator mechanism.

Next steps

• Document mTLS configuration in NiFi monitoring docs
• Update monitoring stack to use mTLS for scraping NiFi metrics chore: update monitoring stack for scraping nifi metrics demos#260
• @sbernauer do some research on why we should add prometheus.io/scrape, prometheus.io/protocol, prometheus.io/port, prometheus.io/path to the -metrics Service over having a dedicated ServiceMonitor per product. Can we also do relabeling there?
• Adopt smoke test v2 such that it tests metrics HTTPS is callable and returns some data/metrics. The test call against the -metrics service (and not some other, such as -headless) chore: add metrics test for NiFi 2.x #823

Future state (Outdated after refinement, kept for reference if needed later)

Metrics scraping should be as simple as it is for any other app exposing endpoints.

Implement common patterns for exposing metrics:

• Unauthenticated access, or authorized access using simple authentication mechanisms (such as Basic auth, and...
• Expose metrics on a separate HTTP port (so that it's exposure is independent of an API), and/or unauthenticated or using basic auth (and not complex authentication mechanisms).

Important acceptance criteria:

• All Nifi versions (1&2) expose a -metrics service, pointing to the according port (either jmx-metrics-exporter or Nifi HTTPS). This is according to 3.1 from https://github.com/stackabletech/decisions/issues/51 => Add metrics service for nifi v2 #819
• If a user has an already running NiFi 2 stacklet, a stackablectl stack install monitoring automatically starts scraping all NiFi Pods (e.g. the monitoring stack Prometheus uses secret-op for mTLS). This was the state with NiFi 1 and was lost with NiFi 2