feat(prometheus): Support all Prometheus parameters

[jsirianni]

Description of Changes

Added support for

• Query Path Prefix
• Remote Write Host, port, and path (endpoint)
• Auth
• TLS

Testing

Prometheus

I followed our Prometheus Installation documentation when building my Prometheus test system.

Chart deployed from the branch with:

helm template \
  --namespace default \
  --values values.yaml \
  bindplane \
  charts/bindplane | kubectl apply -f - 

Port forwarding used to connect to the ui on port 3011.

kubectl port-forward pod/bindplane-0 3011:3001

Tested by ensuring topology view has active measurements.

Default Prometheus Install

config:
  username: bpuser
  password: bppass
  secret_key: 12D8FB6E-1532-4A4C-97AF-95A430BE5E6E
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
prometheus:
  enable: true
  host: prometheus-tls.c.bpcli-dev.internal
dev:
  collector:
    create: true

Basic Auth

Web.yaml config

basic_auth_users:
  admin: $2b$12$maOicLymWgsIQleRCm604ePbaaavp9cKj3bJUg0IrcVXCHB3terLa

config:
  username: bpuser
  password: bppass
  secret_key: 12D8FB6E-1532-4A4C-97AF-95A430BE5E6E
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
prometheus:
  enable: true
  host: prometheus-tls.c.bpcli-dev.internal
  auth:
    type: basic
    username: admin
    password: password
dev:
  collector:
    create: true

TLS

I used step cli to create a CA and signed certificate for my host prometheus-tls.c.bpcli-dev.internal.

I configured web.yml on the prometheus server with:

tls_server_config:
  cert_file: /etc/prometheus/tls/prometheus.crt
  key_file: /etc/prometheus/tls/prometheus.key

I then created a secret with:

kubectl create secret generic prometheus-tls \     
  --from-file ca.crt 

config:
  username: bpuser
  password: bppass
  secret_key: 12D8FB6E-1532-4A4C-97AF-95A430BE5E6E
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
prometheus:
  enable: true
  host: prometheus-tls.c.bpcli-dev.internal
  tls:
    enable: true
    insecure: false
    secret:
      name: prometheus-tls
      caSubPath: ca.crt
dev:
  collector:
    create: true

TLS is tested by configuring the ca and keeping skip verify set to false.

Mutual TLS

I used step cli to create another signed certificate, this time the keypair is used for bindplane's client auth.

The client auth type and client ca file were configured like this:

tls_server_config:
  client_auth_type: RequireAndVerifyClientCert
  client_ca_file: /etc/prometheus/tls/ca.crt
  cert_file: /etc/prometheus/tls/prometheus.crt
  key_file: /etc/prometheus/tls/prometheus.key

After restarting prometheus, bindplane starts logging client auth errors (Expected, bindplane does not have a client keypair configured yet)

{"level":"error","timestamp":"2024-01-12T02:56:36.030Z","message":"failed to get configurationMetrics","error":"query: query: Post \"https://prometheus-tls.c.bpcli-dev.internal:9090/api/v1/query\": remote error: tls: certificate required"}

I created the secret with the bindplane client keypair and the ca

kubectl create secret generic prometheus-tls \
  --from-file prometheus-client.crt \     
  --from-file prometheus-client.key \     
  --from-file ca.crt 

config:
  username: bpuser
  password: bppass
  secret_key: 12D8FB6E-1532-4A4C-97AF-95A430BE5E6E
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
prometheus:
  enable: true
  host: prometheus-tls.c.bpcli-dev.internal
  tls:
    enable: true
    insecure: false
    secret:
      name: prometheus-tls
      caSubPath: ca.crt
      crtSubPath: prometheus-client.crt
      keySubPath: prometheus-client.key
dev:
  collector:
    create: true

Mimir

Using our in house tooling, I deployed mimir to my test cluster with:

kubectl create ns mimir
kustomize build app/priority-class/base | kubectl apply -f -
kustomize build app/mimir/environments/local | kubectl apply --namespace mimir -f -

The mimir config requires the following be set

• prometheus.host
• prometheus.port
• prometheus.queryPathPrefix
• prometheus.remoteWrite.host
• prometheus.remoteWrite.port
• prometheus.remoteWrite.path

The query host and remote write host are not the same.

config:
  username: bpuser
  password: bppass
  secret_key: 12D8FB6E-1532-4A4C-97AF-95A430BE5E6E
  sessions_secret: 4484766F-5016-4077-B8E0-0DE1D637854B
prometheus:
  enable: true
  host: mimir-query-frontend.mimir.svc.cluster.local
  port: 8080
  queryPathPrefix: /prometheus
  remoteWrite:
    host: mimir-distributor-headless.mimir.svc.cluster.local
    port: 8080
    path: /api/v1/push
dev:
  collector:
    create: true

I did not use our recording rule with mimir, so measurements are not exactly working 100% but I did confirm that remote write + query endpoints are working. No errors from bindplane, agents page does show measurements on a brand new install of bindplane.

Please check that the PR fulfills these requirements

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• CI passes