prometheus.md

Sourcegraph Prometheus

WARNING: Looking for Prometheus documentation for Sourcegraph administrators? See the metrics and dashboards documentation.

We ship a custom Prometheus image as part of a standard Sourcegraph distribution. It currently bundles Alertmanager as well as integrations to the Sourcegraph web application. Learn more about it in our monitoring architecture.

Adding recording rules, alerts, etc. to this image is handled by the monitoring generator.

The image is defined in docker-images/prometheus.

Metrics

See the metrics and dashboards documentation.

To learn more about developing metrics, see the observability developer guides.

Prom-wrapper

The entrypoint of the image is a sidecar program called the prom-wrapper. It manages Prometheus and Alertmanager, and provides integration with the Sourcegraph frontend. Learn more about it here.

The source code for this program is currently kept in docker-images/prometheus/cmd/prom-wrapper. The prom-wrapper also exports an API which can be leveraged through the internal/src-prometheus package.

To learn more about developing our observability stack, see the local Sourcegraph monitoring development guide.

Alertmanager

The Sourcegraph Prometheus image ships with Alertmanager, which provides our alerting capabilities.

Note that prom-wrapper uses a fork of Alertmanager to better manipulate Alertmanager configuration—prom-wrapper needs to be able to write alertmanager configuration with secrets, etc, which the Alertmanager project is currently not planning on accepting changes for (alertmanager#2316). This does not affect the version of Alertmanager that we ship with, the fork exists purely for use as a library.

Upgrading Prometheus or Alertmanager

When upgrading, it is better to upgrade both at once since the two projects share some common dependencies. To perform an upgrade:

1. Make the appropriate version and sum changes to the sourcegraph/prometheus Dockerfile
2. Ensure no image update steps are required by checking upstream Dockerfiles where required as noted in the sourcegraph/prometheus Dockerfile where appropriate
3. Upgrade the Alertmanager and Prometheus Go client dependencies where appropriate
   1. For the Alertmanager dependency, the fork needs to be upgraded to the appropriate version first: sourcegraph/alertmanager
4. Ensure the image still builds: ./docker-images/prometheus/build.sh
5. Run the monitoring stack locally and verify that:

   1. If upgrading Prometheus: all Prometheus rules are evaluated successfully (localhost:9090/rules)

   2. If upgrading Alertmanager: Alertmanager starts up correctly (localhost:9090/alertmanager/#/status), and observability.alerts can be configured in site config (check this by adding an entry, e.g. Slack alerts) via the Sourcegraph web application, e.g:

      "observability.alerts": [
        {
          "level": "critical",
          "notifier": {
            "type": "slack",
            "url": "https://sourcegraph.com",
          }
        }
      ]