Skip to content

Commit

Permalink
Merge pull request #4321 from maulindesai/prometheus-auto-service-dis…
Browse files Browse the repository at this point in the history
…covery

Prometheus auto service discovery
  • Loading branch information
hhunter-ms authored Aug 23, 2024
2 parents 967c17f + 431adb6 commit 62bd96c
Show file tree
Hide file tree
Showing 3 changed files with 111 additions and 0 deletions.
111 changes: 111 additions & 0 deletions daprdocs/content/en/operations/observability/metrics/prometheus.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,13 +93,108 @@ helm install dapr-prom prometheus-community/prometheus -n dapr-monitoring
--set alertmanager.persistence.enabled=false --set pushgateway.persistentVolume.enabled=false --set server.persistentVolume.enabled=false
```

For automatic discovery of Dapr targets (Service Discovery), use:

```bash
helm install dapr-prom prometheus-community/prometheus -f values.yaml -n dapr-monitoring --create-namespace
```

### `values.yaml` File

```yaml
alertmanager:
persistence:
enabled: false
pushgateway:
persistentVolume:
enabled: false
server:
persistentVolume:
enabled: false

# Adds additional scrape configurations to prometheus.yml
# Uses service discovery to find Dapr and Dapr sidecar targets
extraScrapeConfigs: |-
- job_name: dapr-sidecars
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
regex: "true"
source_labels:
- __meta_kubernetes_pod_annotation_dapr_io_enabled
- action: keep
regex: "true"
source_labels:
- __meta_kubernetes_pod_annotation_dapr_io_enable_metrics
- action: replace
replacement: ${1}
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
replacement: ${1}
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
regex: (.*);daprd
replacement: ${1}-dapr
source_labels:
- __meta_kubernetes_pod_annotation_dapr_io_app_id
- __meta_kubernetes_pod_container_name
target_label: service
- action: replace
replacement: ${1}:9090
source_labels:
- __meta_kubernetes_pod_ip
target_label: __address__
- job_name: dapr
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
regex: dapr
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- action: keep
regex: dapr
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_part_of
- action: replace
replacement: ${1}
source_labels:
- __meta_kubernetes_pod_label_app
target_label: app
- action: replace
replacement: ${1}
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
replacement: ${1}
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
replacement: ${1}:9090
source_labels:
- __meta_kubernetes_pod_ip
target_label: __address__
```
3. Validation
Ensure Prometheus is running in your cluster.
```bash
kubectl get pods -n dapr-monitoring
```

Expected output:

```bash
NAME READY STATUS RESTARTS AGE
dapr-prom-kube-state-metrics-9849d6cc6-t94p8 1/1 Running 0 4m58s
dapr-prom-prometheus-alertmanager-749cc46f6-9b5t8 2/2 Running 0 4m58s
Expand All @@ -110,6 +205,22 @@ dapr-prom-prometheus-pushgateway-688665d597-h4xx2 1/1 Running 0
dapr-prom-prometheus-server-694fd8d7c-q5d59 2/2 Running 0 4m58s
```

### Access the Prometheus Dashboard

To view the Prometheus dashboard and check service discovery:

```bash
kubectl port-forward svc/dapr-prom-prometheus-server 9090:80 -n dapr-monitoring
```

Open a browser and visit `http://localhost:9090`. Navigate to **Status** > **Service Discovery** to verify that the Dapr targets are discovered correctly.

<img src="/images/prometheus-web-ui.png" alt="Prometheus Web UI" width="1200">

You can see the `job_name` and its discovered targets.

<img src="/images/prometheus-service-discovery.png" alt="Prometheus Service Discovery" width="1200">

## Example

<div class="embed-responsive embed-responsive-16by9">
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added daprdocs/static/images/prometheus-web-ui.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 62bd96c

Please sign in to comment.