Skip to content

Commit

Permalink
Merge branch 'v1.11' into aacrawfi/split-workflow
Browse files Browse the repository at this point in the history
  • Loading branch information
@msfussell
msfussell authored Jun 17, 2023
2 parents ae422e4 + ee25a6a commit 1cd9f4c
Showing 1 changed file with 103 additions and 101 deletions.
204 changes: 103 additions & 101 deletions daprdocs/content/en/operations/monitoring/metrics/azure-monitor.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,118 +15,120 @@ description: "Enable Dapr metrics and logs with Azure Monitor for Azure Kubernet

## Enable Prometheus metric scrape using config map

1. Make sure that omsagents are running

```bash
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
...
omsagent-75qjs 1/1 Running 1 44h
omsagent-c7c4t 1/1 Running 0 44h
omsagent-rs-74f488997c-dshpx 1/1 Running 1 44h
omsagent-smtk7 1/1 Running 1 44h
...
```

2. Apply config map to enable Prometheus metrics endpoint scrape.

You can use [azm-config-map.yaml](/docs/azm-config-map.yaml) to enable prometheus metrics endpoint scrape.

If you installed Dapr to the different namespace, you need to change the `monitor_kubernetes_pod_namespaces` array values. For example:

```yaml
...
prometheus-data-collection-settings: |-
[prometheus_data_collection_settings.cluster]
interval = "1m"
monitor_kubernetes_pods = true
monitor_kubernetes_pods_namespaces = ["dapr-system", "default"]
[prometheus_data_collection_settings.node]
interval = "1m"
...
```

Apply config map:

```bash
kubectl apply -f ./azm-config.map.yaml
```
1. Make sure that Azure Monitor Agents (AMA) are running.

```bash
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
...
ama-logs-48kpv 2/2 Running 0 2d13h
ama-logs-mx24c 2/2 Running 0 2d13h
ama-logs-rs-f9bbb9898-vbt6k 1/1 Running 0 30h
ama-logs-sm2mz 2/2 Running 0 2d13h
ama-logs-z7p4c 2/2 Running 0 2d13h
...
```

1. Apply config map to enable Prometheus metrics endpoint scrape.

You can use [azm-config-map.yaml](/docs/azm-config-map.yaml) to enable Prometheus metrics endpoint scrape.

If you installed Dapr to a different namespace, you need to change the `monitor_kubernetes_pod_namespaces` array values. For example:

```yaml
...
prometheus-data-collection-settings: |-
[prometheus_data_collection_settings.cluster]
interval = "1m"
monitor_kubernetes_pods = true
monitor_kubernetes_pods_namespaces = ["dapr-system", "default"]
[prometheus_data_collection_settings.node]
interval = "1m"
...
```

Apply config map:

```bash
kubectl apply -f ./azm-config.map.yaml
```

## Install Dapr with JSON formatted logs

1. Install Dapr with enabling JSON-formatted logs

```bash
helm install dapr dapr/dapr --namespace dapr-system --set global.logAsJson=true
```

2. Enable JSON formatted log in Dapr sidecar and add Prometheus annotations.

> Note: OMS Agent scrapes the metrics only if replicaset has Prometheus annotations.
Add `dapr.io/log-as-json: "true"` annotation to your deployment yaml.

Example:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: pythonapp
namespace: default
labels:
app: python
spec:
replicas: 1
selector:
matchLabels:
app: python
template:
metadata:
labels:
app: python
annotations:
dapr.io/enabled: "true"
dapr.io/app-id: "pythonapp"
dapr.io/log-as-json: "true"
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/path: "/"

...
```
1. Install Dapr with enabling JSON-formatted logs.

```bash
helm install dapr dapr/dapr --namespace dapr-system --set global.logAsJson=true
```

1. Enable JSON formatted log in Dapr sidecar and add Prometheus annotations.

> Note: The Azure Monitor Agents (AMA) only sends the metrics if the Prometheus annotations are set.

Add `dapr.io/log-as-json: "true"` annotation to your deployment yaml.

Example:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: pythonapp
namespace: default
labels:
app: python
spec:
replicas: 1
selector:
matchLabels:
app: python
template:
metadata:
labels:
app: python
annotations:
dapr.io/enabled: "true"
dapr.io/app-id: "pythonapp"
dapr.io/log-as-json: "true"
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/path: "/"
...
```

## Search metrics and logs with Azure Monitor

1. Go to Azure Monitor
1. Go to Azure Monitor in the Azure portal.

2. Search Dapr logs
1. Search Dapr **Logs**.

Here is an example query, to parse JSON formatted logs and query logs from dapr system processes.
Here is an example query, to parse JSON formatted logs and query logs from Dapr system processes.

```
ContainerLog
| extend parsed=parse_json(LogEntry)
| project Time=todatetime(parsed['time']), app_id=parsed['app_id'], scope=parsed['scope'],level=parsed['level'], msg=parsed['msg'], type=parsed['type'], ver=parsed['ver'], instance=parsed['instance']
| where level != ""
| sort by Time
```
```
ContainerLog
| extend parsed=parse_json(LogEntry)
| project Time=todatetime(parsed['time']), app_id=parsed['app_id'], scope=parsed['scope'],level=parsed['level'], msg=parsed['msg'], type=parsed['type'], ver=parsed['ver'], instance=parsed['instance']
| where level != ""
| sort by Time
```

3. Search metrics
1. Search **Metrics**.

This query, queries process_resident_memory_bytes Prometheus metrics for Dapr system processes and renders timecharts
This query, queries `process_resident_memory_bytes` Prometheus metrics for Dapr system processes and renders timecharts.

```
InsightsMetrics
| where Namespace == "prometheus" and Name == "process_resident_memory_bytes"
| extend tags=parse_json(Tags)
| project TimeGenerated, Name, Val, app=tostring(tags['app'])
| summarize memInBytes=percentile(Val, 99) by bin(TimeGenerated, 1m), app
| where app startswith "dapr-"
| render timechart
```
```
InsightsMetrics
| where Namespace == "prometheus" and Name == "process_resident_memory_bytes"
| extend tags=parse_json(Tags)
| project TimeGenerated, Name, Val, app=tostring(tags['app'])
| summarize memInBytes=percentile(Val, 99) by bin(TimeGenerated, 1m), app
| where app startswith "dapr-"
| render timechart
```

# References
## References

* [Configure scraping of Prometheus metrics with Azure Monitor for containers](https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-prometheus-integration)
* [Configure agent data collection for Azure Monitor for containers](https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-agent-config)
* [Azure Monitor Query](https://docs.microsoft.com/azure/azure-monitor/log-query/query-language)
- [Configure scraping of Prometheus metrics with Azure Monitor for containers](https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-prometheus-integration)
- [Configure agent data collection for Azure Monitor for containers](https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-agent-config)
- [Azure Monitor Query](https://docs.microsoft.com/azure/azure-monitor/log-query/query-language)

0 comments on commit 1cd9f4c

Please sign in to comment.