Unable to scrape metrics from Azure monitor

[tomkerkhove]

Discussed in #2011

^{Originally posted by anoop2503 April 21, 2022}
Hi,

I could manage to set up the promitor application in kubernetes environment. The application is up and running without any issues. However, it does not scrape metrics from Azure monitor.

Here is the configuration for setting up the promitor scraper agent (without resourceDiscovery).

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: promitor-agent-scraper
  namespace: monitoring
  labels:
    app: promitor-agent-scraper
data:
  runtime.yaml: |-
      server:
        httpPort: "5000"
      authentication:
        mode: UserAssignedManagedIdentity
        identityId: metrics-identity
      metricSinks:
        prometheusScrapingEndpoint:
          metricUnavailableValue: "NaN"
          enableMetricTimestamps: "true"
          baseUriPath: "/metrics"
          labels:
            transformation: "None"
      metricsConfiguration:
        absolutePath: /config/metrics-declaration.yaml
      telemetry:
        applicationInsights:
          isEnabled: "false"
        containerLogs:
          isEnabled: "true"
          verbosity: "Debug"
        defaultVerbosity: "Debug"
  metrics-declaration.yaml: |-
    version: v1
    azureMetadata:
      tenantId: xxxx-xxxxx-xxxxx-xxxx-xxxxx
      subscriptionId: xxxx-xxx-xxx-xxx-xxxx
      resourceGroupName: xxxxxxxxx
      cloud: Global
    metricDefaults:
      aggregation:
        interval: 00:02:00
      scraping:
        schedule: "*/2 * * * *"
    metrics:
      - name: azure_load_balancer_traffic_bytes
        description: "Average amount of bytes sent through an Azure Load Balancer"
        resourceType: LoadBalancer
        scraping:
          schedule: "*/2 * * * *"
        azureMetricConfiguration:
          metricName: ByteCount
          aggregation:
            type: Average
            interval: 00:05:00
        resources: # Optional, required when no resource discovery is configured
        - loadBalancerName: bda-azr-dev-wu2-zonal-agent-pool-internal
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: promitor-agent-scraper
  namespace: monitoring
  labels:
    aadpodidbinding: metrics-identity
    app: promitor-agent-scraper
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  selector:
    matchLabels:
      app: promitor-agent-scraper
  template:
    metadata:
      labels:
        app: promitor-agent-scraper
        aadpodidbinding: metrics-identity
    spec:
      containers:
        - name: promitor-agent-scraper
          image: "ghcr.io/tomkerkhove/promitor-agent-scraper:latest"
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 5000
              protocol: TCP
          volumeMounts:
          - name: config-volume-promitor-agent-scraper
            mountPath: /config/
          - name: tmp
            mountPath: /tmp/
      imagePullSecrets:
      - name: registry-credentials
      volumes:
        - name: config-volume-promitor-agent-scraper
          configMap:
            name: promitor-agent-scraper
        - emptyDir: {}
          name: tmp

I can see there is some authentication failure logs in the container logs though. Here is the container logs:

██████╗ ██████╗  ██████╗ ███╗   ███╗██╗████████╗ ██████╗ ██████╗ 
██╔══██╗██╔══██╗██╔═══██╗████╗ ████║██║╚══██╔══╝██╔═══██╗██╔══██╗
██████╔╝██████╔╝██║   ██║██╔████╔██║██║   ██║   ██║   ██║██████╔╝
██╔═══╝ ██╔══██╗██║   ██║██║╚██╔╝██║██║   ██║   ██║   ██║██╔══██╗
██║     ██║  ██║╚██████╔╝██║ ╚═╝ ██║██║   ██║   ╚██████╔╝██║  ██║
╚═╝     ╚═╝  ╚═╝ ╚═════╝ ╚═╝     ╚═╝╚═╝   ╚═╝    ╚═════╝ ╚═╝  ╚═╝
[14:06:55 INF] Booting up Promitor v2.6.1 running .NET .NET 6.0.3 - Thank you for using Promitor!
[14:06:55 INF] Running .NET 6.0.3 on Linux (alpine.3.14-x64 | Linux 5.10.61-flatcar #1 SMP Tue Aug 31 15:41:58 -00 2021).
[14:06:55 INF] Using configuration folder '/config/'
[14:06:56 INF] Starting validation of Promitor setup
[14:06:56 INF] Scrape configuration found at '/config/metrics-declaration.yaml'
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                              │ Outcome │ Details                        ┃
┠───────────────────────────────────┼─────────┼────────────────────────────────┨
┃ Metrics Declaration Path          │ Success │ Everything is well-configured. ┃
┃ Azure Authentication              │ Success │ Everything is well-configured. ┃
┃ Metrics Declaration               │ Success │ Everything is well-configured. ┃
┃ Resource Discovery                │ Success │ Everything is well-configured. ┃
┃ StatsD Metric Sink                │ Success │ Everything is well-configured. ┃
┃ Prometheus Scraping Endpoint      │ Success │ Everything is well-configured. ┃
┃ Metric Sink                       │         │                                ┃
┃ Atlassian Statuspage Metric Sink  │ Success │ Everything is well-configured. ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                                   Validation                                   
[14:06:56 INF] Promitor configuration is valid, we are good to go.
[14:06:56 INF] Here's an overview of what was configured:
╭───────────────────┬───────────────────┬───────────────────┬──────────────────╮
│ Tenant Id         │ Subscription Id   │ Resource Group    │ Azure Cloud      │
│                   │                   │ Name (Default)    │                  │
├───────────────────┼───────────────────┼───────────────────┼──────────────────┤
│ xxxx-xxxx-xxxxxxx │ xxxxxxxx-xxxx-xxx │ xxxxxxxxxxxxxxxxx │ Azure Global     │
│ x-xxxx-xxxxxxxxxx │ x-xxxx-xxxxxxxxxx │ xx                │ Cloud            │
│ xx                │ xx                │                   │                  │
╰───────────────────┴───────────────────┴───────────────────┴──────────────────╯
                                 Azure Metadata                                 
╭─────────────┬─────────────┬────────┬─────────────┬─────────────┬─────────────╮
│ Metric Name │ Resource    │ Labels │ Azure       │ Resources   │ Resource    │
│             │ Type        │        │ Monitor     │ To Scrape   │ Discovery   │
│             │             │        │ Metric      │             │ Groups To   │
│             │             │        │             │             │ Scrape      │
├─────────────┼─────────────┼────────┼─────────────┼─────────────┼─────────────┤
│ azure_load_ │ Load        │ None   │ ByteCount   │ xxx-azr-dev │ None        │
│ balancer_tr │ Balancer    │        │             │ -xxx-zonal- │             │
│ affic_bytes │             │        │             │ agent-pool- │             │
│             │             │        │             │ internal    │             │
╰─────────────┴─────────────┴────────┴─────────────┴─────────────┴─────────────╯
                               Configured Metrics                               
[14:06:57 INF] [Job][f7926029-4df3-43de-b973-baf17deefa69-k8sdevwu2clusterbda-azure_load_balancer_traffic_bytes-xxx-azr-dev-xxx-zonal-agent-pool-internal-fd6a9af8-799f-4136-b69a-8dd648655f61] is running under time zone: UTC
[14:06:57 INF] Scraping Azure Monitor - 04/21/2022 14:06:57 +00:00
[14:06:57 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:06:57 INF] Adding Prometheus sink to expose on /metrics
[14:06:57 INF] Adding Prometheus sink to expose on /metrics
[14:06:57 INF] Now listening on: http://[::]:5000
[14:06:57 INF] Application started. Press Ctrl+C to shut down.
[14:06:57 INF] Hosting environment: Production
[14:06:57 INF] Content root path: /app
[14:08:00 INF] Scraping Azure Monitor - 04/21/2022 14:08:00 +00:00
[14:08:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:10:00 INF] Scraping Azure Monitor - 04/21/2022 14:10:00 +00:00
[14:10:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:12:00 INF] Scraping Azure Monitor - 04/21/2022 14:12:00 +00:00
[14:12:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:14:00 INF] Scraping Azure Monitor - 04/21/2022 14:14:00 +00:00
[14:14:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:16:00 INF] Scraping Azure Monitor - 04/21/2022 14:16:00 +00:00
[14:16:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:18:00 INF] Scraping Azure Monitor - 04/21/2022 14:18:00 +00:00
[14:18:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:20:00 INF] Scraping Azure Monitor - 04/21/2022 14:20:00 +00:00
[14:20:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:22:00 INF] Scraping Azure Monitor - 04/21/2022 14:22:00 +00:00
[14:22:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:24:00 INF] Scraping Azure Monitor - 04/21/2022 14:24:00 +00:00
[14:24:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:26:00 INF] Scraping Azure Monitor - 04/21/2022 14:26:00 +00:00
[14:26:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:28:00 INF] Scraping Azure Monitor - 04/21/2022 14:28:00 +00:00
[14:28:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:30:00 INF] Scraping Azure Monitor - 04/21/2022 14:30:00 +00:00
[14:30:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:32:00 INF] Scraping Azure Monitor - 04/21/2022 14:32:00 +00:00
[14:32:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:34:00 INF] Scraping Azure Monitor - 04/21/2022 14:34:00 +00:00
[14:34:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:36:00 INF] Scraping Azure Monitor - 04/21/2022 14:36:00 +00:00
[14:36:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:38:00 INF] Scraping Azure Monitor - 04/21/2022 14:38:00 +00:00
[14:38:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:40:00 INF] Scraping Azure Monitor - 04/21/2022 14:40:00 +00:00
[14:40:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:42:00 INF] Scraping Azure Monitor - 04/21/2022 14:42:00 +00:00
[14:42:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:44:00 INF] Scraping Azure Monitor - 04/21/2022 14:44:00 +00:00
[14:44:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:46:00 INF] Scraping Azure Monitor - 04/21/2022 14:46:00 +00:00
[14:46:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:48:00 INF] Scraping Azure Monitor - 04/21/2022 14:48:00 +00:00
[14:48:00 INF] Scraping azure_load_balancer_traffic_bytes for resource type LoadBalancer
[14:48:26 FTL] Failed to scrape resource for metric 'azure_load_balancer_traffic_bytes'
Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSIMaxRetryReachedException: MSI: Failed to acquire tokens after retrying %20 times
   at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSITokenProvider.RetrieveTokenFromIMDSWithRetryAsync(String resource, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSITokenProvider.GetTokenFromIMDSEndpointAsync(String resource, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSITokenProvider.GetAuthenticationHeaderForVirtualMachineAsync(String resource, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.MSITokenProvider.GetAuthenticationHeaderAsync(CancellationToken cancellationToken)
   at Microsoft.Rest.TokenCredentials.ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.AzureCredentials.ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsOperations.ListWithHttpMessagesAsync(String resourceUri, String metricnamespace, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsOperationsExtensions.ListAsync(IMetricDefinitionsOperations operations, String resourceUri, String metricnamespace, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsImpl.ListByResourceAsync(String resourceId, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Monitor.Fluent.MetricDefinitionsImpl.Microsoft.Azure.Management.Monitor.Fluent.IMetricDefinitions.ListByResourceAsync(String resourceId, CancellationToken cancellationToken)
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.GetMetricDefinitionsAsync(String resourceId) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 124
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient.QueryMetricAsync(String metricName, String metricDimension, AggregationType aggregationType, TimeSpan aggregationInterval, String resourceId, String metricFilter, Nullable`1 metricLimit) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 80
   at Promitor.Core.Scraping.AzureMonitorScraper`1.ScrapeResourceAsync(String subscriptionId, ScrapeDefinition`1 scrapeDefinition, TResourceDefinition resourceDefinition, AggregationType aggregationType, TimeSpan aggregationInterval) in /src/Promitor.Core.Scraping/AzureMonitorScraper.cs:line 54
   at Promitor.Core.Scraping.Scraper`1.ScrapeAsync(ScrapeDefinition`1 scrapeDefinition) in /src/Promitor.Core.Scraping/Scraper.cs:line 78

Here are the metrics what promitor is exposing now:

# HELP promitor_runtime_dotnet_collection_count_total GC collection count
# TYPE promitor_runtime_dotnet_collection_count_total counter
promitor_runtime_dotnet_collection_count_total{generation="0"} 2
promitor_runtime_dotnet_collection_count_total{generation="1"} 1
promitor_runtime_dotnet_collection_count_total{generation="2"} 0
# HELP promitor_runtime_dotnet_totalmemory Total known allocated memory
# TYPE promitor_runtime_dotnet_totalmemory gauge
promitor_runtime_dotnet_totalmemory 10389832
# HELP promitor_runtime_process_cpu_seconds_total Total user and system CPU time spent in seconds
# TYPE promitor_runtime_process_cpu_seconds_total counter
promitor_runtime_process_cpu_seconds_total 10.1
# HELP promitor_runtime_process_virtual_bytes Process virtual memory size
# TYPE promitor_runtime_process_virtual_bytes gauge
promitor_runtime_process_virtual_bytes 23826468864
# HELP promitor_runtime_process_working_set Process working set
# TYPE promitor_runtime_process_working_set gauge
promitor_runtime_process_working_set 195641344
# HELP promitor_runtime_process_private_bytes Process private memory size
# TYPE promitor_runtime_process_private_bytes gauge
promitor_runtime_process_private_bytes 174456832
# HELP promitor_runtime_process_num_threads Total number of threads
# TYPE promitor_runtime_process_num_threads gauge
promitor_runtime_process_num_threads 28
# HELP promitor_runtime_process_processid Process ID
# TYPE promitor_runtime_process_processid gauge
promitor_runtime_process_processid 1
# HELP promitor_runtime_process_start_time_seconds Start time of the process since unix epoch in seconds
# TYPE promitor_runtime_process_start_time_seconds gauge
promitor_runtime_process_start_time_seconds 1650550015
# HELP promitor_runtime_http_request_duration_seconds duration histogram of http responses labeled with: status_code, method, path
# TYPE promitor_runtime_http_request_duration_seconds histogram
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.005"} 0
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.01"} 0
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.025"} 0
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.05"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.075"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.1"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.25"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="0.75"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="1"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="2.5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="7.5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="10"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="200",method="GET",path="/metrics",le="+Inf"} 1
promitor_runtime_http_request_duration_seconds_sum{status_code="200",method="GET",path="/metrics"} 0.040377444
promitor_runtime_http_request_duration_seconds_count{status_code="200",method="GET",path="/metrics"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.005"} 0
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.01"} 0
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.025"} 0
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.05"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.075"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.1"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.25"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="0.75"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="1"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="2.5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="7.5"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="10"} 1
promitor_runtime_http_request_duration_seconds_bucket{status_code="404",method="GET",path="/favicon.ico",le="+Inf"} 1
promitor_runtime_http_request_duration_seconds_sum{status_code="404",method="GET",path="/favicon.ico"} 0.030159532
promitor_runtime_http_request_duration_seconds_count{status_code="404",method="GET",path="/favicon.ico"} 1
# HELP promitor_scrape_error Provides an indication that the scraping of the resource has failed
# TYPE promitor_scrape_error gauge
promitor_scrape_error{tenant_id="xxxx-xxxx-xxxx-xxxx-xxxxxx",subscription_id="xxxxxx-xxxx-xxxxx-xxxxx-xxxxxx",resource_type="LoadBalancer",resource_name="xxx-azr-dev-xxx-zonal-agent-pool-internal",resource_group="xxxxxxxxxxxxx",metric_name="azure_load_balancer_traffic_bytes"} 1 1650552506423
# HELP promitor_scrape_success Provides an indication that the scraping of the resource was successful
# TYPE promitor_scrape_success gauge
promitor_scrape_success{tenant_id="xxxx-xxxx-xxxx-xxxx-xxxxxx",subscription_id="xxxxxx-xxxx-xxxxx-xxxxx-xxxxxx",resource_type="LoadBalancer",resource_name="xxx-azr-dev-xxx-zonal-agent-pool-internal",resource_group="xxxxxxxxxxxxx",metric_name="azure_load_balancer_traffic_bytes"} 0 1650552506422

What I understood from the container log is, the UserAssignedManagedIdentity authentication is failing when scraping metrics from azure monitor. But, I tried the same UserAssignedManagedIdentity as aadpodidbinding with azure_metrics_exporter and it returns the metrics without any issues. So, wondering if I have done any mistakes in the configurations side?

For authentication.identityId in runtime.yaml, I am using the same value what I am using for aadpodidbinding in deployment label section. I believe that is the correct configuration.

Please help.

Thanks,

[Mimetis]

Here are the checks I would do to investigate this issue (following the docs http://docs.promitor.io/walkthrough/use-promitor-with-managed-identity)

Be sure you have installed your AKS cluster using the managed identity option. Query the identity from the cluster:

echo "Retrieving cluster identity ID"
export aks_mi_identity="$(az aks show -g ${RG_NAME} -n ${CLUSTER_NAME} --query identityProfile.kubeletidentity.clientId -otsv)"

If I remember well, you may have the managed identity item also in the AKS resource group (called MC_....)

If you have created another managed identity (to respect the separation of concerns), be sure this identity has the "Monitor Reader" role assigned on your subscription / resource group

az role assignment list --assignee $AD_POD_IDENTITY_CLIENT_ID -g $RG_NAME | jq -r '.[].roleDefinitionName'

If you are using the system assigned identity, you can make the same check

From your cluster, check that your AAD Pod identity is correctly configured

kubectl get azureidentity
kubectl get azureidentitybinding

If you have correctly bind the AAD Pod Identity to your Managed Identity (System assigne or User Assigned) you should be able to use it.
As an example, you can spin up a Azure CLI using this identity:

kubectl run azure-cli -it --image=mcr.microsoft.com/azure-cli --labels=aadpodidbinding=$AD_POD_IDENTITY_NAME /bin/bash

az login -i --debug

If you CLI is launching correctly that means your AAD pod Identity is configured correctly and ready to use.

Once you are sure everything is correctly configured on your AKS cluster, you can configure Promitor

azureAuthentication:
  mode: SystemAssignedManagedIdentity
  identity:
    binding: <aad-pod-identity-name>  

[anoop2503]

@Mimetis Thank you so much for clear instructions.

I think the issue was with authentication mode. Based on the steps mentioned in your comment I tried the command "az login -i --debug" and found that the identity which I was using SystemAssignedManagedIdentity and not UserAssignedManagedIdentity. The application started scraping metrics from Azure monitor without any issues after changing the azureAuthentication mode to "SystemAssignedManagedIdentity".

Thanks for your help.

[tomkerkhove]

Great to hear that helped. Would it help if this is a troubleshooting guide on docs.promitor.io @anoop2503?

Thank you @Mimetis!