Prometheus collector deployment to different node pool

[ValdasDienas]

Hi guys
Is it possible to change ama metrics pod to be deployed to different node pool ?

[vishiy]

yes, for the replica pod . please see here - https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/prometheus-metrics-scrape-scale#schedule-ama-metrics-replica-pod-on-a-node-pool-with-more-resources

[bragi92]

Hey @ValdasDienas. The Daemonset pods will keep getting re-created on the individual nodes and can't be moved. This can only be done for the replicaset pod which has a name similar to ama-metrics-5cff76fc9-5h8nx (whereas the daemonset pods are similar to ama-metrics-node-gbwvk).

You can do a kubectl get pods -n kube-system -o wide | findstr "ama-metrics*" to get which pods are running on which nodes.

For e.g. on my cluster the replicaset was running on the node aks-nodepool1-33073163-vmss000001

I wanted to switch it to run on the aks-nodepool1-33073163-vmss000002 node so I ran the command
kubectl label nodes aks-nodepool1-33073163-vmss000002 azuremonitor/metrics.replica.preferred="true"

After that I did a delete on my replicaset pod with kubectl delete pod ama-metrics-5cff76fc9-5h8nx -n kube-system

Now the new pod is running on the aks-nodepool1-33073163-vmss000002 node:

Does the above steps help out with your scenario? If not, can you share the details of kubectl describe pod ama-metrics-5cff76fc9-5rhtm -n kube-system for your replicaset pod?

[ValdasDienas]

Hi @bragi92 all steps you mentioned were done but pod keeps sitting in the same machine. Just for you additional info in my cluster there are two node pools system and user I want to move ama metrics pod to user pool machine because its consuming a lot of ram

[bragi92]

@ValdasDienas I see, can you share the results of the kubectl describe command for the replicaset pod? And also the results of the kubectl describe command for the node on which you applied the azuremonitor/metrics.replica.preferred="true" label?

[ValdasDienas]

valdas@Valdass-MacBook-Pro ~ % kubectl describe pod ama-metrics-dd58cd6dc-6tnvx -n kube-system
Name: ama-metrics-dd58cd6dc-6tnvx
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Node: aks-agentpool-12461548-vmss00000r/10.224.0.222
Start Time: Wed, 10 May 2023 17:07:56 +0300
Labels: kubernetes.azure.com/managedby=aks
pod-template-hash=dd58cd6dc
rsName=ama-metrics
Annotations: agentVersion: 0.0.0.1
cluster-autoscaler.kubernetes.io/safe-to-evict: true
kubectl.kubernetes.io/restartedAt: 2023-05-10T14:07:56Z
schema-versions: v1
Status: Running
IP: 10.224.0.226
IPs:
IP: 10.224.0.226
Controlled By: ReplicaSet/ama-metrics-dd58cd6dc
Containers:
prometheus-collector:
Container ID: containerd://4f6dd10aa63e91990434b1e3d6d06dddafcba18d0d48770a2f21bd578ac9c6ac
Image: mcr.microsoft.com/azuremonitor/containerinsights/ciprod/prometheus-collector/images:6.6.0-main-04-25-2023-2eb2a81c
Image ID: mcr.microsoft.com/azuremonitor/containerinsights/ciprod/prometheus-collector/images@sha256:060ed5df4a0f10b895b066b702417c96522c2b0ca25cc425a606a0065e7ce563
Port:
Host Port:
State: Running
Started: Sun, 14 May 2023 10:00:48 +0300
Last State: Terminated
Reason: Unknown
Exit Code: 255
Started: Wed, 10 May 2023 17:07:57 +0300
Finished: Sun, 14 May 2023 10:00:27 +0300
Ready: True
Restart Count: 1
Limits:
cpu: 7
memory: 14Gi
Requests:
cpu: 150m
memory: 500Mi
Liveness: exec [/bin/bash -c /opt/microsoft/liveness/livenessprobe.sh] delay=60s timeout=5s period=15s #success=1 #failure=3
Environment:
MINIMAL_INGESTION_PROFILE: true
KUBERNETES_PORT_443_TCP_ADDR:
KUBERNETES_PORT_443_TCP:
CLUSTER:
CONTROLLER_TYPE: ReplicaSet
NODE_NAME: (v1:spec.nodeName)
KUBE_STATE_VERSION: mcr.microsoft.com/oss/kubernetes/kube-state-metrics:v2.8.1
WINMODE:
NODE_IP: (v1:status.hostIP)
POD_NAME: ama-metrics-dd58cd6dc-6tnvx (v1:metadata.name)
AGENT_VERSION: 6.6.0-main-04-25-2023-2eb2a81c
KUBERNETES_PORT:
NODE_EXPORTER_TARGETPORT: 19100
NODE_EXPORTER_VERSION:
KUBERNETES_SERVICE_HOST:
AKSREGION: westeurope
AZMON_COLLECT_ENV: false
customEnvironment:
POD_NAMESPACE: kube-system (v1:metadata.namespace)
CONTAINER_MEMORY_LIMIT: 14336 (limits.memory)
MODE: advanced
MAC: true
OMS_TLD: opinsights.azure.com
CONTAINER_CPU_LIMIT: 7000 (limits.cpu)
KUBE_STATE_NAME: ama-metrics-ksm
NODE_EXPORTER_NAME:
Mounts:
/anchors/mariner from anchors-mariner (ro)
/anchors/ubuntu from anchors-ubuntu (ro)
/etc/config/settings from settings-vol-config (ro)
/etc/config/settings/prometheus from prometheus-config-vol (ro)
/var/log/containers from host-log-containers (ro)
/var/log/pods from host-log-pods (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wvfgm (ro)
addon-token-adapter:
Container ID: containerd://d91a1a1da15397ffb6227a51eb04a8fc54b39c3239ee5c4e0d12c1fc5f302992
Image: mcr.microsoft.com/aks/msi/addon-token-adapter:master.221118.2
Image ID: sha256:6284cbe7030dfa0d79382ea3f86cf3b892d0ee17807b01d921e97a7877e27313
Port:
Host Port:
Command:
/addon-token-adapter
Args:
--secret-namespace=kube-system
--secret-name=aad-msi-auth-token
--token-server-listening-port=7777
--health-server-listening-port=9999
State: Running
Started: Sun, 14 May 2023 10:00:52 +0300
Last State: Terminated
Reason: Unknown
Exit Code: 255
Started: Wed, 10 May 2023 17:07:57 +0300
Finished: Sun, 14 May 2023 10:00:28 +0300
Ready: True
Restart Count: 1
Limits:
cpu: 500m
memory: 500Mi
Requests:
cpu: 20m
memory: 30Mi
Liveness: http-get http://:9999/healthz delay=10s timeout=1s period=60s #success=1 #failure=3
Environment:
KUBERNETES_PORT_443_TCP_ADDR:
KUBERNETES_PORT:
KUBERNETES_PORT_443_TCP:
KUBERNETES_SERVICE_HOST:
AZMON_COLLECT_ENV: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wvfgm (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
settings-vol-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: ama-metrics-settings-configmap
Optional: true
prometheus-config-vol:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: ama-metrics-prometheus-config
Optional: true
host-log-containers:
Type: HostPath (bare host directory volume)
Path: /var/log/containers
HostPathType:
host-log-pods:
Type: HostPath (bare host directory volume)
Path: /var/log/pods
HostPathType:
anchors-mariner:
Type: HostPath (bare host directory volume)
Path: /etc/pki/ca-trust/anchors/
HostPathType: DirectoryOrCreate
anchors-ubuntu:
Type: HostPath (bare host directory volume)
Path: /usr/local/share/ca-certificates/
HostPathType: DirectoryOrCreate
kube-api-access-wvfgm:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: :NoExecute op=Exists
:NoSchedule op=Exists
CriticalAddonsOnly op=Exists
Events:

[ValdasDienas]

valdas@Valdass-MacBook-Pro ~ % kubectl describe node aks-prometheus-37861458-vmss000000
Name: aks-prometheus-37861458-vmss000000
Roles: agent
Labels: agentpool=prometheus
azuremonitor/metrics.replica.preferred=true
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=Standard_B2ms
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=westeurope
failure-domain.beta.kubernetes.io/zone=0
kubernetes.azure.com/agentpool=prometheus
kubernetes.azure.com/cluster=
kubernetes.azure.com/kubelet-identity-client-id=dc325f26-2cfd-4a04-8951-83b4f30a9901
kubernetes.azure.com/mode=user
kubernetes.azure.com/node-image-version=AKSUbuntu-1804gen2containerd-202304.20.0
kubernetes.azure.com/os-sku=Ubuntu
kubernetes.azure.com/role=agent
kubernetes.azure.com/storageprofile=managed
kubernetes.azure.com/storagetier=Premium_LRS
kubernetes.io/arch=amd64
kubernetes.io/hostname=aks-prometheus-37861458-vmss000000
kubernetes.io/os=linux
kubernetes.io/role=agent
node-role.kubernetes.io/agent=
node.kubernetes.io/instance-type=Standard_B2ms
storageprofile=managed
storagetier=Premium_LRS
topology.disk.csi.azure.com/zone=
topology.kubernetes.io/region=westeurope
topology.kubernetes.io/zone=0
Annotations: csi.volume.kubernetes.io/nodeid:
{"blob.csi.azure.com":"aks-prometheus-37861458-vmss000000","disk.csi.azure.com":"aks-prometheus-37861458-vmss000000","file.csi.azure.com":...
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 08 May 2023 09:47:02 +0300
Taints:
Unschedulable: false
Lease:
HolderIdentity: aks-prometheus-37861458-vmss000000
AcquireTime:
RenewTime: Mon, 15 May 2023 09:27:01 +0300
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message

---

FrequentDockerRestart False Mon, 15 May 2023 09:26:39 +0300 Sun, 14 May 2023 19:15:03 +0300 NoFrequentDockerRestart docker is functioning properly
ContainerRuntimeProblem False Mon, 15 May 2023 09:26:39 +0300 Sun, 14 May 2023 19:15:03 +0300 ContainerRuntimeIsUp container runtime service is up
KernelDeadlock False Mon, 15 May 2023 09:26:39 +0300 Sun, 14 May 2023 19:15:03 +0300 KernelHasNoDeadlock kernel has no deadlock
ReadonlyFilesystem False Mon, 15 May 2023 09:26:39 +0300 Sun, 14 May 2023 19:15:03 +0300 FilesystemIsNotReadOnly Filesystem is not read-only
KubeletProblem False Mon, 15 May 2023 09:26:39 +0300 Sun, 14 May 2023 19:15:03 +0300 KubeletIsUp kubelet service is up
FrequentContainerdRestart False Mon, 15 May 2023 09:26:39 +0300 Sun, 14 May 2023 19:15:03 +0300 NoFrequentContainerdRestart containerd is functioning properly
FilesystemCorruptionProblem False Mon, 15 May 2023 09:26:39 +0300 Sun, 14 May 2023 19:15:03 +0300 FilesystemIsOK Filesystem is healthy
FrequentUnregisterNetDevice False Mon, 15 May 2023 09:26:39 +0300 Sun, 14 May 2023 19:15:03 +0300 NoFrequentUnregisterNetDevice node is functioning properly
VMEventScheduled False Mon, 15 May 2023 09:26:39 +0300 Sun, 14 May 2023 19:15:03 +0300 NoVMEventScheduled VM has no scheduled event
FrequentKubeletRestart False Mon, 15 May 2023 09:26:39 +0300 Sun, 14 May 2023 19:15:03 +0300 NoFrequentKubeletRestart kubelet is functioning properly
MemoryPressure False Mon, 15 May 2023 09:23:06 +0300 Mon, 08 May 2023 09:47:02 +0300 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Mon, 15 May 2023 09:23:06 +0300 Mon, 08 May 2023 09:47:02 +0300 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Mon, 15 May 2023 09:23:06 +0300 Mon, 08 May 2023 09:47:02 +0300 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Mon, 15 May 2023 09:23:06 +0300 Mon, 08 May 2023 09:47:13 +0300 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 10.224.0.4
Hostname: aks-prometheus-37861458-vmss000000
Capacity:
cpu: 2
ephemeral-storage: 129886128Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 8148452Ki
pods: 30
Allocatable:
cpu: 1900m
ephemeral-storage: 119703055367
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 5493220Ki
pods: 30
System Info:
Machine ID: ef3395b441bd4126acb1063daa0c2d97
System UUID: 1e4b4700-cdb2-430f-beb9-3c3a0a593ce0
Boot ID: 30517b70-c5a0-4283-82a0-739aa2e22f55
Kernel Version: 5.4.0-1106-azure
OS Image: Ubuntu 18.04.6 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.6.18+azure-1
Kubelet Version: v1.24.10
Kube-Proxy Version: v1.24.10
ProviderID: azure:///subscriptions/3c8d215a-71e8-4012-b93f-5292b4abef2a/resourceGroups/un-rg-test-weu-aks/providers/Microsoft.Compute/virtualMachineScaleSets/aks-prometheus-37861458-vmss/virtualMachines/0
Non-terminated Pods: (11 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age

---

ingress-nginx ingress-nginx-controller-dgr9z 100m (5%) 0 (0%) 90Mi (1%) 0 (0%) 6d23h
kube-system ama-metrics-node-5t5k8 95m (5%) 700m (36%) 180Mi (3%) 1524Mi (28%) 5d14h
kube-system azure-ip-masq-agent-l68gl 100m (5%) 500m (26%) 50Mi (0%) 250Mi (4%) 6d23h
kube-system cloud-node-manager-zbd4z 50m (2%) 0 (0%) 50Mi (0%) 512Mi (9%) 6d23h
kube-system csi-azuredisk-node-hxvwf 30m (1%) 0 (0%) 60Mi (1%) 400Mi (7%) 6d23h
kube-system csi-azurefile-node-9pjhf 30m (1%) 0 (0%) 60Mi (1%) 600Mi (11%) 6d23h
kube-system csi-blob-node-65kmq 30m (1%) 0 (0%) 60Mi (1%) 600Mi (11%) 6d23h
kube-system kube-proxy-4w2rm 100m (5%) 0 (0%) 0 (0%) 0 (0%) 6d23h
kube-system nmi-wsg6b 100m (5%) 200m (10%) 256Mi (4%) 512Mi (9%) 6d23h
meilisearch meilisearch-0 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5d7h
origin-ca-issuer origin-ca-issuer-694c65b94c-nc6pv 100m (5%) 100m (5%) 100Mi (1%) 100Mi (1%) 5d6h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits

---

cpu 735m (38%) 1500m (78%)
memory 906Mi (16%) 4498Mi (83%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:

[vishiy]

@bragi92

[vishiy]

Hi @ValdasDienas - After investigating, as per AKS weights on scheduling labels, system node pools have more weight on scheduling AKS managed addon pods as best practice. So if you add this label to an user/app pool, it will be a hit or miss and hence the behavior you are seeing. Would you be able to create a better system pool ? you can either create a new system pool with better node sku as you may need and delete the old system pool or just add a new system pool . either ways, you would be adding the label to one of those system node pool as best practice (rather than individual nodes, which might come & go).

1. please see here about system pools -- https://learn.microsoft.com/en-us/azure/aks/use-system-pools?tabs=azure-cli#system-and-user-node-pools
2. please see here on how to add labels to node pool if you need -- https://learn.microsoft.com/en-us/azure/aks/use-labels#create-a-node-pool-with-a-label

I will be updating our documentation to reflect the above same.

Thank you for your responses on this issue. Please let us know if we can close this discussion once you have tried the solution.

[vishiy]

@ValdasDienas - please let us know if we can close this discussion or you have any more comments. Thank you !

[vishiy]

Closing this as our docs has been updated. Please let us know if you need any more help on this.