the problem of getting data from multiple replicas under Kubernetes deployment controller and how to distinguish replica labels based on podname

[Dkrismile81]

[root@master ~]# kubectl -n test-service get deployments.apps file
NAME READY UP-TO-DATE AVAILABLE AGE
file 2/2 2 2 3d20h
[root@master ~]# kubectl -n test-service get svc file
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
file NodePort 10.96.0.122 5000:40206/TCP,8080:40226/TCP,8081:40240/TCP 9d
[root@master ~]# kubectl -n test-service get pod |grep file
file-867694979f-bdlwv 2/2 Running 0 147m
file-867694979f-gxbgd 2/2 Running 0 136m

There are two copies of the deployment file above. The port opened by my jmx-export is 8081, but I only obtained the data of pod file-867694979F-bdlwv through the corresponding service access interface. I did not get the data of pod file-867694979f-gxbgd. Do you have any help to solve the doubts? Thank you very much ！ ！ ！ ！

Another question is how can I make Prometheus have a pod name tag that identifies which pod the data belongs to, thanks a lot!!

[Dkrismile81]

Should I use kubernetes_sd_configs to get data for each pod?

[Dkrismile81]

I have solved this problem by taking the data from kubernetes_sd_configs and taging the pod with JMX-Export enabled with a jmx=true tag. Here is an example of this in my Prometheus file

• job_name: 'kubernetes-jmx'
  kubernetes_sd_configs:
  • role: pod
    relabel_configs:
  • source_labels: [__meta_kubernetes_pod_label_jmx] # Matches only Pods that contain jmx tags
    regex: 'true' # The jmx tag value is true
    action: keep # Keep the matching Pod
  • source_labels: [address]
    regex: '([^:]+)(:\d+)?'
    replacement: '${1}:8081'
    target_label: address
    action: replace # The replacement port is 8081
  • action: labelmap
    regex: _meta_kubernetes_pod_label(.+)
  • source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  • source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name