Horizontal Pod Autoscale with Custom Prometheus Metrics

[mangreen]

https://dev.to/mjace/horizontal-pod-autoscale-with-custom-prometheus-metrics-5gem

[mangreen]

k8s 监控（一）安装 Prometheus
https://juejin.cn/post/6844903908251451406

k8s 監控（二）監控集群組件和 pod
https://juejin.cn/post/6844903921207492621

k8s 監控（三）prometheus-adapter
https://juejin.cn/post/6844903967218991117

k8s 監控（四）監控宿主機
https://juejin.cn/post/6844904057098731534

[mangreen]

• Prometheus-adapter配置写法
• HPA在k8s中的结构

[mangreen]

基于Custom-metrics-apiserver实现Kubernetes的HPA

部署 kube-prometheus-stack (prometheus-operator)

  additionalServiceMonitors: 
    - name: myapp-monitor
      selector:
        matchLabels:
          app.kubernetes.io/instance: myapp
          app.kubernetes.io/name: myapp
      namespaceSelector:
        # any: true
        matchNames:
          - myapp
      endpoints:
        - port: http
          interval: 10s
          path: /actuator/prometheus

部署 prometheus-adapter

rules:
  default: false

  custom: 
    - seriesQuery: 'myapp_http_requests_count'
      resources:
        template: <<.Resource>>
      name:
        matches: "^(.*)_count"
        as: "${1}_per_second"
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)

記得覆寫 prometheus.url 為 http://<kube-prometheus-stack service>.<namespace>.svc.cluster.local

• 透過 --set prometheus.url=

helm install <adapter-name> -n <namespace> --set prometheus.url=http://kube-prometheus-stack.monitoring.svc.cluster.local --set prometheus.port=9090 --set prometheus.path=/

• 透過 -f values.yaml

...
prometheus:
  url: http://prometheus-opreated.monitoring.svc
  port: 9090
  path: /
...

部署 APP 和 Service

建立 HPA

基於 POD

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-custom-metrics-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp-deploy-v1.0
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: myapp_http_requests_count
      target:
        type: AverageValue
        averageValue: 5000m

自訂資源指標含義是基於 myapp-deploy-v1.0 這個 deployment 的全部 pod 來計算http_requests的平局值，如果達到 5000m就進行擴容。

• scaleTargetRef：這裡定義的是自動擴容縮容的對象，可以是Deployment或者ReplicaSet，這裡寫具體的Deployment的名稱。
• metrics：這裡是指標的目標值。在type中定義類型；通過target來定義指標的閾值，系統將在指標達到閾值的時候出發擴縮容操作。
  metrics中的type有如下類型：
  • Resource：基於資源的指標，可以是CPU或者是記憶體，如果基於這個類型的指標來做只需要部署Metric-server即可，不需要部署自訂APISERVER。
  • Pods：基於Pod的指標，系統將對Deployment中的全部Pod副本指標進行平均值計算，如果是Pod則該指標必須來源於Pod本身。
  • Object：基於Ingress或者其他自訂指標，比如ServiceMonitor。它的target類型可以是Value或者AverageValue（根據Pod副本數計算平均值）。

基於 Service

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-custom-metrics-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp-deploy-v1.0
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Object
    object:
      target:
        kind: Service
        # 這裡是你自己的APP的service
        name: myapp-svc
      metric:
        name: http_requests
      target:
        type: AverageValue
        averageValue: 200
--
apiVersion: v1
kind: Service
metadata:
  name: myapp-svc
  labels:
    appname: myapp-svc
  annotations:
    prometheus.io/scrape: "true"  # 新增內容
    prometheus.io/port: "5555"  # 新增內容
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 5555
    targetPort: 5555
  selector:
    appname: myapp

然後 prometheus-adapter 規則，增加如下內容：

rules:
  default: false

  custom:
    - seriesQuery: '{__name__=~"^http_requests_.*",kubernetes_name!="",kubernetes_namespace!=""}'
      seriesFilters: []
      resources:
        overrides:
          kubernetes_namespace:
            resource: namespace
          kubernetes_name:
            resource: service
      name:
        matches: ^(.*)_(total)$
        as: "${1}"
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)