Adding nginx-prometheus-exporter as a sidecar in nginx-ingress helm chart.

[jcholowaty]

Hi! i'm using the NGINX Ingress Controller helm chart v0.18.0 in GKE and trying to add a sidecar the nginx-prometheus-exporter v0.11.0 but the that container fail.

nginxStatus:
    ## Enable the NGINX stub_status, or the NGINX Plus API.
    enable: true

    ## Set the port where the NGINX stub_status or the NGINX Plus API is exposed.
    port: 8080

    ## Add IPv4 IP/CIDR blocks to the allow list for NGINX stub_status or the NGINX Plus API. Separate multiple IP/CIDR by commas.
    allowCidrs: "127.0.0.1,::1"

prometheus:
  ## Expose NGINX or NGINX Plus metrics in the Prometheus format.
  create: true

  ## Configures the port to scrape the metrics.
  port: 9113

  ## Specifies the namespace/name of a Kubernetes TLS Secret which will be used to protect the Prometheus endpoint.
  secret: ""

  ## Configures the HTTP scheme used.
  scheme: http

extraContainers:
  - name: nginx-exporter
    image: nginx/nginx-prometheus-exporter:0.11.0
    args:
      - '-nginx.scrape-uri=http://nginx-gke-argocd-stg-nginx-ingress-controller:9113/metrics'
    resources:
      limits:
        memory: 128Mi
        cpu: 500m
    ports:
      - containerPort: 9113


kubectl get svc -n nginx-ingress
NAME                                            TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                      AGE
nginx-gke-argocd-stg-nginx-ingress-controller   LoadBalancer   10.215.1.108   34.86.221.10   80:32348/TCP,443:31387/TCP   4h44m

In the args i configured the svc of the nginx-ingress deploy and in the first place the :8080/stub_status and 9113/metrics but nothing works.

How can I make the exporter work? Thanks a lot for your help!!

[github-actions[bot]]

Hi @jcholowaty thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this 🙂

Cheers!

[brianehlert]

The prometheus exporter is built-in to the ingress controller (from this project) you don't need to use the exporter as a sidecar.
you just need to enable it.

[jholowaty]

@brianehlert Hi! thanks for your reply! I supose that but i can't figure out how scrape the metrics... I have enable this:

nginxStatus:
    ## Enable the NGINX stub_status, or the NGINX Plus API.
    enable: true

    ## Set the port where the NGINX stub_status or the NGINX Plus API is exposed.
    port: 8080

    ## Add IPv4 IP/CIDR blocks to the allow list for NGINX stub_status or the NGINX Plus API. Separate multiple IP/CIDR by commas.
    allowCidrs: "127.0.0.1"

---

prometheus:
  ## Expose NGINX or NGINX Plus metrics in the Prometheus format.
  create: true

  ## Configures the port to scrape the metrics.
  port: 9113

  ## Specifies the namespace/name of a Kubernetes TLS Secret which will be used to protect the Prometheus endpoint.
  secret: ""

  ## Configures the HTTP scheme used.
  scheme: http

---


kubectl get svc -n nginx-ingress
NAME                                            TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                      AGE
nginx-gke-argocd-stg-nginx-ingress-controller   LoadBalancer   10.215.1.108   34.86.221.10   80:32348/TCP,443:31387/TCP   21h
PYUYA3612:obs-stg juan.holowaty$ kubectl get all -n nginx-ingress
NAME                                                                 READY   STATUS    RESTARTS   AGE
pod/nginx-gke-argocd-stg-nginx-ingress-controller-798b89fcbf-pmqhs   1/1     Running   0          173m

NAME                                                    TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                      AGE
service/nginx-gke-argocd-stg-nginx-ingress-controller   LoadBalancer   10.215.1.108   34.86.221.10   80:32348/TCP,443:31387/TCP   21h

NAME                                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-gke-argocd-stg-nginx-ingress-controller   1/1     1            1           21h

NAME                                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-gke-argocd-stg-nginx-ingress-controller-798b89fcbf   1         1         1       19h

I don't see the port 9113 available or how can i scrape from prometheus?

[jjngx]

@jcholowaty have you followed this docs https://docs.nginx.com/nginx-ingress-controller/logging-and-monitoring/prometheus/ ?

a list of exposed metrics: https://github.com/nginxinc/nginx-prometheus-exporter#exported-metrics

Note: you need to configure NIC (at this step - https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/#4-deploying-nginx-ingress-controller) to allow Prometheus metrics before starting a new NGINX Ingress Controller. To configure NIC follow the first link in this comment.

kubernetes-ingress/deployments/deployment/nginx-plus-ingress.yaml

Lines 16 to 19 in bdb9d7f

	#annotations:
	#prometheus.io/scrape: "true"
	#prometheus.io/port: "9113"
	#prometheus.io/scheme: http

kubernetes-ingress/deployments/deployment/nginx-plus-ingress.yaml

Line 102 in bdb9d7f

#- -enable-prometheus-metrics

[jholowaty]

@jjngx Hi! Yes!! maybe i'm doing something wrong and i don't see it. in the values.yaml is all enable and i'm trying to config the scrape config like this:

• job_name: nginx
  scrape_interval: 5s
  scheme: "http"
  metrics_path: "/metrics"
  static_configs:
  • targets: ["10.215.1.108:9113"]

That ip is the ClusterIP but i tried with the External IP too.

And this with the external ip

• job_name: nginx
  scrape_interval: 5s
  scheme: "http"
  metrics_path: "/metrics"
  static_configs:
  • targets: ["34.86.221.10:9113"]

[brianehlert]

First of all, we would never want to expose your Prometheus exporter through an external load balancer. Even one you have restricted. The potential for exposure is too great.

Each NIC pod in the NIC service will expose the exporter on the port 9113 by default.
So you don't scrape the service, you scrape the individual pods.

I am assuming you have uncommented all of these from your values file and deployed or updated.
https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/command-line-arguments/#-enable-prometheus-metrics

Commonly, customers have a Prometheus agent installed in the cluster and it (very magically) discovers new exporters and begins collecting.
https://prometheus.io/blog/2021/11/16/agent/

[jorge815]

Em sex., 14 de jul. de 2023 às 17:24, Brian Ehlert ***@***.***> escreveu:

…

First of all, we would never want to expose your Prometheus exporter through an external load balancer. Even one you have restricted. The potential for exposure is too great. Each NIC pod in the NIC service will expose the exporter on the port 9113 by default. So you don't scrape the service, you scrape the individual pods. I am assuming you have uncommented all of these from your values file and deployed or updated. *https://docs.nginx.com/nginx-ingress-controller/configurariam /global-configuration/command-line-arguments/#-enable-prometheus-metrics* Commonly, customers have a Prometheus agent installed in the cluster and it (very magically) discovers new exporters and begins collecting. https://prometheus.io/blog/2021/11/16/agent/ — Reply to this email directly, view it on GitHub, or unsubscribe. You are [image: Documento do Google] Dicas para embarque <https://docs.google.com/document/d/1-EmLgkCtMoZEW7wHZ_HbGllw8gVBOXH_0G7cqkdWhQA/edit> receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jholowaty]

Hi! @brianehlert yes! i have uncommented tha principals values. I share my values.yaml

controller:
  ## The name of the Ingress Controller daemonset or deployment.
  name: controller

  ## The kind of the Ingress Controller installation - deployment or daemonset.
  kind: deployment

  ## Annotations for deployments and daemonsets
  annotations: {}

  ## Deploys the Ingress Controller for NGINX Plus.
  nginxplus: false

  # Timeout in milliseconds which the Ingress Controller will wait for a successful NGINX reload after a change or at the initial start.
  nginxReloadTimeout: 60000

  ## Support for App Protect WAF
  appprotect:
    ## Enable the App Protect WAF module in the Ingress Controller.
    enable: false
    ## Sets log level for App Protect WAF. Allowed values: fatal, error, warn, info, debug, trace
    # logLevel: fatal

  ## Support for App Protect DoS
  appprotectdos:
    ## Enable the App Protect DoS module in the Ingress Controller.
    enable: false
    ## Enable debugging for App Protect DoS.
    debug: false
    ## Max number of nginx processes to support.
    maxWorkers: 0
    ## Max number of ADMD instances.
    maxDaemons: 0
    ## RAM memory size to consume in MB.
    memory: 0

  ## Enables the Ingress Controller pods to use the host's network namespace.
  hostNetwork: false

  ## DNS policy for the Ingress Controller pods
  dnsPolicy: ClusterFirst

  ## Enables debugging for NGINX. Uses the nginx-debug binary. Requires error-log-level: debug in the ConfigMap via `controller.config.entries`.
  nginxDebug: false

  ## The log level of the Ingress Controller.
  logLevel: 1

  ## A list of custom ports to expose on the NGINX Ingress Controller pod. Follows the conventional Kubernetes yaml syntax for container ports.
  customPorts: []

  image:
    ## The image repository of the Ingress Controller.
    repository: nginx/nginx-ingress

    ## The tag of the Ingress Controller image. If not specified the appVersion from Chart.yaml is used as a tag.
    tag: "3.2.0"

    ## The digest of the Ingress Controller image.
    ## If digest is specified it has precedence over tag and will be used instead
    # digest: "sha256:CHANGEME"

    ## The pull policy for the Ingress Controller image.
    pullPolicy: IfNotPresent

  ## The lifecycle of the Ingress Controller pods.
  lifecycle: {}

  ## The custom ConfigMap to use instead of the one provided by default
  customConfigMap: ""

  config:
    ## The name of the ConfigMap used by the Ingress Controller.
    ## Autogenerated if not set or set to "".
    # name: nginx-config

    ## The annotations of the Ingress Controller configmap.
    annotations: {}

    ## The entries of the ConfigMap for customizing NGINX configuration.
    entries: {}

  ## It is recommended to use your own TLS certificates and keys
  defaultTLS:
    ## The base64-encoded TLS certificate for the default HTTPS server. By default, a pre-generated self-signed certificate is used.
    ## Note: It is recommended that you specify your own certificate. Alternatively, omitting the default server secret completely will configure NGINX to reject TLS connections to the default server.
    cert: ""

    ## The base64-encoded TLS key for the default HTTPS server. By default, a pre-generated key is used.
    ## Note: It is recommended that you specify your own key. Alternatively, omitting the default server secret completely will configure NGINX to reject TLS connections to the default server.
    key: ""

    ## The secret with a TLS certificate and key for the default HTTPS server.
    ## The value must follow the following format: `<namespace>/<name>`.
    ## Used as an alternative to specifying a certificate and key using `controller.defaultTLS.cert` and `controller.defaultTLS.key` parameters.
    ## Note: Alternatively, omitting the default server secret completely will configure NGINX to reject TLS connections to the default server.
    ## Format: <namespace>/<secret_name>
    secret: ""

  wildcardTLS:
    ## The base64-encoded TLS certificate for every Ingress/VirtualServer host that has TLS enabled but no secret specified.
    ## If the parameter is not set, for such Ingress/VirtualServer hosts NGINX will break any attempt to establish a TLS connection.
    cert: ""

    ## The base64-encoded TLS key for every Ingress/VirtualServer host that has TLS enabled but no secret specified.
    ## If the parameter is not set, for such Ingress/VirtualServer hosts NGINX will break any attempt to establish a TLS connection.
    key: ""

    ## The secret with a TLS certificate and key for every Ingress/VirtualServer host that has TLS enabled but no secret specified.
    ## The value must follow the following format: `<namespace>/<name>`.
    ## Used as an alternative to specifying a certificate and key using `controller.wildcardTLS.cert` and `controller.wildcardTLS.key` parameters.
    ## Format: <namespace>/<secret_name>
    secret: ""

  ## The node selector for pod assignment for the Ingress Controller pods.
  # nodeSelector: {}

  ## The termination grace period of the Ingress Controller pod.
  terminationGracePeriodSeconds: 30

  ## HorizontalPodAutoscaling (HPA)
  autoscaling:
    ## Enables HorizontalPodAutoscaling.
    enabled: false
    ## The annotations of the Ingress Controller HorizontalPodAutoscaler.
    annotations: {}
    ## Minimum number of replicas for the HPA.
    minReplicas: 1
    ## Maximum number of replicas for the HPA.
    maxReplicas: 3
    ## The target cpu utilization percentage.
    targetCPUUtilizationPercentage: 50
    ## The target memory utilization percentage.
    targetMemoryUtilizationPercentage: 50

  ## The resources of the Ingress Controller pods.
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
  # limits:
  #   cpu: 1
  #   memory: 1Gi

  ## The tolerations of the Ingress Controller pods.
  tolerations: []

  ## The affinity of the Ingress Controller pods.
  affinity: {}

  ## The topology spread constraints of the Ingress controller pods.
  # topologySpreadConstraints: {}

  ## The additional environment variables to be set on the Ingress Controller pods.
  env: []
  # - name: MY_VAR
  #   value: myvalue

  ## The volumes of the Ingress Controller pods.
  volumes: []
  # - name: extra-conf
  #   configMap:
  #     name: extra-conf

  ## The volumeMounts of the Ingress Controller pods.
  volumeMounts: []
  # - name: extra-conf
  #   mountPath: /etc/nginx/conf.d/extra.conf
  #   subPath: extra.conf

  ## InitContainers for the Ingress Controller pods.
  initContainers: []
  # - name: init-container
  #   image: busybox:1.34
  #   command: ['sh', '-c', 'echo this is initial setup!']

  ## The minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing, for it to be considered available.
  minReadySeconds: 0

  ## Pod disruption budget for the Ingress Controller pods.
  podDisruptionBudget:
    ## Enables PodDisruptionBudget.
    enabled: false
    ## The annotations of the Ingress Controller pod disruption budget.
    annotations: {}
    ## The number of Ingress Controller pods that should be available. This is a mutually exclusive setting with "maxUnavailable".
    # minAvailable: 1
    ## The number of Ingress Controller pods that can be unavailable. This is a mutually exclusive setting with "minAvailable".
    # maxUnavailable: 1

    ## Strategy used to replace old Pods by new ones. .spec.strategy.type can be "Recreate" or "RollingUpdate" for Deployments, and "OnDelete" or "RollingUpdate" for Daemonsets. "RollingUpdate" is the default value.
  strategy: {}

  ## Extra containers for the Ingress Controller pods.
  extraContainers: []
  # - name: container
  #   image: busybox:1.34
  #   command: ['sh', '-c', 'echo this is a sidecar!']

  # extraContainers:
  # - name: nginx-exporter
  #   image: nginx/nginx-prometheus-exporter:0.11.0
  #   args:
  #     - '-nginx.scrape-uri=http://nginx-gke-argocd-stg-nginx-ingress-controller:8080/stub_status'
  #   resources:
  #     limits:
  #       memory: 128Mi
  #       cpu: 500m
  #   ports:
  #     - containerPort: 9113
  
  ## The number of replicas of the Ingress Controller deployment.
  replicaCount: 1

  ## A class of the Ingress Controller.

  ## IngressClass resource with the name equal to the class must be deployed. Otherwise,
  ## the Ingress Controller will fail to start.
  ## The Ingress Controller only processes resources that belong to its class - i.e. have the "ingressClassName" field resource equal to the class.

  ## The Ingress Controller processes all the resources that do not have the "ingressClassName" field for all versions of kubernetes.
  ingressClass: nginx

  ## New Ingresses without an ingressClassName field specified will be assigned the class specified in `controller.ingressClass`.
  setAsDefaultIngress: false

  ## Comma separated list of namespaces to watch for Ingress resources. By default the Ingress Controller watches all namespaces. Mutually exclusive with "controller.watchNamespaceLabel".
  watchNamespace: ""

  ## Configures the Ingress Controller to watch only those namespaces with label foo=bar. By default the Ingress Controller watches all namespaces. Mutually exclusive with "controller.watchNamespace".
  watchNamespaceLabel: ""

  ## Comma separated list of namespaces to watch for Secret resources. By default the Ingress Controller watches all namespaces.
  watchSecretNamespace: ""

  ## Enable the custom resources.
  enableCustomResources: true

  ## Enable preview policies. This parameter is deprecated. To enable OIDC Policies please use controller.enableOIDC instead.
  enablePreviewPolicies: false

  ## Enable OIDC policies.
  enableOIDC: false

  ## Include year in log header. This parameter will be removed in release 2.7 and the year will be included by default.
  includeYear: false

  ## Enable TLS Passthrough on port 443. Requires controller.enableCustomResources.
  enableTLSPassthrough: false

  ## Enable cert manager for Virtual Server resources. Requires controller.enableCustomResources.
  enableCertManager: false

  ## Enable external DNS for Virtual Server resources. Requires controller.enableCustomResources.
  enableExternalDNS: false

  globalConfiguration:
    ## Creates the GlobalConfiguration custom resource. Requires controller.enableCustomResources.
    create: false

    ## The spec of the GlobalConfiguration for defining the global configuration parameters of the Ingress Controller.
    spec: {}
      # listeners:
      # - name: dns-udp
      #   port: 5353
      #   protocol: UDP
      # - name: dns-tcp
      #   port: 5353
      #   protocol: TCP

  ## Enable custom NGINX configuration snippets in Ingress, VirtualServer, VirtualServerRoute and TransportServer resources.
  enableSnippets: false

  ## Add a location based on the value of health-status-uri to the default server. The location responds with the 200 status code for any request.
  ## Useful for external health-checking of the Ingress Controller.
  healthStatus: true

  ## Sets the URI of health status location in the default server. Requires controller.healthStatus.
  healthStatusURI: "/nginx-health"

  nginxStatus:
    ## Enable the NGINX stub_status, or the NGINX Plus API.
    enable: true

    ## Set the port where the NGINX stub_status or the NGINX Plus API is exposed.
    port: 8080

    ## Add IPv4 IP/CIDR blocks to the allow list for NGINX stub_status or the NGINX Plus API. Separate multiple IP/CIDR by commas.
    allowCidrs: "127.0.0.1"

  service:
    ## Creates a service to expose the Ingress Controller pods.
    create: true

    ## The type of service to create for the Ingress Controller.
    type: LoadBalancer

    ## The externalTrafficPolicy of the service. The value Local preserves the client source IP.
    externalTrafficPolicy: Local

    ## The annotations of the Ingress Controller service.
    annotations: {}

    ## The extra labels of the service.
    extraLabels: {}

    ## The static IP address for the load balancer. Requires controller.service.type set to LoadBalancer. The cloud provider must support this feature.
    loadBalancerIP: ""

    ## The list of external IPs for the Ingress Controller service.
    externalIPs: []

    ## The IP ranges (CIDR) that are allowed to access the load balancer. Requires controller.service.type set to LoadBalancer. The cloud provider must support this feature.
    loadBalancerSourceRanges: []

    ## Whether to automatically allocate NodePorts (only for LoadBalancers).
    # allocateLoadBalancerNodePorts: false

    ## Dual stack preference.
    ## Valid values: SingleStack, PreferDualStack, RequireDualStack
    # ipFamilyPolicy: SingleStack

    ## List of IP families assigned to this service.
    ## Valid values: IPv4, IPv6
    # ipFamilies:
    #   - IPv6

    httpPort:
      ## Enables the HTTP port for the Ingress Controller service.
      enable: true

      ## The HTTP port of the Ingress Controller service.
      port: 80

      ## The custom NodePort for the HTTP port. Requires controller.service.type set to NodePort.
      # nodePort: 80

      ## The HTTP port on the POD where the Ingress Controller service is running.
      targetPort: 80

    httpsPort:
      ## Enables the HTTPS port for the Ingress Controller service.
      enable: true

      ## The HTTPS port of the Ingress Controller service.
      port: 443

      ## The custom NodePort for the HTTPS port. Requires controller.service.type set to NodePort.
      # nodePort: 443

      ## The HTTPS port on the POD where the Ingress Controller service is running.
      targetPort: 443

    ## A list of custom ports to expose through the Ingress Controller service. Follows the conventional Kubernetes yaml syntax for service ports.
    customPorts: []

  serviceAccount:
    ## The annotations of the service account of the Ingress Controller pods.
    annotations: {}

    ## The name of the service account of the Ingress Controller pods. Used for RBAC.
    ## Autogenerated if not set or set to "".
    # name: nginx-ingress

    ## The name of the secret containing docker registry credentials.
    ## Secret must exist in the same namespace as the helm release.
    imagePullSecretName: ""

  serviceMonitor:
    ## Creates a serviceMonitor to expose statistics on the kubernetes pods.
    create: false

    ## Kubernetes object labels to attach to the serviceMonitor object.
    labels: {}

    ## A set of labels to allow the selection of endpoints for the ServiceMonitor.
    selectorMatchLabels: {}

    ## A list of endpoints allowed as part of this ServiceMonitor.
    endpoints: []

  reportIngressStatus:
    ## Updates the address field in the status of Ingress resources with an external address of the Ingress Controller.
    ## You must also specify the source of the external address either through an external service via controller.reportIngressStatus.externalService,
    ## controller.reportIngressStatus.ingressLink or the external-status-address entry in the ConfigMap via controller.config.entries.
    ## Note: controller.config.entries.external-status-address takes precedence over the others.
    enable: true

    ## Specifies the name of the service with the type LoadBalancer through which the Ingress Controller is exposed externally.
    ## The external address of the service is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources.
    ## controller.reportIngressStatus.enable must be set to true.
    ## The default is autogenerated and matches the created service (see controller.service.create).
    # externalService: nginx-ingress

    ## Specifies the name of the IngressLink resource, which exposes the Ingress Controller pods via a BIG-IP system.
    ## The IP of the BIG-IP system is used when reporting the status of Ingress, VirtualServer and VirtualServerRoute resources.
    ## controller.reportIngressStatus.enable must be set to true.
    ingressLink: ""

    ## Enable Leader election to avoid multiple replicas of the controller reporting the status of Ingress resources. controller.reportIngressStatus.enable must be set to true.
    enableLeaderElection: true

    ## Specifies the name of the ConfigMap, within the same namespace as the controller, used as the lock for leader election. controller.reportIngressStatus.enableLeaderElection must be set to true.
    ## Autogenerated if not set or set to "".
    # leaderElectionLockName: "nginx-ingress-leader-election"

    ## The annotations of the leader election configmap.
    annotations: {}

  pod:
    ## The annotations of the Ingress Controller pod.
    annotations: {}

    ## The additional extra labels of the Ingress Controller pod.
    extraLabels: {}

  ## The PriorityClass of the Ingress Controller pods.
  # priorityClassName: ""

  readyStatus:
    ## Enables readiness endpoint "/nginx-ready". The endpoint returns a success code when NGINX has loaded all the config after startup.
    enable: true

    ## Set the port where the readiness endpoint is exposed.
    port: 8081

    ## The number of seconds after the Ingress Controller pod has started before readiness probes are initiated.
    initialDelaySeconds: 0

  ## Enable collection of latency metrics for upstreams. Requires prometheus.create.
  enableLatencyMetrics: false

  ## Disable IPV6 listeners explicitly for nodes that do not support the IPV6 stack.
  disableIPV6: false

  ## Configure root filesystem as read-only and add volumes for temporary data.
  readOnlyRootFilesystem: false

rbac:
  ## Configures RBAC.
  create: true

prometheus:
  ## Expose NGINX or NGINX Plus metrics in the Prometheus format.
  create: true

  ## Configures the port to scrape the metrics.
  port: 9113

  ## Specifies the namespace/name of a Kubernetes TLS Secret which will be used to protect the Prometheus endpoint.
  secret: ""

  ## Configures the HTTP scheme used.
  scheme: http

serviceInsight:
  ## Expose NGINX Plus Service Insight endpoint.
  create: false

  ## Configures the port to expose endpoint.
  port: 9114

  ## Specifies the namespace/name of a Kubernetes TLS Secret which will be used to protect the Service Insight endpoint.
  secret: ""

  ## Configures the HTTP scheme used.
  scheme: http

nginxServiceMesh:
  ## Enables integration with NGINX Service Mesh.
  enable: false

  ## Enables NGINX Service Mesh workload to route egress traffic through the Ingress Controller.
  ## Requires nginxServiceMesh.enable
  enableEgress: false

Even this i have not seen the port and i can't scrape it in prometheus. I'm using the kube-prometheus-stack.

[jholowaty]

@brianehlert and this is the pod's yaml.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "9113"
    prometheus.io/scheme: http
    prometheus.io/scrape: "true"
  creationTimestamp: "2023-07-17T10:29:02Z"
  generateName: nginx-gke-argocd-stg-nginx-ingress-controller-5c676bb95f-
  labels:
    app.kubernetes.io/instance: nginx-gke-argocd-stg
    app.kubernetes.io/name: nginx-ingress
    app.kubernetes.io/version: 3.2.0
    app.nginx.org/version: 1.25.1
    pod-template-hash: 5c676bb95f
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:prometheus.io/port: {}
          f:prometheus.io/scheme: {}
          f:prometheus.io/scrape: {}
        f:generateName: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/instance: {}
          f:app.kubernetes.io/name: {}
          f:pod-template-hash: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"41537475-3e18-4440-9241-b1c6449643c3"}: {}
      f:spec:
        f:automountServiceAccountToken: {}
        f:containers:
          k:{"name":"nginx-ingress"}:
            .: {}
            f:args: {}
            f:env:
              .: {}
              k:{"name":"POD_NAME"}:
                .: {}
                f:name: {}
                f:valueFrom:
                  .: {}
                  f:fieldRef: {}
              k:{"name":"POD_NAMESPACE"}:
                .: {}
                f:name: {}
                f:valueFrom:
                  .: {}
                  f:fieldRef: {}
            f:image: {}
            f:imagePullPolicy: {}
            f:name: {}
            f:ports:
              .: {}
              k:{"containerPort":80,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:name: {}
                f:protocol: {}
              k:{"containerPort":443,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:name: {}
                f:protocol: {}
              k:{"containerPort":8081,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:name: {}
                f:protocol: {}
              k:{"containerPort":9113,"protocol":"TCP"}:
                .: {}
                f:containerPort: {}
                f:name: {}
                f:protocol: {}
            f:readinessProbe:
              .: {}
              f:failureThreshold: {}
              f:httpGet:
                .: {}
                f:path: {}
                f:port: {}
                f:scheme: {}
              f:periodSeconds: {}
              f:successThreshold: {}
              f:timeoutSeconds: {}
            f:resources:
              .: {}
              f:requests:
                .: {}
                f:cpu: {}
                f:memory: {}
            f:securityContext:
              .: {}
              f:allowPrivilegeEscalation: {}
              f:capabilities:
                .: {}
                f:add: {}
                f:drop: {}
              f:readOnlyRootFilesystem: {}
              f:runAsNonRoot: {}
              f:runAsUser: {}
            f:terminationMessagePath: {}
            f:terminationMessagePolicy: {}
        f:dnsPolicy: {}
        f:enableServiceLinks: {}
        f:restartPolicy: {}
        f:schedulerName: {}
        f:securityContext:
          .: {}
          f:seccompProfile:
            .: {}
            f:type: {}
        f:serviceAccount: {}
        f:serviceAccountName: {}
        f:terminationGracePeriodSeconds: {}
    manager: kube-controller-manager
    operation: Update
    time: "2023-07-17T10:29:02Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:app.kubernetes.io/version: {}
          f:app.nginx.org/version: {}
    manager: nginx-ingress
    operation: Update
    time: "2023-07-17T10:29:08Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          k:{"type":"ContainersReady"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Initialized"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
          k:{"type":"Ready"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:status: {}
            f:type: {}
        f:containerStatuses: {}
        f:hostIP: {}
        f:phase: {}
        f:podIP: {}
        f:podIPs:
          .: {}
          k:{"ip":"10.214.1.13"}:
            .: {}
            f:ip: {}
        f:startTime: {}
    manager: kubelet
    operation: Update
    subresource: status
    time: "2023-07-17T10:29:09Z"
  name: nginx-gke-argocd-stg-nginx-ingress-controller-5c676bb95f-qgvqq
  namespace: nginx-ingress
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: nginx-gke-argocd-stg-nginx-ingress-controller-5c676bb95f
    uid: 41537475-3e18-4440-9241-b1c6449643c3
  resourceVersion: "4963981"
  uid: 23b27632-3dad-4cc8-b732-e9fe1485e720
spec:
  automountServiceAccountToken: true
  containers:
  - args:
    - -nginx-plus=false
    - -nginx-reload-timeout=60000
    - -enable-app-protect=false
    - -enable-app-protect-dos=false
    - -nginx-configmaps=$(POD_NAMESPACE)/nginx-gke-argocd-stg-nginx-ingress
    - -ingress-class=nginx
    - -health-status=true
    - -health-status-uri=/nginx-health
    - -nginx-debug=false
    - -v=1
    - -nginx-status=true
    - -nginx-status-port=8080
    - -nginx-status-allow-cidrs=127.0.0.1
    - -report-ingress-status
    - -external-service=nginx-gke-argocd-stg-nginx-ingress-controller
    - -enable-leader-election=true
    - -leader-election-lock-name=nginx-gke-argocd-stg-nginx-ingress-leader-election
    - -enable-prometheus-metrics=true
    - -prometheus-metrics-listen-port=9113
    - -prometheus-tls-secret=
    - -enable-service-insight=false
    - -service-insight-listen-port=9114
    - -service-insight-tls-secret=
    - -enable-custom-resources=true
    - -enable-snippets=false
    - -include-year=false
    - -disable-ipv6=false
    - -enable-tls-passthrough=false
    - -enable-preview-policies=false
    - -enable-cert-manager=false
    - -enable-oidc=false
    - -enable-external-dns=false
    - -ready-status=true
    - -ready-status-port=8081
    - -enable-latency-metrics=false
    env:
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    image: nginx/nginx-ingress:3.2.0
    imagePullPolicy: IfNotPresent
    name: nginx-ingress
    ports:
    - containerPort: 80
      name: http
      protocol: TCP
    - containerPort: 443
      name: https
      protocol: TCP
    - containerPort: 9113
      name: prometheus
      protocol: TCP
    - containerPort: 8081
      name: readiness-port
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /nginx-ready
        port: readiness-port
        scheme: HTTP
      periodSeconds: 1
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
        - NET_BIND_SERVICE
        drop:
        - ALL
      readOnlyRootFilesystem: false
      runAsNonRoot: true
      runAsUser: 101
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-wh7sb
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: gke-peya-data-platfo-apps-nodepool-e5-3d36ceab-i721
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: gke.io/optimize-utilization-scheduler
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  serviceAccount: nginx-gke-argocd-stg-nginx-ingress
  serviceAccountName: nginx-gke-argocd-stg-nginx-ingress
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-wh7sb
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-07-17T10:29:02Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-07-17T10:29:09Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-07-17T10:29:09Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-07-17T10:29:02Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://3fe927a28acec62a8c2be7fc07bd0dbfae12c8277d2ba5c1c1b24753c0c7fa7e
    image: docker.io/nginx/nginx-ingress:3.2.0
    imageID: docker.io/nginx/nginx-ingress@sha256:768ed715d7c87bc4bb9090dc1ac89eafaf509c318eea900a1f94951bc6152b1a
    lastState: {}
    name: nginx-ingress
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-07-17T10:29:08Z"
  hostIP: 10.213.0.55
  phase: Running
  podIP: 10.214.1.13
  podIPs:
  - ip: 10.214.1.13
  qosClass: Burstable
  startTime: "2023-07-17T10:29:02Z"

[jjngx]

@jcholowaty have you verified that you can access the prometheus endpoint (/metrics) and see all available metrics? When you follow the docs, please forward port 9113 to your local machine:
example:

kubectl port-forward -n nginx-ingress <nginx-ingress-pod-id> 9113:9113

Then open a web browser on localhost:9113/metrics. All available metrics can be accessed from there.