Skip to content

Latest commit

 

History

History
476 lines (376 loc) · 23.5 KB

prometheus.md

File metadata and controls

476 lines (376 loc) · 23.5 KB
title keywords description
prometheus
Apache APISIX
API Gateway
Plugin
Prometheus
The prometheus Plugin provides the capability to integrate APISIX with Prometheus for metric collection and continuous monitoring.

Description

The prometheus Plugin provides the capability to integrate APISIX with Prometheus.

After enabling the Plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a text-based exposition format to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.

Static Configurations

By default, prometheus configurations are pre-configured in the default configuration.

To customize these values, add the corresponding configurations to config.yaml. For example:

plugin_attr:
  prometheus:                               # Plugin: prometheus attributes
    export_uri: /apisix/prometheus/metrics  # Set the URI for the Prometheus metrics endpoint.
    metric_prefix: apisix_                  # Set the prefix for Prometheus metrics generated by APISIX.
    enable_export_server: true              # Enable the Prometheus export server.
    export_addr:                            # Set the address for the Prometheus export server.
      ip: 127.0.0.1                         # Set the IP.
      port: 9091                            # Set the port.
    # metrics:                              # Create extra labels for metrics.
    #  http_status:                         # These metrics will be prefixed with `apisix_`.
    #    extra_labels:                      # Set the extra labels for http_status metrics.
    #      - upstream_addr: $upstream_addr
    #      - status: $upstream_status
    #    expire: 0                          # The expiration time of metrics in seconds.
                                            # 0 means the metrics will not expire.
    #  http_latency:
    #    extra_labels:                      # Set the extra labels for http_latency metrics.
    #      - upstream_addr: $upstream_addr
    #    expire: 0                          # The expiration time of metrics in seconds.
                                            # 0 means the metrics will not expire.
    #  bandwidth:
    #    extra_labels:                      # Set the extra labels for bandwidth metrics.
    #      - upstream_addr: $upstream_addr
    #    expire: 0                          # The expiration time of metrics in seconds.
                                            # 0 means the metrics will not expire.
    # default_buckets:                      # Set the default buckets for the `http_latency` metrics histogram.
    #   - 10
    #   - 50
    #   - 100
    #   - 200
    #   - 500
    #   - 1000
    #   - 2000
    #   - 5000
    #   - 10000
    #   - 30000
    #   - 60000
    #   - 500

You can use the Nginx variable to create extra_labels. See add extra labels.

Reload APISIX for changes to take effect.

Attribute

Name Type Required Default Valid values Description
prefer_name boolean False If true, export Route/Service name instead of their ID in Prometheus metrics.

Metrics

There are different types of metrics in Prometheus. To understand their differences, see metrics types.

The following metrics are exported by the prometheus Plugin by default. See get APISIX metrics for an example. Note that some metrics, such as apisix_batch_process_entries, are not readily visible if there are no data.

Name Type Description
apisix_bandwidth counter Total amount of traffic flowing through APISIX in bytes.
apisix_etcd_modify_indexes gauge Number of changes to etcd by APISIX keys.
apisix_batch_process_entries gauge Number of remaining entries in a batch when sending data in batches, such as with http logger, and other logging Plugins.
apisix_etcd_reachable gauge Whether APISIX can reach etcd. A value of 1 represents reachable and 0 represents unreachable.
apisix_http_status counter HTTP status codes returned from upstream Services.
apisix_http_requests_total gauge Number of HTTP requests from clients.
apisix_nginx_http_current_connections gauge Number of current connections with clients.
apisix_nginx_metric_errors_total counter Total number of nginx-lua-prometheus errors.
apisix_http_latency histogram HTTP request latency in milliseconds.
apisix_node_info gauge Information of the APISIX node, such as host name.
apisix_shared_dict_capacity_bytes gauge The total capacity of an NGINX shared dictionary.
apisix_shared_dict_free_space_bytes gauge The remaining space in an NGINX shared dictionary.
apisix_upstream_status gauge Health check status of upstream nodes, available if health checks are configured on the upstream. A value of 1 represents healthy and 0 represents unhealthy.
apisix_stream_connection_total counter Total number of connections handled per Stream Route.

Labels

Labels are attributes of metrics that are used to differentiate metrics.

For example, the apisix_http_status metric can be labeled with route information to identify which Route the HTTP status originates from.

The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.

Labels for apisix_http_status

The following labels are used to differentiate apisix_http_status metrics.

Name Description
code HTTP response code returned by the upstream node.
route ID of the Route that the HTTP status originates from when prefer_name is false (default), and name of the Route when prefer_name to true. Default to an empty string if a request does not match any Route.
matched_uri URI of the Route that matches the request. Default to an empty string if a request does not match any Route.
matched_host Host of the Route that matches the request. Default to an empty string if a request does not match any Route, or host is not configured on the Route.
service ID of the Service that the HTTP status originates from when prefer_name is false (default), and name of the Service when prefer_name to true. Default to the configured value of host on the Route if the matched Route does not belong to any Service.
consumer Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request.
node IP address of the upstream node.

Labels for apisix_bandwidth

The following labels are used to differentiate apisix_bandwidth metrics.

Name Description
type Type of traffic, egress or ingress.
route ID of the Route that bandwidth corresponds to when prefer_name is false (default), and name of the Route when prefer_name to true. Default to an empty string if a request does not match any Route.
service ID of the Service that bandwidth corresponds to when prefer_name is false (default), and name of the Service when prefer_name to true. Default to the configured value of host on the Route if the matched Route does not belong to any Service.
consumer Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request.
node IP address of the upstream node.

Labels for apisix_http_latency

The following labels are used to differentiate apisix_http_latency metrics.

Name Description
type Type of latencies. See latency types for details.
route ID of the Route that latencies correspond to when prefer_name is false (default), and name of the Route when prefer_name to true. Default to an empty string if a request does not match any Route.
service ID of the Service that latencies correspond to when prefer_name is false (default), and name of the Service when prefer_name to true. Default to the configured value of host on the Route if the matched Route does not belong to any Service.
consumer Name of the Consumer associated with latencies. Default to an empty string if no Consumer is associated with the request.
node IP address of the upstream node associated with latencies.

Latency Types

apisix_http_latency can be labeled with one of the three types:

  • request represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.

  • upstream represents the time elapsed waiting on responses from the upstream Service.

  • apisix represents the difference between the request latency and upstream latency.

In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:

APISIX latency
  = downstream request time - upstream response time
  = downstream traffic latency + NGINX latency

Labels for apisix_upstream_status

The following labels are used to differentiate apisix_upstream_status metrics.

Name Description
name Resource ID corresponding to the upstream configured with health checks, such as /apisix/routes/1 and /apisix/upstreams/1.
ip IP address of the upstream node.
port Port number of the node.

Examples

The examples below demonstrate how you can work with the prometheus Plugin for different scenarios.

Get APISIX Metrics

The following example demonstrates how you can get metrics from APISIX.

The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the static configuration. If you would like to customize these configuration, update config.yaml and reload APISIX.

If you deploy APISIX in a containerized environment and would like to access the Prometheus metrics endpoint externally, update the configuration file as follows and reload APISIX:

plugin_attr:
  prometheus:
    export_addr:
      ip: 0.0.0.0

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_bandwidth Total bandwidth in bytes consumed per Service in Apisix
# TYPE apisix_bandwidth counter
apisix_bandwidth{type="egress",route="",service="",consumer="",node=""} 8417
apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1"} 1420
apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1"} 1420
apisix_bandwidth{type="ingress",route="",service="",consumer="",node=""} 189
apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1"} 332
apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1"} 332
# HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys
# TYPE apisix_etcd_modify_indexes gauge
apisix_etcd_modify_indexes{key="consumers"} 0
apisix_etcd_modify_indexes{key="global_rules"} 0
...

Expose APISIX Metrics on Public API Endpoint

The following example demonstrates how you can disable the Prometheus export server that, by default, exposes an endpoint on port 9091, and expose APISIX Prometheus metrics on a new public API endpoint on port 9080, which APISIX uses to listen to other client requests.

Disable the Prometheus export server in the configuration file and reload APISIX for changes to take effect:

plugin_attr:
  prometheus:
    enable_export_server: false

Next, create a Route with public-api Plugin and expose a public API endpoint for APISIX metrics:

curl "http://127.0.0.1:9180/apisix/admin/routes/prometheus-metrics" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "uri": "/apisix/prometheus/metrics",
    "plugins": {
      "public-api": {}
    }
  }'

Send a request to the new metrics endpoint to verify:

curl "http://127.0.0.1:9080/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_http_requests_total The total number of client requests since APISIX started
# TYPE apisix_http_requests_total gauge
apisix_http_requests_total 1
# HELP apisix_nginx_http_current_connections Number of HTTP connections
# TYPE apisix_nginx_http_current_connections gauge
apisix_nginx_http_current_connections{state="accepted"} 1
apisix_nginx_http_current_connections{state="active"} 1
apisix_nginx_http_current_connections{state="handled"} 1
apisix_nginx_http_current_connections{state="reading"} 0
apisix_nginx_http_current_connections{state="waiting"} 0
apisix_nginx_http_current_connections{state="writing"} 1
...

Monitor Upstream Health Statuses

The following example demonstrates how to monitor the health status of upstream nodes.

Create a Route with the prometheus Plugin and configure upstream active health checks:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "id": "prometheus-route",
    "uri": "/get",
    "plugins": {
      "prometheus": {}
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1,
        "127.0.0.1:20001": 1
      },
      "checks": {
        "active": {
          "timeout": 5,
          "http_path": "/status",
          "healthy": {
            "interval": 2,
            "successes": 1
          },
          "unhealthy": {
            "interval": 1,
            "http_failures": 2
          }
        },
        "passive": {
          "healthy": {
            "http_statuses": [200, 201],
            "successes": 3
          },
          "unhealthy": {
            "http_statuses": [500],
            "http_failures": 3,
            "tcp_failures": 3
          }
        }
      }
    }
  }'

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_upstream_status upstream status from health check
# TYPE apisix_upstream_status gauge
apisix_upstream_status{name="/apisix/routes/1",ip="54.237.103.220",port="80"} 1
apisix_upstream_status{name="/apisix/routes/1",ip="127.0.0.1",port="20001"} 0

This shows that the upstream node httpbin.org:80 is healthy and the upstream node 127.0.0.1:20001 is unhealthy.

Add Extra Labels for Metrics

The following example demonstrates how to add additional labels to metrics and use the Nginx variable in label values.

Currently, only the following metrics support extra labels:

  • apisix_http_status
  • apisix_http_latency
  • apisix_bandwidth

Include the following configurations in the configuration file to add labels for metrics and reload APISIX for changes to take effect:

plugin_attr:
  prometheus:                                # Plugin: prometheus
    metrics:                                 # Create extra labels from the NGINX variables.
      http_status:
        extra_labels:                        # Set the extra labels for http_status metrics.
          - upstream_addr: $upstream_addr    # Add an extra upstream_addr label with value being the NGINX variable $upstream_addr.
          - route_name: $route_name          # Add an extra route_name label with value being the APISIX variable $route_name.

Note that if you define a variable in the label value but it does not correspond to any existing APISIX variables and Nginx variable, the label value will default to an empty string.

Create a Route with the prometheus Plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "id": "prometheus-route",
Include the following configurations in the configuration file to add labels for metrics and reload APISIX for changes to take effect:
    "name": "extra-label",
    "plugins": {
      "prometheus": {}
    },
    "upstream": {
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

Send a request to the Route to verify:

curl -i "http://127.0.0.1:9080/get"

You should see an HTTP/1.1 200 OK response.

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_http_status HTTP status codes per Service in APISIX
# TYPE apisix_http_status counter
apisix_http_status{code="200",route="1",matched_uri="/get",matched_host="",service="",consumer="",node="54.237.103.220",upstream_addr="54.237.103.220:80",route_name="extra-label"} 1

Monitor TCP/UDP Traffic with Prometheus

The following example demonstrates how to collect TCP/UDP traffic metrics in APISIX.

Include the following configurations in config.yaml to enable stream proxy and prometheus Plugin for stream proxy. Reload APISIX for changes to take effect:

apisix:
  proxy_mode: http&stream   # Enable both L4 & L7 proxies
  stream_proxy:             # Configure L4 proxy
    tcp:
      - 9100                # Set TCP proxy listening port
    udp:
      - 9200                # Set UDP proxy listening port

stream_plugins:
  - prometheus              # Enable prometheus for stream proxy

Create a Stream Route with the prometheus Plugin:

curl "http://127.0.0.1:9180/apisix/admin/stream_routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
Include the following configurations in `config.yaml` to enable stream proxy and enable `prometheus` Plugin for stream proxy. Reload APISIX for changes to take effect:
    "plugins": {
      "prometheus":{}
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

Send a request to the Stream Route to verify:

curl -i "http://127.0.0.1:9100"

You should see an HTTP/1.1 200 OK response.

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_stream_connection_total Total number of connections handled per Stream Route in APISIX
# TYPE apisix_stream_connection_total counter
apisix_stream_connection_total{route="1"} 1