prometheus_express

A Prometheus SDK for CircuitPython and MicroPython boards, helping integrate I2C and SPI sensors into existing Prometheus monitoring infrastructure.

• only depends on socket
• runs on CircuitPython 4.x and MicroPython for ESP32 and M0/M4 devices
• runs even faster on CPython 3.x for local testing
• API compatible with the official prometheus/client_python
• basic HTTP server with path/method routing
• not terribly slow (20 rps on an ESP32, 50 rps on an M4 core, 5k on an i7 core)

For those unfamiliar with Prometheus, the examples expose an HTTP server on port :8080 that serves /metrics in a readable, plaintext format:

# HELP prom_express_gas gas from the bme680 sensor
# TYPE prom_express_gas gauge
prom_express_gas 1060948
# HELP prom_express_humidity humidity from both sensors
# TYPE prom_express_humidity gauge
prom_express_humidity{sensor="None"} 0
prom_express_humidity{sensor="bme680"} 49.4062
prom_express_humidity{sensor="si7021"} 49.7976
# HELP prom_express_pressure pressure from the bme680 sensor
# TYPE prom_express_pressure gauge
prom_express_pressure 983.25
# HELP prom_express_temperature temperature from both sensors
# TYPE prom_express_temperature gauge
prom_express_temperature{sensor="None"} 0
prom_express_temperature{sensor="bme680"} 24.7359
prom_express_temperature{sensor="si7021"} 24.3325

The Prometheus server will occasionally scrape each device, collecting sensor readings with metadata about their source, and passing the samples on to long-term storage.

Contents

• prometheus_express
  • Contents
  • Status
  • Releases
  • Supported Hardware
  • Supported Features
    • HTTP Server
    • Metric Labels
    • Metric Types
      • Counter
      • Gauge
      • Summary
    • Registries
  • Planned Features
  • Known Issues
    • OSError 3, 4, or 7
    • OSError 112

Status

Releases

Supported Hardware

This library is tested on:

• the Adafruit Feather M4 Express running MicroPython 4.1.0 or better, with an Adafruit Ethernet FeatherWing attached (using the Wiznet5500 driver).
• the Olimex ESP32-POE running CircuitPython 1.12.0 or better, with wifi or wired ethernet (using the LAN8720 driver)

Supported Features

HTTP Server

This module implements a very rudimentary HTTP server that almost certainly violates some part of the spec. However, it works with Chrome, curl, and Prometheus itself. Depending on the platform, it may also work with wrk benchmarks (known issues).

Metric Labels

Labels are stored and used to accumulate values. Missing labels are reported as None.

If a metric is constructed with labels, a default entry with all values set to None will also be reported.

Metric Types

Counter, Gauge, and Summary are implemented with labels.

Counter

Incremental values. Implements inc(value) and dec(value).

Gauge

Absolute values. Extends counter with set(value).

Summary

Individual values. Prints count and total of observe(value).

Registries

Registries may be created with a namespace: CollectorRegistry(namespace='foo')

Call registry.render() to format metrics in Prometheus' plain text exposition format:

# HELP prom_express_test_counter a test counter
# TYPE prom_express_test_counter counter
prom_express_test_counter 1588
# HELP prom_express_test_gauge a test gauge
# TYPE prom_express_test_gauge gauge
prom_express_test_gauge 3887

Metrics may be registered with multiple registries.

Planned Features

• push support: push_to_gateway
• remaining metric types: Histogram
• tests for API compatibility

Known Issues

OSError 3, 4, or 7

Load testing the HTTP endpoint may cause one of a variety of OSErrors, often errno 3, 4, or 7.

This is related to the Wiznet5500 driver for ethernet FeatherWings, and may not appear on other devices:

> ./wrk -c 1 -d 60s -t 1 http://server:8080/
Running 1m test @ http://server:8080/
  1 threads and 1 connections
    Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.64ms  485.57us  12.81ms   97.75%
    Req/Sec   111.60      5.21   121.00     71.00%
  2222 requests in 20.61s, 671.83KB read
  Socket errors: connect 0, read 1, write 2743, timeout 0
Requests/sec:    107.82
Transfer/sec:     32.60KB

Some are fatal:

Connection from ('client', 8080)
Accepting...
Connection from ('client', 8080)
Accepting...
Error accepting request: [Errno 128] ENOTCONN
Binding: server:8080
Traceback (most recent call last):
  File "code.py", line 90, in <module>
  File "code.py", line 87, in main
  File "code.py", line 87, in main
  File "code.py", line 55, in bind
  File "/lib/prometheus_express/http.py", line 11, in start_http_server
OSError: 4

Others require the socket to be rebound:

Connection from ('client', 8080)
Accepting...
Error accepting request: 7
Binding: server:8080
Accepting...

OSError 112

Certain crashes may leak open sockets, causing the device to log an OSError 112 during startup.

The error prevents the HTTP server from binding to the ethernet device, so an HTTP watchdog on the switch can be used to restart POE devices. The board can also be reset by calling machine.reset() from the REPL.