Skip to content

Commit

Permalink
implemnt Prometheus sink (#681)
Browse files Browse the repository at this point in the history
Signed-off-by: zirain <[email protected]>
  • Loading branch information
zirain authored Sep 16, 2024
1 parent 3a7d0e0 commit 28b1629
Show file tree
Hide file tree
Showing 8 changed files with 517 additions and 16 deletions.
104 changes: 104 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
- [DogStatsD](#dogstatsd)
- [Example](#example)
- [Continued example:](#continued-example)
- [Prometheus](#prometheus)
- [HTTP Port](#http-port)
- [/json endpoint](#json-endpoint)
- [Debug Port](#debug-port)
Expand Down Expand Up @@ -901,6 +902,109 @@ Then, declare additional rules for the `DESCRIPTOR` mogrifier
2. `DOG_STATSD_MOGRIFIER_HITS_NAME`: `ratelimit.service.rate_limit.$3`
3. `DOG_STATSD_MOGRIFIER_HITS_TAGS`: `domain:$1,descriptor:$2`
## Prometheus
To enable Prometheus integration set:
1. `USE_PROMETHEUS`: `true` to use [Prometheus](https://prometheus.io/)
2. `PROMETHEUS_ADDR`: The port to listen on for Prometheus metrics. Defaults to `:9090`
3. `PROMETHEUS_PATH`: The path to listen on for Prometheus metrics. Defaults to `/metrics`
4. `PROMETHEUS_MAPPER_YAML`: The path to the YAML file that defines the mapping from statsd to prometheus metrics.
Define the mapping from statsd to prometheus metrics in a YAML file.
Find more information about the mapping in the [Metric Mapping and Configuration](https://github.com/prometheus/statsd_exporter?tab=readme-ov-file#metric-mapping-and-configuration).
The default setting is:
```yaml
mappings: # Requires statsd exporter >= v0.6.0 since it uses the "drop" action.
- match: "ratelimit.service.rate_limit.*.*.near_limit"
name: "ratelimit_service_rate_limit_near_limit"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
- match: "ratelimit.service.rate_limit.*.*.over_limit"
name: "ratelimit_service_rate_limit_over_limit"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
- match: "ratelimit.service.rate_limit.*.*.total_hits"
name: "ratelimit_service_rate_limit_total_hits"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
- match: "ratelimit.service.rate_limit.*.*.within_limit"
name: "ratelimit_service_rate_limit_within_limit"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
- match: "ratelimit.service.rate_limit.*.*.*.near_limit"
name: "ratelimit_service_rate_limit_near_limit"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
key2: "$3"
- match: "ratelimit.service.rate_limit.*.*.*.over_limit"
name: "ratelimit_service_rate_limit_over_limit"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
key2: "$3"
- match: "ratelimit.service.rate_limit.*.*.*.total_hits"
name: "ratelimit_service_rate_limit_total_hits"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
key2: "$3"
- match: "ratelimit.service.rate_limit.*.*.*.within_limit"
name: "ratelimit_service_rate_limit_within_limit"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
key2: "$3"
- match: "ratelimit.service.call.should_rate_limit.*"
name: "ratelimit_service_should_rate_limit_error"
match_metric_type: counter
labels:
err_type: "$1"
- match: "ratelimit_server.*.total_requests"
name: "ratelimit_service_total_requests"
match_metric_type: counter
labels:
grpc_method: "$1"
- match: "ratelimit_server.*.response_time"
name: "ratelimit_service_response_time_seconds"
timer_type: histogram
labels:
grpc_method: "$1"
- match: "ratelimit.service.config_load_success"
name: "ratelimit_service_config_load_success"
match_metric_type: counter
- match: "ratelimit.service.config_load_error"
name: "ratelimit_service_config_load_error"
match_metric_type: counter
- match: "ratelimit.service.rate_limit.*.*.*.shadow_mode"
name: "ratelimit_service_rate_limit_shadow_mode"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
key2: "$3"
```
# HTTP Port
The ratelimit service listens to HTTP 1.1 (by default on port 8080) with two endpoints:
Expand Down
8 changes: 8 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ require (
github.com/lyft/goruntime v0.3.0
github.com/lyft/gostats v0.4.14
github.com/mediocregopher/radix/v3 v3.8.1
github.com/prometheus/client_golang v1.19.1
github.com/prometheus/statsd_exporter v0.26.1
github.com/sirupsen/logrus v1.9.3
github.com/stretchr/testify v1.9.0
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.53.0
Expand All @@ -37,19 +39,25 @@ require (
cel.dev/expr v0.15.0 // indirect
github.com/Microsoft/go-winio v0.5.0 // indirect
github.com/alicebob/gopher-json v0.0.0-20230218143504-906a9b012302 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/census-instrumentation/opencensus-proto v0.4.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cncf/xds/go v0.0.0-20240423153145-555b57ec207b // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/go-kit/log v0.2.1 // indirect
github.com/go-logfmt/logfmt v0.6.0 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0 // indirect
github.com/planetscale/vtprotobuf v0.5.1-0.20231212170721-e7d721933795 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_model v0.6.0 // indirect
github.com/prometheus/common v0.48.0 // indirect
github.com/prometheus/procfs v0.12.0 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/yuin/gopher-lua v1.1.1 // indirect
go.opentelemetry.io/otel/metric v1.28.0 // indirect
Expand Down
18 changes: 18 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ github.com/alicebob/gopher-json v0.0.0-20230218143504-906a9b012302/go.mod h1:SGn
github.com/alicebob/miniredis/v2 v2.33.0 h1:uvTF0EDeu9RLnUEG27Db5I68ESoIxTiXbNUiji6lZrA=
github.com/alicebob/miniredis/v2 v2.33.0/go.mod h1:MhP4a3EU7aENRi9aO+tHfTBZicLqQevyi/DJpoj6mi0=
github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/bradfitz/gomemcache v0.0.0-20230905024940-24af94b03874 h1:N7oVaKyGp8bttX0bfZGmcGkjz7DLQXhAn3DNd3T0ous=
github.com/bradfitz/gomemcache v0.0.0-20230905024940-24af94b03874/go.mod h1:r5xuitiExdLAJ09PR7vBVENGvp4ZuTBeWTGtxuX3K+c=
github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
Expand Down Expand Up @@ -42,7 +44,11 @@ github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4
github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA=
github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM=
github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY=
github.com/go-kit/log v0.2.1 h1:MRVx0/zhvdseW+Gza6N9rVzU/IVzaeE1SFI4raAhmBU=
github.com/go-kit/log v0.2.1/go.mod h1:NwTd00d/i8cPZ3xOwwiv2PO5MOcx78fFErGNcVmBjv0=
github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A=
github.com/go-logfmt/logfmt v0.6.0 h1:wGYYu3uicYdqXVgoYbvnkrPVXkuLM1p1ifugDMEdRi4=
github.com/go-logfmt/logfmt v0.6.0/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
Expand All @@ -51,6 +57,8 @@ github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE=
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.6.0 h1:ErTB+efbowRARo13NNdxyJji2egdxLGQhRaY+DUumQc=
github.com/golang/mock v1.6.0/go.mod h1:p6yTPP+5HYm5mzsMV8JkE6ZKdX+/wYM6Hr+LicevLPs=
Expand Down Expand Up @@ -101,7 +109,17 @@ github.com/planetscale/vtprotobuf v0.5.1-0.20231212170721-e7d721933795 h1:pH+U6p
github.com/planetscale/vtprotobuf v0.5.1-0.20231212170721-e7d721933795/go.mod h1:t/avpk3KcrXxUnYOhZhMXJlSEyie6gQbtLq5NM3loB8=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.19.1 h1:wZWJDwK+NameRJuPGDhlnFgx8e8HN3XHQeLaYJFJBOE=
github.com/prometheus/client_golang v1.19.1/go.mod h1:mP78NwGzrVks5S2H6ab8+ZZGJLZUq1hoULYBAYBw1Ho=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/client_model v0.6.0 h1:k1v3CzpSRUTrKMppY35TLwPvxHqBu0bYgxZzqGIgaos=
github.com/prometheus/client_model v0.6.0/go.mod h1:NTQHnmxFpouOD0DpvP4XujX3CdOAGQPoaGhyTchlyt8=
github.com/prometheus/common v0.48.0 h1:QO8U2CdOzSn1BBsmXJXduaaW+dY/5QLjfB8svtSzKKE=
github.com/prometheus/common v0.48.0/go.mod h1:0/KsvlIEfPQCQ5I2iNSAWKPZziNCvRs5EC6ILDTlAPc=
github.com/prometheus/procfs v0.12.0 h1:jluTpSng7V9hY0O2R9DzzJHYb2xULk9VTR1V1R/k6Bo=
github.com/prometheus/procfs v0.12.0/go.mod h1:pcuDEFsWDnvcgNzo4EEweacyhjeA9Zk3cnaOZAZEfOo=
github.com/prometheus/statsd_exporter v0.26.1 h1:ucbIAdPmwAUcA+dU+Opok8Qt81Aw8HanlO+2N/Wjv7w=
github.com/prometheus/statsd_exporter v0.26.1/go.mod h1:XlDdjAmRmx3JVvPPYuFNUg+Ynyb5kR69iPPkQjxXFMk=
github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8=
github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4=
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
Expand Down
39 changes: 23 additions & 16 deletions src/service_cmd/runner/runner.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,25 +9,22 @@ import (
"sync"
"time"

"github.com/envoyproxy/ratelimit/src/godogstats"
"github.com/envoyproxy/ratelimit/src/metrics"
"github.com/envoyproxy/ratelimit/src/stats"
"github.com/envoyproxy/ratelimit/src/trace"

gostats "github.com/lyft/gostats"

"github.com/coocood/freecache"

pb "github.com/envoyproxy/go-control-plane/envoy/service/ratelimit/v3"

gostats "github.com/lyft/gostats"
logger "github.com/sirupsen/logrus"

"github.com/envoyproxy/ratelimit/src/godogstats"
"github.com/envoyproxy/ratelimit/src/limiter"
"github.com/envoyproxy/ratelimit/src/memcached"
"github.com/envoyproxy/ratelimit/src/metrics"
"github.com/envoyproxy/ratelimit/src/redis"
"github.com/envoyproxy/ratelimit/src/server"
ratelimit "github.com/envoyproxy/ratelimit/src/service"
"github.com/envoyproxy/ratelimit/src/settings"
"github.com/envoyproxy/ratelimit/src/stats"
"github.com/envoyproxy/ratelimit/src/stats/prom"
"github.com/envoyproxy/ratelimit/src/trace"
"github.com/envoyproxy/ratelimit/src/utils"
)

Expand All @@ -42,14 +39,14 @@ type Runner struct {
func NewRunner(s settings.Settings) Runner {
var store gostats.Store

if s.DisableStats {
switch {
case s.DisableStats:
logger.Info("Stats disabled")
store = gostats.NewStore(gostats.NewNullSink(), false)
} else if s.UseDogStatsd {
if s.UseStatsd {
logger.Fatalf("Error: unable to use both stats sink at the same time. Set either USE_DOG_STATSD or USE_STATSD but not both.")
case s.UseDogStatsd:
if s.UseStatsd || s.UsePrometheus {
logger.Fatalf("Error: unable to use more than one stats sink at the same time. Set one of USE_DOG_STATSD, USE_STATSD, USE_PROMETHEUS.")
}
var err error
sink, err := godogstats.NewSink(
godogstats.WithStatsdHost(s.StatsdHost),
godogstats.WithStatsdPort(s.StatsdPort),
Expand All @@ -59,10 +56,20 @@ func NewRunner(s settings.Settings) Runner {
}
logger.Info("Stats initialized for dogstatsd")
store = gostats.NewStore(sink, false)
} else if s.UseStatsd {
case s.UseStatsd:
if s.UseDogStatsd || s.UsePrometheus {
logger.Fatalf("Error: unable to use more than one stats sink at the same time. Set one of USE_DOG_STATSD, USE_STATSD, USE_PROMETHEUS.")
}
logger.Info("Stats initialized for statsd")
store = gostats.NewStore(gostats.NewTCPStatsdSink(gostats.WithStatsdHost(s.StatsdHost), gostats.WithStatsdPort(s.StatsdPort)), false)
} else {
case s.UsePrometheus:
if s.UseDogStatsd || s.UseStatsd {
logger.Fatalf("Error: unable to use more than one stats sink at the same time. Set one of USE_DOG_STATSD, USE_STATSD, USE_PROMETHEUS.")
}
logger.Info("Stats initialized for Prometheus")
store = gostats.NewStore(prom.NewPrometheusSink(prom.WithAddr(s.PrometheusAddr),
prom.WithPath(s.PrometheusPath), prom.WithMapperYamlPath(s.PrometheusMapperYaml)), false)
default:
logger.Info("Stats initialized for stdout")
store = gostats.NewStore(gostats.NewLoggingSink(), false)
}
Expand Down
4 changes: 4 additions & 0 deletions src/settings/settings.go
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,10 @@ type Settings struct {
ExtraTags map[string]string `envconfig:"EXTRA_TAGS" default:""`
StatsFlushInterval time.Duration `envconfig:"STATS_FLUSH_INTERVAL" default:"10s"`
DisableStats bool `envconfig:"DISABLE_STATS" default:"false"`
UsePrometheus bool `envconfig:"USE_PROMETHEUS" default:"false"`
PrometheusAddr string `envconfig:"PROMETHEUS_ADDR" default:":9090"`
PrometheusPath string `envconfig:"PROMETHEUS_PATH" default:"/metrics"`
PrometheusMapperYaml string `envconfig:"PROMETHEUS_MAPPER_YAML" default:""`

// Settings for rate limit configuration
RuntimePath string `envconfig:"RUNTIME_ROOT" default:"/srv/runtime_data/current"`
Expand Down
89 changes: 89 additions & 0 deletions src/stats/prom/default_mapper.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Requires statsd exporter >= v0.6.0 since it uses the "drop" action.
mappings:
- match: "ratelimit.service.rate_limit.*.*.near_limit"
name: "ratelimit_service_rate_limit_near_limit"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
- match: "ratelimit.service.rate_limit.*.*.over_limit"
name: "ratelimit_service_rate_limit_over_limit"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
- match: "ratelimit.service.rate_limit.*.*.total_hits"
name: "ratelimit_service_rate_limit_total_hits"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
- match: "ratelimit.service.rate_limit.*.*.within_limit"
name: "ratelimit_service_rate_limit_within_limit"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"

- match: "ratelimit.service.rate_limit.*.*.*.near_limit"
name: "ratelimit_service_rate_limit_near_limit"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
key2: "$3"
- match: "ratelimit.service.rate_limit.*.*.*.over_limit"
name: "ratelimit_service_rate_limit_over_limit"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
key2: "$3"
- match: "ratelimit.service.rate_limit.*.*.*.total_hits"
name: "ratelimit_service_rate_limit_total_hits"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
key2: "$3"
- match: "ratelimit.service.rate_limit.*.*.*.within_limit"
name: "ratelimit_service_rate_limit_within_limit"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
key2: "$3"

- match: "ratelimit.service.call.should_rate_limit.*"
name: "ratelimit_service_should_rate_limit_error"
match_metric_type: counter
labels:
err_type: "$1"

- match: "ratelimit_server.*.total_requests"
name: "ratelimit_service_total_requests"
match_metric_type: counter
labels:
grpc_method: "$1"

- match: "ratelimit_server.*.response_time"
name: "ratelimit_service_response_time_seconds"
timer_type: histogram
labels:
grpc_method: "$1"

- match: "ratelimit.service.config_load_success"
name: "ratelimit_service_config_load_success"
match_metric_type: counter

- match: "ratelimit.service.config_load_error"
name: "ratelimit_service_config_load_error"
match_metric_type: counter

- match: "ratelimit.service.rate_limit.*.*.*.shadow_mode"
name: "ratelimit_service_rate_limit_shadow_mode"
timer_type: "histogram"
labels:
domain: "$1"
key1: "$2"
key2: "$3"
Loading

0 comments on commit 28b1629

Please sign in to comment.