Histogram prometheus metrics unusable in production due to cardinality explosion

[kevinkokomani]

Describe the problem

There was an explosion of metrics from CRDB, causing a user to disable all histogram metrics as a result. Because histograms are not composable, every one of these metrics results in a new metric being created. And over time the binning changes, too, which further compounds the problem.

To Reproduce

Example of the issue:

# HELP exec_latency Latency of batch KV requests executed on this node
# TYPE exec_latency histogram
exec_latency_bucket{le="6143"} 1
exec_latency_bucket{le="6911"} 2
exec_latency_bucket{le="7167"} 20
exec_latency_bucket{le="7423"} 85
exec_latency_bucket{le="7679"} 247
exec_latency_bucket{le="7935"} 549
exec_latency_bucket{le="8191"} 1089
exec_latency_bucket{le="8703"} 3069
exec_latency_bucket{le="9215"} 6347
exec_latency_bucket{le="9727"} 10810
exec_latency_bucket{le="10239"} 15774
exec_latency_bucket{le="10751"} 20843
exec_latency_bucket{le="11263"} 25490
exec_latency_bucket{le="11775"} 29693

This represents the bucket distribution for a single node. Refreshing the stats page results in a different node with different binning for metrics.

# HELP exec_latency Latency of batch KV requests executed on this node
# TYPE exec_latency histogram
exec_latency_bucket{le="6655"} 1
exec_latency_bucket{le="6911"} 7
exec_latency_bucket{le="7167"} 41
exec_latency_bucket{le="7423"} 121
exec_latency_bucket{le="7679"} 304
exec_latency_bucket{le="7935"} 632
exec_latency_bucket{le="8191"} 1233
exec_latency_bucket{le="8703"} 3246
exec_latency_bucket{le="9215"} 6443
exec_latency_bucket{le="9727"} 10377
exec_latency_bucket{le="10239"} 14718
exec_latency_bucket{le="10751"} 19293
exec_latency_bucket{le="11263"} 23635
exec_latency_bucket{le="11775"} 27739
exec_latency_bucket{le="12287"} 31652
exec_latency_bucket{le="12799"} 35280

Expected behavior

Not a new metric to be created every time, and instead using log-linear histograms so that buckets are composable. Suggested links: https://github.com/openhistogram/circonusllhist & https://github.com/circonus-labs/circllhist-paper/blob/master/circllhist.pdf

CRDB versions: 20.2.x

Impact

Unable to monitor or alert on crdb with any metric that derives from a histogram, which is a large blindspot.

gz#8409

Jira issue: CRDB-7362

[dhartunian]

Collecting code trail of where the histogram is defined:

Metric defn: https://github.com/cockroachdb/cockroach/blob/master/pkg/server/node.go#L72-L77
Metrics impl: https://github.com/cockroachdb/cockroach/blob/master/pkg/server/node.go#L125
Histogram impl: https://github.com/cockroachdb/cockroach/blob/master/pkg/util/metric/metric.go#L199-L212

I think maybe a quick prototype of replacing the hdrhistogram implementation with the circonus one mentioned in the issue description should give us a sense of whether this will work as a drop-in replacement.

[ajwerner]

I'm pretty certain HDR histograms indeed are log-linear and are composable. The prometheus code just omits buckets which have the same value as the previous one and also empty buckets below the min. Encoding histograms this way is the prometheus norm. I feel like I'm also unsure of what you mean in terms of composability. See https://prometheus.io/docs/concepts/metric_types/#histogram.

I feel like maybe you're complaining about the configuration of our histograms, which, most of the time, is pretty dumb. It affords 1 sigFig all the way down to nanoseconds.

[ajwerner]

To clarify, I believe our histograms may have as many as 496 buckets, which is totally crazy!

https://play.golang.org/p/A3tUzhT_4cW

Given https://github.com/HdrHistogram/hdrhistogram-go/blob/master/hdr.go#L166

Note that our choice of max value and sig figs seems stupid in retrospect for many sql level things (min value of 0, and max value of 10s).

See also #36543.

FWIW I absolutely love this topic. Doing anything at all here is going to cause real migration pains and is going to be hard. I do not think this is a solved problem in the industry. I think we could consider migrating to better settings on an HDR histogram but we'll need to think very carefully about it.

In the much longer term, I dream of something like tdunning/t-digest#127 (comment)

[tbg]

@lunevalex I am not working on this - #81181 is something I have handed off to obs-inf for picking up.

@nkodali with #82200 we're going to add a bunch of histograms so we will likely have to get #81181 in before the end of the release, or our customers are going to see a ton more timeseries than before which may cause issues.

[tbg]

Another thing, I was now also personally hit by the high cardinality here. I wanted to have nice latency heatmaps as per this article but it actually crashes my browser (!) when I try to put it in Grafana. Plotting quantiles works though.