From d74f88ab9fc0bf6fa0fccef462201a9e54500aea Mon Sep 17 00:00:00 2001 From: Chris Marchbanks Date: Mon, 29 Jan 2024 11:29:12 -0700 Subject: [PATCH 1/9] Proposal for Native Histogram Text Format Create an initial prosal for supporting native histograms in the text format. It mainly links to the design doc for ease of comments while we decide on a path forward. After a proposal from the design doc has some support the rest of the markdown proposal will be filled out. Signed-off-by: Chris Marchbanks --- ...024-01-29_native_histograms_text_format.md | 60 +++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 proposals/2024-01-29_native_histograms_text_format.md diff --git a/proposals/2024-01-29_native_histograms_text_format.md b/proposals/2024-01-29_native_histograms_text_format.md new file mode 100644 index 0000000..ca58d18 --- /dev/null +++ b/proposals/2024-01-29_native_histograms_text_format.md @@ -0,0 +1,60 @@ +## Native Histograms Text Format + +* **Owners:** + * Chris Marchbanks [@csmarchbanks](https://github.com/csmarchbanks) + +* **Implementation Status:** Not implemented + +* **Related Issues and PRs:** + * `` + +* **Other docs or links:** + * [Design Doc for choosing a proposal](https://docs.google.com/document/d/1qoHf24cKMpa1QHskIjgzyf3oFhIPvacyJj8Tbe6fIrY/edit#heading=h.5sybau7waq2q) + +> TL;DR: This design doc is proposing a format for exposing native histograms in the OpenMetrics text format. + +## Why + +Today it is only possible to export native histograms using the protobuf scrape format. Many users prefer the text format, and some client libraries, such as the Python client, want to avoid adding a dependency on protobuf. + +During a [dev summit in 2022](https://docs.google.com/document/d/11LC3wJcVk00l8w5P3oLQ-m3Y37iom6INAMEu2ZAGIIE/edit#bookmark=id.c3e7ur6rn5d2) there was consensus we would continue to support the text format. Including native histograms as part of the text format shows commitment to that consensus. + +See the linked design doc in Google Docs for additional background information. + +### Pitfalls of the current solution + +Prometheus client libraries such as Python do not want to require a dependency on protobuf in order to expose native histograms, and in some languages protobuf is painful to use. Gating native histograms only to clients/users willing to use protobuf hurts adoption of native histograms, therefore, we would like a way to represent a histogram in the text based format. + +## Goals + +Goals and use cases for the solution as proposed in [How](#how): + +* Support native histograms in the text format +* (Secondary) Encode/decode efficiency +* (Secondary) Ease of implementation for client libraries +* (Secondary) Human readibility of the format + +Note that the goals of efficiency and human readability are commonly at odds with each other. + +### Audience + +Client library maintainers, OpenMetrics, and Prometheus scrape maintainers. + +## Non-Goals + +* Requiring backwards compatability (OpenMetrics 2.0 would be ok) + +## How + +To be filled out after one of the proposals in the linked design doc has support. + +## Alternatives + +To be filled out after one of the proposals in the linked design doc has support. + +## Action Plan + +The tasks to do in order to migrate to the new idea. + +* [ ] Task one +* [ ] Task two ... From 6ce4170f79549d07cb1eb6ee987303190b2f507e Mon Sep 17 00:00:00 2001 From: Chris Marchbanks Date: Thu, 21 Mar 2024 14:18:11 -0600 Subject: [PATCH 2/9] Address review feedback Signed-off-by: Chris Marchbanks --- proposals/2024-01-29_native_histograms_text_format.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/2024-01-29_native_histograms_text_format.md b/proposals/2024-01-29_native_histograms_text_format.md index ca58d18..5903431 100644 --- a/proposals/2024-01-29_native_histograms_text_format.md +++ b/proposals/2024-01-29_native_histograms_text_format.md @@ -15,7 +15,7 @@ ## Why -Today it is only possible to export native histograms using the protobuf scrape format. Many users prefer the text format, and some client libraries, such as the Python client, want to avoid adding a dependency on protobuf. +Today it is only possible to export native histograms using the Protocol Buffers (protobuf) scrape format. Many users prefer the text format, and some client libraries, such as the Python client, want to avoid adding a dependency on protobuf. During a [dev summit in 2022](https://docs.google.com/document/d/11LC3wJcVk00l8w5P3oLQ-m3Y37iom6INAMEu2ZAGIIE/edit#bookmark=id.c3e7ur6rn5d2) there was consensus we would continue to support the text format. Including native histograms as part of the text format shows commitment to that consensus. @@ -23,7 +23,7 @@ See the linked design doc in Google Docs for additional background information. ### Pitfalls of the current solution -Prometheus client libraries such as Python do not want to require a dependency on protobuf in order to expose native histograms, and in some languages protobuf is painful to use. Gating native histograms only to clients/users willing to use protobuf hurts adoption of native histograms, therefore, we would like a way to represent a histogram in the text based format. +Prometheus client libraries such as Python do not want to require a dependency on protobuf in order to expose native histograms, and in some languages protobuf is painful to use. Gating native histograms only to clients/users willing to use protobuf hurts adoption of native histograms, therefore, we would like a way to represent a native histogram in the text based format. ## Goals From b1283f12ce8d3fdbc2331f48517bc52e17cf5ee7 Mon Sep 17 00:00:00 2001 From: Chris Marchbanks Date: Thu, 21 Mar 2024 20:46:00 -0600 Subject: [PATCH 3/9] Update proposal with the results of the design doc Signed-off-by: Chris Marchbanks --- ...024-01-29_native_histograms_text_format.md | 40 +++++++++++++++---- 1 file changed, 33 insertions(+), 7 deletions(-) diff --git a/proposals/2024-01-29_native_histograms_text_format.md b/proposals/2024-01-29_native_histograms_text_format.md index 5903431..6f3a689 100644 --- a/proposals/2024-01-29_native_histograms_text_format.md +++ b/proposals/2024-01-29_native_histograms_text_format.md @@ -6,7 +6,8 @@ * **Implementation Status:** Not implemented * **Related Issues and PRs:** - * `` + * [Native Histogram Support in client_python](https://github.com/prometheus/client_python/issues/918) + * [OpenMetrics Protobuf format PR](https://github.com/OpenObservability/OpenMetrics/pull/256) * **Other docs or links:** * [Design Doc for choosing a proposal](https://docs.google.com/document/d/1qoHf24cKMpa1QHskIjgzyf3oFhIPvacyJj8Tbe6fIrY/edit#heading=h.5sybau7waq2q) @@ -19,8 +20,6 @@ Today it is only possible to export native histograms using the Protocol Buffers During a [dev summit in 2022](https://docs.google.com/document/d/11LC3wJcVk00l8w5P3oLQ-m3Y37iom6INAMEu2ZAGIIE/edit#bookmark=id.c3e7ur6rn5d2) there was consensus we would continue to support the text format. Including native histograms as part of the text format shows commitment to that consensus. -See the linked design doc in Google Docs for additional background information. - ### Pitfalls of the current solution Prometheus client libraries such as Python do not want to require a dependency on protobuf in order to expose native histograms, and in some languages protobuf is painful to use. Gating native histograms only to clients/users willing to use protobuf hurts adoption of native histograms, therefore, we would like a way to represent a native histogram in the text based format. @@ -46,15 +45,42 @@ Client library maintainers, OpenMetrics, and Prometheus scrape maintainers. ## How -To be filled out after one of the proposals in the linked design doc has support. +Extend the OpenMetrics text format to allow structured values instead of only float values. This structured value will be used to encode a structure with the same fields as is exposed using the [protobuf exposition format](https://github.com/OpenObservability/OpenMetrics/pull/256). Starting with an example and then breaking up the format: +``` +# TYPE nativehistogram histogram +nativehistogram {count:24,sum:100,schema:0,zero_threshold:0.001,zero_count:4,positive_span:[0:2,1:2],negative_span:[0:2,1:2],positive_delta:[2,1,-2,3],negative_delta:[2,1,-2,3]} +``` +The metric will have no "magic" suffixes, then the value for each series is a custom struct format with the following fields: +* `sum: float64` - The sum of all observations for this histogram. Could be negative in cases with negative observations. +* `count: uint64` - The number of samples that were observed for this histogram. +* `schema: int32` - The schema used for this histogram, currently supported values are -4 -> 8. +* `zero_threshold: float64` - The width of the zero bucket. +* `zero_count: uint64` - The number of observations inside the zero bucket. +* `negative_span: []BucketSpan` - The buckets corresponding to negative observations, optional. +* `negative_delta: []int64` - The delta of counts compared to the previous bucket. +* `positive_span: []BucketSpan` - The buckets corresponding to negative observations, optional. +* `positive_delta: []int64` - The delta of counts compared to the previous bucket. + +A bucket span is the combination of an `int32` offset and a `uint32` length. It is encoded as `:`. Lists/arrays are encoded within square brackets with elements separated by commas. Compared to JSON this avoids consistently repeating keys and curly braces. + +Positive infinity, negative infinity, and non number values will be represented as case insensitive versions of `+Inf`, `-Inf`, and `NaN` respectively in any field. This is the same behavior for values in OpenMetrics today. + +Note that in this initial implementation float histograms are not supported. ## Alternatives -To be filled out after one of the proposals in the linked design doc has support. +### Do nothing + +One valid option is to avoid this extra format and require anyone who desires to use native histograms to use protobuf for exposition. It would go against the consensus of the Prometheus team members from 2022 however. + +### Alternate exposition formats + +See the alternate exposition formats proposed in the [design document](https://docs.google.com/document/d/1qoHf24cKMpa1QHskIjgzyf3oFhIPvacyJj8Tbe6fIrY/edit#heading=h.5sybau7waq2q). The other formats generally added in additional readability/verbosity at the expense of performance. ## Action Plan The tasks to do in order to migrate to the new idea. -* [ ] Task one -* [ ] Task two ... +* [ ] Implement an encoder and parser in client_python +* [ ] Implement an experimental parser in the Prometheus server +* [ ] Update OpenMetrics with formalized syntax From f80fafac14b6dfd29b4dc470c94e520ac131e935 Mon Sep 17 00:00:00 2001 From: Chris Marchbanks Date: Thu, 23 May 2024 19:13:31 -0600 Subject: [PATCH 4/9] Add section explaining backwards compatability and versioning plan Signed-off-by: Chris Marchbanks --- proposals/2024-01-29_native_histograms_text_format.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/proposals/2024-01-29_native_histograms_text_format.md b/proposals/2024-01-29_native_histograms_text_format.md index 6f3a689..2231a2d 100644 --- a/proposals/2024-01-29_native_histograms_text_format.md +++ b/proposals/2024-01-29_native_histograms_text_format.md @@ -41,7 +41,7 @@ Client library maintainers, OpenMetrics, and Prometheus scrape maintainers. ## Non-Goals -* Requiring backwards compatability (OpenMetrics 2.0 would be ok) +* Requiring backwards compatability (OpenMetrics 2.0 would be ok), and especially forwards compatability (not required in the OpenMetrics spec). ## How @@ -67,6 +67,10 @@ Positive infinity, negative infinity, and non number values will be represented Note that in this initial implementation float histograms are not supported. +### Backwards compatability and semantic versioning + +After discussions with a few people it is believed that these changes can be made in a 1.x release of OpenMetrics. OpenMetrics 1.x parsers that support native histograms will still be able to read OpenMetrics 1.0 responses, therefore this change is backwards compatible. However, this change is not forwards compatible, i.e. an OpenMetrics 1.0 parser will not be able to read an OpenMetrics >= 1.1 response. Any producers implementing native histograms MUST also implement content negotiation and fall back to OpenMetrics 1.0.0, and therefore not expose native histograms, if a supported version cannot be negotiated. Note that the behavior to fall back to 1.0.0 is already part of the [OpenMetrics spec](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#protocol-negotiation). + ## Alternatives ### Do nothing From 2920fb261dbfda41be77a16e92498d0dfd75e75c Mon Sep 17 00:00:00 2001 From: Chris Marchbanks Date: Thu, 30 May 2024 13:36:12 -0600 Subject: [PATCH 5/9] Address review feedback The significant changes are: 1. Add additional examples. 2. Add a proposal for how content negotiation will be done until OM 1.1 is released. 3. Plural form of arrays in the schema. 4. Added support for multiple exemplars as a non-goal. Signed-off-by: Chris Marchbanks --- ...024-01-29_native_histograms_text_format.md | 40 ++++++++++++++----- 1 file changed, 30 insertions(+), 10 deletions(-) diff --git a/proposals/2024-01-29_native_histograms_text_format.md b/proposals/2024-01-29_native_histograms_text_format.md index 2231a2d..3ec4039 100644 --- a/proposals/2024-01-29_native_histograms_text_format.md +++ b/proposals/2024-01-29_native_histograms_text_format.md @@ -16,9 +16,9 @@ ## Why -Today it is only possible to export native histograms using the Protocol Buffers (protobuf) scrape format. Many users prefer the text format, and some client libraries, such as the Python client, want to avoid adding a dependency on protobuf. +Today it is only possible to export native histograms using the classic (not OpenMetrics) Protocol Buffers (protobuf) scrape format. Many users prefer the text format, and some client libraries, such as the Python client, want to avoid adding a dependency on protobuf. Prometheus content negotiation prefers OpenMetrics to the classic Prometheus text based format, therefore to support native histograms in Prometheus the OpenMetrics text format will also need to support native histograms. -During a [dev summit in 2022](https://docs.google.com/document/d/11LC3wJcVk00l8w5P3oLQ-m3Y37iom6INAMEu2ZAGIIE/edit#bookmark=id.c3e7ur6rn5d2) there was consensus we would continue to support the text format. Including native histograms as part of the text format shows commitment to that consensus. +There is already an open pull request (see Related Issues and PRs above) to add support for native histograms to OpenMetrics, and during a [dev summit in 2022](https://docs.google.com/document/d/11LC3wJcVk00l8w5P3oLQ-m3Y37iom6INAMEu2ZAGIIE/edit#bookmark=id.c3e7ur6rn5d2) there was consensus we would continue to support the text format for new features as well. Including native histograms as part of the text format shows commitment to that consensus. ### Pitfalls of the current solution @@ -42,13 +42,24 @@ Client library maintainers, OpenMetrics, and Prometheus scrape maintainers. ## Non-Goals * Requiring backwards compatability (OpenMetrics 2.0 would be ok), and especially forwards compatability (not required in the OpenMetrics spec). +* Support for multiple exemplars (this will be done in a future proposal). ## How -Extend the OpenMetrics text format to allow structured values instead of only float values. This structured value will be used to encode a structure with the same fields as is exposed using the [protobuf exposition format](https://github.com/OpenObservability/OpenMetrics/pull/256). Starting with an example and then breaking up the format: +Extend the OpenMetrics text format to allow structured values instead of only float values. This structured value will be used to encode a structure with the same fields as is exposed using the [protobuf exposition format](https://github.com/prometheus/client_model/blob/master/io/prometheus/client/metrics.proto). Starting with examples and then breaking up the format: ``` # TYPE nativehistogram histogram -nativehistogram {count:24,sum:100,schema:0,zero_threshold:0.001,zero_count:4,positive_span:[0:2,1:2],negative_span:[0:2,1:2],positive_delta:[2,1,-2,3],negative_delta:[2,1,-2,3]} +nativehistogram {count:24,sum:100,schema:0,zero_threshold:0.001,zero_count:4,positive_spans:[0:2,1:2],negative_spans:[0:2,1:2],positive_deltas:[2,1,-2,3],negative_deltas:[2,1,-2,3]} + +# TYPE hist_with_labels histogram +hist_with_labels{foo="bar",baz="qux"} {count:24,sum:100,schema:0,zero_threshold:0.001,zero_count:4,positive_spans:[0:2,1:2],negative_spans:[0:2,1:2],positive_deltas:[2,1,-2,3],negative_deltas:[2,1,-2,3]} + +# TYPE hist_with_classic_buckets histogram +hist_with_classic_buckets {count:24,sum:100,schema:0,zero_threshold:0.001,zero_count:4,positive_spans:[0:2,1:2],negative_spans:[0:2,1:2],positive_deltas:[2,1,-2,3],negative_deltas:[2,1,-2,3]} +hist_with_classic_buckets_bucket{le="0.001"} 4 +hist_with_classic_buckets_bucket{le="+Inf"} 24 +hist_with_classic_buckets_count 24 +hist_with_classic_buckets_sum 100 ``` The metric will have no "magic" suffixes, then the value for each series is a custom struct format with the following fields: * `sum: float64` - The sum of all observations for this histogram. Could be negative in cases with negative observations. @@ -56,21 +67,30 @@ The metric will have no "magic" suffixes, then the value for each series is a cu * `schema: int32` - The schema used for this histogram, currently supported values are -4 -> 8. * `zero_threshold: float64` - The width of the zero bucket. * `zero_count: uint64` - The number of observations inside the zero bucket. -* `negative_span: []BucketSpan` - The buckets corresponding to negative observations, optional. -* `negative_delta: []int64` - The delta of counts compared to the previous bucket. -* `positive_span: []BucketSpan` - The buckets corresponding to negative observations, optional. -* `positive_delta: []int64` - The delta of counts compared to the previous bucket. +* `negative_spans: []BucketSpan` - The buckets corresponding to negative observations, optional. +* `negative_deltas: []int64` - The delta of counts compared to the previous bucket. +* `positive_spans: []BucketSpan` - The buckets corresponding to negative observations, optional. +* `positive_deltas: []int64` - The delta of counts compared to the previous bucket. A bucket span is the combination of an `int32` offset and a `uint32` length. It is encoded as `:`. Lists/arrays are encoded within square brackets with elements separated by commas. Compared to JSON this avoids consistently repeating keys and curly braces. Positive infinity, negative infinity, and non number values will be represented as case insensitive versions of `+Inf`, `-Inf`, and `NaN` respectively in any field. This is the same behavior for values in OpenMetrics today. -Note that in this initial implementation float histograms are not supported. +Note that in this initial implementation float histograms are not supported. Float histograms are rarely used in exposition, and OpenMetrics does not support classic float histograms either. Support could be added in the future by adding fields for `count_float`, `zero_count_float`, `negative_counts`, and `positive_counts`. -### Backwards compatability and semantic versioning +### Backwards compatibility and semantic versioning After discussions with a few people it is believed that these changes can be made in a 1.x release of OpenMetrics. OpenMetrics 1.x parsers that support native histograms will still be able to read OpenMetrics 1.0 responses, therefore this change is backwards compatible. However, this change is not forwards compatible, i.e. an OpenMetrics 1.0 parser will not be able to read an OpenMetrics >= 1.1 response. Any producers implementing native histograms MUST also implement content negotiation and fall back to OpenMetrics 1.0.0, and therefore not expose native histograms, if a supported version cannot be negotiated. Note that the behavior to fall back to 1.0.0 is already part of the [OpenMetrics spec](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#protocol-negotiation). +Until a version of OpenMetrics is released that contains a stable version of native histograms consumers can determine if native histograms may be present by asking for a `nativehistogram` pre-release identifier. For example, +``` +Accept: application/openmetrics-text;version=1.1.0-nativehistogram.*,application/openmetrics-text;version=1.0.0,text/plain;version=0.0.4 +``` +would mean the consumer can accept any nativehistogram enabled pre-release version of OpenMetrics 1.1.0, the base 1.0.0 version of OpenMetrics, or the 0.0.4 version of the classic Prometheus text format. Producers must include the proper content type for their version, such as the first nativehistogram pre-release: +``` +Content-Type: application/openmetrics-text;version=1.1.0-nativehistogram.0 +``` + ## Alternatives ### Do nothing From cec73ef38c6b96b35f0a764648cff52c2f8a051c Mon Sep 17 00:00:00 2001 From: Chris Marchbanks Date: Tue, 4 Jun 2024 14:41:36 -0600 Subject: [PATCH 6/9] Address review comments * Tightened up some language of how to tell classic and native histograms apart. * Added that whitespace is not allowed inside of a value. Signed-off-by: Chris Marchbanks --- .../2024-01-29_native_histograms_text_format.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/proposals/2024-01-29_native_histograms_text_format.md b/proposals/2024-01-29_native_histograms_text_format.md index 3ec4039..9bd0395 100644 --- a/proposals/2024-01-29_native_histograms_text_format.md +++ b/proposals/2024-01-29_native_histograms_text_format.md @@ -31,7 +31,7 @@ Goals and use cases for the solution as proposed in [How](#how): * Support native histograms in the text format * (Secondary) Encode/decode efficiency * (Secondary) Ease of implementation for client libraries -* (Secondary) Human readibility of the format +* (Secondary) Human readability of the format Note that the goals of efficiency and human readability are commonly at odds with each other. @@ -46,7 +46,7 @@ Client library maintainers, OpenMetrics, and Prometheus scrape maintainers. ## How -Extend the OpenMetrics text format to allow structured values instead of only float values. This structured value will be used to encode a structure with the same fields as is exposed using the [protobuf exposition format](https://github.com/prometheus/client_model/blob/master/io/prometheus/client/metrics.proto). Starting with examples and then breaking up the format: +Extend the OpenMetrics text format to allow structured values instead of only float values for specific series of a histogram type. This structured value will be used to encode a structure with the same fields as is exposed using the [protobuf exposition format](https://github.com/prometheus/client_model/blob/master/io/prometheus/client/metrics.proto). Starting with examples and then breaking up the format: ``` # TYPE nativehistogram histogram nativehistogram {count:24,sum:100,schema:0,zero_threshold:0.001,zero_count:4,positive_spans:[0:2,1:2],negative_spans:[0:2,1:2],positive_deltas:[2,1,-2,3],negative_deltas:[2,1,-2,3]} @@ -61,18 +61,21 @@ hist_with_classic_buckets_bucket{le="+Inf"} 24 hist_with_classic_buckets_count 24 hist_with_classic_buckets_sum 100 ``` -The metric will have no "magic" suffixes, then the value for each series is a custom struct format with the following fields: + +Native histograms will share the "histogram" type with classic histograms. Classic and native histograms can be differentiated by looking at the "magic" suffixes for classic histogram series (`_bucket`, `_count`, `_sum`), and no suffix for native histogram series. This allows producers to expose native histograms and classic histograms together if desired, such as desiring custom bucket boundaries. An optional `_created` series can be created if desired just like a classic histogram as well. + +The value for each series of a native histogram is a custom struct format with the following fields inside curly braces: * `sum: float64` - The sum of all observations for this histogram. Could be negative in cases with negative observations. * `count: uint64` - The number of samples that were observed for this histogram. * `schema: int32` - The schema used for this histogram, currently supported values are -4 -> 8. * `zero_threshold: float64` - The width of the zero bucket. * `zero_count: uint64` - The number of observations inside the zero bucket. * `negative_spans: []BucketSpan` - The buckets corresponding to negative observations, optional. -* `negative_deltas: []int64` - The delta of counts compared to the previous bucket. +* `negative_deltas: []int64` - The delta of counts compared to the previous bucket, optional. * `positive_spans: []BucketSpan` - The buckets corresponding to negative observations, optional. -* `positive_deltas: []int64` - The delta of counts compared to the previous bucket. +* `positive_deltas: []int64` - The delta of counts compared to the previous bucket, optional. -A bucket span is the combination of an `int32` offset and a `uint32` length. It is encoded as `:`. Lists/arrays are encoded within square brackets with elements separated by commas. Compared to JSON this avoids consistently repeating keys and curly braces. +A bucket span is the combination of an `int32` offset and a `uint32` length. It is encoded as `:`. Lists/arrays are encoded within square brackets with elements separated by commas. Compared to JSON this avoids consistently repeating keys and curly braces. White space is not allowed inside of the structure to make a value as easy as possible to parse. Positive infinity, negative infinity, and non number values will be represented as case insensitive versions of `+Inf`, `-Inf`, and `NaN` respectively in any field. This is the same behavior for values in OpenMetrics today. From 6ffbe743bcfdce6ff25f08c3f11cc751fc5c84d6 Mon Sep 17 00:00:00 2001 From: Chris Marchbanks Date: Tue, 4 Jun 2024 15:18:37 -0600 Subject: [PATCH 7/9] Add example for combined histogram with labels Illustrate that ordering matters for this new format to aid parsing. Signed-off-by: Chris Marchbanks --- ...024-01-29_native_histograms_text_format.md | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/proposals/2024-01-29_native_histograms_text_format.md b/proposals/2024-01-29_native_histograms_text_format.md index 9bd0395..ba85209 100644 --- a/proposals/2024-01-29_native_histograms_text_format.md +++ b/proposals/2024-01-29_native_histograms_text_format.md @@ -49,12 +49,15 @@ Client library maintainers, OpenMetrics, and Prometheus scrape maintainers. Extend the OpenMetrics text format to allow structured values instead of only float values for specific series of a histogram type. This structured value will be used to encode a structure with the same fields as is exposed using the [protobuf exposition format](https://github.com/prometheus/client_model/blob/master/io/prometheus/client/metrics.proto). Starting with examples and then breaking up the format: ``` # TYPE nativehistogram histogram +# HELP nativehistogram Is a basic example of a native histogram. nativehistogram {count:24,sum:100,schema:0,zero_threshold:0.001,zero_count:4,positive_spans:[0:2,1:2],negative_spans:[0:2,1:2],positive_deltas:[2,1,-2,3],negative_deltas:[2,1,-2,3]} # TYPE hist_with_labels histogram +# HELP hist_with_labels Is an example of a native histogram with labels. hist_with_labels{foo="bar",baz="qux"} {count:24,sum:100,schema:0,zero_threshold:0.001,zero_count:4,positive_spans:[0:2,1:2],negative_spans:[0:2,1:2],positive_deltas:[2,1,-2,3],negative_deltas:[2,1,-2,3]} # TYPE hist_with_classic_buckets histogram +# HELP hist_with_classic_buckets Is an example of native and classic histograms together. hist_with_classic_buckets {count:24,sum:100,schema:0,zero_threshold:0.001,zero_count:4,positive_spans:[0:2,1:2],negative_spans:[0:2,1:2],positive_deltas:[2,1,-2,3],negative_deltas:[2,1,-2,3]} hist_with_classic_buckets_bucket{le="0.001"} 4 hist_with_classic_buckets_bucket{le="+Inf"} 24 @@ -81,6 +84,24 @@ Positive infinity, negative infinity, and non number values will be represented Note that in this initial implementation float histograms are not supported. Float histograms are rarely used in exposition, and OpenMetrics does not support classic float histograms either. Support could be added in the future by adding fields for `count_float`, `zero_count_float`, `negative_counts`, and `positive_counts`. +If native histograms and a classic histogram are exposed simultaneously the native histogram must come first for any labelset. For example: +``` +# TYPE hist_with_classic_buckets histogram +# HELP hist_with_classic_buckets Is an example of combining native and classic histograms for two labelsets. +hist_with_classic_buckets{foo="bar"} {count:24,sum:100,schema:0,zero_threshold:0.001,zero_count:4,positive_spans:[0:2,1:2],negative_spans:[0:2,1:2],positive_deltas:[2,1,-2,3],negative_deltas:[2,1,-2,3]} +hist_with_classic_buckets_bucket{foo="bar",le="0.001"} 4 +hist_with_classic_buckets_bucket{foo="bar",le="+Inf"} 24 +hist_with_classic_buckets_count{foo="bar"} 24 +hist_with_classic_buckets_sum{foo="bar"} 100 +hist_with_classic_buckets_created{foo="bar"} 1717536092 +hist_with_classic_buckets{foo="baz"} {count:24,sum:100,schema:0,zero_threshold:0.001,zero_count:4,positive_spans:[0:2,1:2],negative_spans:[0:2,1:2],positive_deltas:[2,1,-2,3],negative_deltas:[2,1,-2,3]} +hist_with_classic_buckets_bucket{foo="baz",le="0.001"} 4 +hist_with_classic_buckets_bucket{foo="baz",le="+Inf"} 24 +hist_with_classic_buckets_count{foo="baz"} 24 +hist_with_classic_buckets_sum{foo="baz"} 100 +hist_with_classic_buckets_created{foo="baz"} 1717536098 +``` + ### Backwards compatibility and semantic versioning After discussions with a few people it is believed that these changes can be made in a 1.x release of OpenMetrics. OpenMetrics 1.x parsers that support native histograms will still be able to read OpenMetrics 1.0 responses, therefore this change is backwards compatible. However, this change is not forwards compatible, i.e. an OpenMetrics 1.0 parser will not be able to read an OpenMetrics >= 1.1 response. Any producers implementing native histograms MUST also implement content negotiation and fall back to OpenMetrics 1.0.0, and therefore not expose native histograms, if a supported version cannot be negotiated. Note that the behavior to fall back to 1.0.0 is already part of the [OpenMetrics spec](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#protocol-negotiation). From 7af982722f6e06f49cf4e19a44361a983388db91 Mon Sep 17 00:00:00 2001 From: Chris Marchbanks Date: Thu, 6 Jun 2024 15:04:55 -0600 Subject: [PATCH 8/9] Include support for multiple exemplars Provide a paragraph describing support for multiple exemplars as part of this proposal. Currently it is left ambiguous if a list of exemplars can be included on more than native histograms, we can define that explicitly when writing the new OpenMetrics spec. Signed-off-by: Chris Marchbanks --- proposals/2024-01-29_native_histograms_text_format.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/proposals/2024-01-29_native_histograms_text_format.md b/proposals/2024-01-29_native_histograms_text_format.md index ba85209..4807283 100644 --- a/proposals/2024-01-29_native_histograms_text_format.md +++ b/proposals/2024-01-29_native_histograms_text_format.md @@ -42,7 +42,6 @@ Client library maintainers, OpenMetrics, and Prometheus scrape maintainers. ## Non-Goals * Requiring backwards compatability (OpenMetrics 2.0 would be ok), and especially forwards compatability (not required in the OpenMetrics spec). -* Support for multiple exemplars (this will be done in a future proposal). ## How @@ -102,6 +101,13 @@ hist_with_classic_buckets_sum{foo="baz"} 100 hist_with_classic_buckets_created{foo="baz"} 1717536098 ``` +Finally, multiple exemplars will also be supported in the exposition format by providing a list of exemplars at the end of any line, separated by `#`. Note that having spaces around the hashes is required and matches the [ABNF specification in OpenMetrics](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#abnf). For example: +``` +# TYPE exemplar_example histogram +# HELP exemplar_example Is an example of a native histogram with exemplars. +nativehistogram {count:24,sum:100,schema:0,zero_threshold:0.001,zero_count:4,positive_spans:[0:2,1:2],negative_spans:[0:2,1:2],positive_deltas:[2,1,-2,3],negative_deltas:[2,1,-2,3]} # {trace_id="KOO5S4vxi0o"} 0.67 # {trace_id="oHg5SJYRHA0"} 9.8 1520879607.789 +``` + ### Backwards compatibility and semantic versioning After discussions with a few people it is believed that these changes can be made in a 1.x release of OpenMetrics. OpenMetrics 1.x parsers that support native histograms will still be able to read OpenMetrics 1.0 responses, therefore this change is backwards compatible. However, this change is not forwards compatible, i.e. an OpenMetrics 1.0 parser will not be able to read an OpenMetrics >= 1.1 response. Any producers implementing native histograms MUST also implement content negotiation and fall back to OpenMetrics 1.0.0, and therefore not expose native histograms, if a supported version cannot be negotiated. Note that the behavior to fall back to 1.0.0 is already part of the [OpenMetrics spec](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#protocol-negotiation). From 3717133376515f5f5c0f5c8352826e3b58fadf56 Mon Sep 17 00:00:00 2001 From: Chris Marchbanks Date: Thu, 20 Jun 2024 10:03:40 -0600 Subject: [PATCH 9/9] Add additional non-goals Explicitly state that changing created timestamp semantics and allowing custom buckets as part of the native histogram structure are not goals for this proposal. Signed-off-by: Chris Marchbanks --- proposals/2024-01-29_native_histograms_text_format.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/2024-01-29_native_histograms_text_format.md b/proposals/2024-01-29_native_histograms_text_format.md index 4807283..fe0ce4d 100644 --- a/proposals/2024-01-29_native_histograms_text_format.md +++ b/proposals/2024-01-29_native_histograms_text_format.md @@ -42,6 +42,8 @@ Client library maintainers, OpenMetrics, and Prometheus scrape maintainers. ## Non-Goals * Requiring backwards compatability (OpenMetrics 2.0 would be ok), and especially forwards compatability (not required in the OpenMetrics spec). +* Support for custom buckets as part of the native histogram structure. If needed by a consumer, custom buckets will need to be parsed from classic buckets. +* Changing created timestamp semantics. Created timestamps will continue to be exposed as an additional series as with OpenMetrics 1.0. ## How