Skip to content

Commit

Permalink
[sdk-metrics] Add experimental envvar for setting ExemplarFilter for …
Browse files Browse the repository at this point in the history 
…histograms (#5611)

Co-authored-by: Cijo Thomas <[email protected]>
  • Loading branch information
@CodeBlanch @cijothomas
CodeBlanch and cijothomas authored May 16, 2024
1 parent 808abc8 commit 1e065cb
Show file tree
Hide file tree
Showing 8 changed files with 180 additions and 54 deletions.
45 changes: 39 additions & 6 deletions docs/metrics/customizing-the-sdk/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -352,26 +352,40 @@ tutorial](../exemplars/README.md) demonstrates how to use exemplars to achieve
correlation from metrics to traces, which is one of the primary use cases for
exemplars.

#### Default behavior

Exemplars in OpenTelemetry .NET are **off by default**
(`ExemplarFilterType.AlwaysOff`). The [OpenTelemetry
Specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#exemplarfilter)
recommends Exemplars collection should be on by default
(`ExemplarFilterType.TraceBased`) however there is a performance cost associated
with Exemplars so OpenTelemetry .NET has taken a more conservative stance for
its default behavior.

#### ExemplarFilter

`ExemplarFilter` determines which measurements are offered to the configured
`ExemplarReservoir`, which makes the final decision about whether or not the
offered measurement gets recorded as an `Exemplar`. Generally `ExemplarFilter`
is a mechanism to control the overhead associated with `Exemplar` offering.
is a mechanism to control the overhead associated with the offering and
recording of `Exemplar`s.

OpenTelemetry SDK comes with the following `ExemplarFilters` (defined on
OpenTelemetry SDK comes with the following `ExemplarFilter`s (defined on
`ExemplarFilterType`):

* `AlwaysOff`: Makes no measurements eligible for becoming an `Exemplar`. Using
this is as good as turning off the `Exemplar` feature and is the current
default.
* (Default behavior) `AlwaysOff`: Makes no measurements eligible for becoming an
`Exemplar`. Using this disables `Exemplar` collection and avoids all
performance costs associated with `Exemplar`s.
* `AlwaysOn`: Makes all measurements eligible for becoming an `Exemplar`.
* `TraceBased`: Makes those measurements eligible for becoming an `Exemplar`
which are recorded in the context of a sampled `Activity` (span).

The `SetExemplarFilter` extension method on `MeterProviderBuilder` can be used
to set the desired `ExemplarFilterType` and enable `Exemplar` collection:

> [!NOTE]
> The `SetExemplarFilter` API was added in the `1.9.0` release.
```csharp
using OpenTelemetry;
using OpenTelemetry.Metrics;
Expand All @@ -382,6 +396,24 @@ using var meterProvider = Sdk.CreateMeterProviderBuilder()
.Build();
```

It is also possible to configure the `ExemplarFilter` by using following
environmental variables:

> [!NOTE]
> Programmatically calling `SetExemplarFilter` will override any defaults set
using environment variables or configuration.

| Environment variable | Description | Notes |
| -------------------------- | -------------------------------------------------- |-------|
| `OTEL_METRICS_EXEMPLAR_FILTER` | Sets the default `ExemplarFilter` to use for all metrics. | Added in `1.9.0` |
| `OTEL_DOTNET_EXPERIMENTAL_METRICS_EXEMPLAR_FILTER_HISTOGRAMS` | Sets the default `ExemplarFilter` to use for histogram metrics. If set `OTEL_DOTNET_EXPERIMENTAL_METRICS_EXEMPLAR_FILTER_HISTOGRAMS` takes precedence over `OTEL_METRICS_EXEMPLAR_FILTER` for histogram metrics. | Experimental key (may be removed or changed in the future). Added in `1.9.0` |

Allowed values:

* `always_off`: Equivalent to `ExemplarFilterType.AlwaysOff`
* `always_on`: Equivalent to `ExemplarFilterType.AlwaysOn`
* `trace_based`: Equivalent to `ExemplarFilterType.TraceBased`

#### ExemplarReservoir

`ExemplarReservoir` receives the measurements sampled by the `ExemplarFilter`
Expand All @@ -398,7 +430,8 @@ metrics except Histograms with buckets. It has a fixed reservoir pool, and
implements the equivalent of [naive
reservoir](https://en.wikipedia.org/wiki/Reservoir_sampling). The reservoir pool
size (currently defaulting to 1) determines the maximum number of exemplars
stored.
stored. Exponential histograms use a `SimpleFixedSizeExemplarReservoir` with a
pool size equal to the number of buckets up to a max of `20`.

> [!NOTE]
> Currently there is no ability to change or configure `ExemplarReservoir`.
Expand Down
2 changes: 0 additions & 2 deletions examples/AspNetCore/Program.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -84,9 +84,7 @@
// Ensure the MeterProvider subscribes to any custom Meters.
builder
.AddMeter(Instrumentation.MeterName)
#if EXPOSE_EXPERIMENTAL_FEATURES
.SetExemplarFilter(ExemplarFilterType.TraceBased)
#endif
.AddRuntimeInstrumentation()
.AddHttpClientInstrumentation()
.AddAspNetCoreInstrumentation();
Expand Down
7 changes: 7 additions & 0 deletions src/OpenTelemetry/CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,13 @@
which has always been supported.
([#5614](https://github.com/open-telemetry/opentelemetry-dotnet/pull/5614))

* The `ExemplarFilter` used by SDK `MeterProvider`s for histogram metrics can
now be controlled via the experimental
`OTEL_DOTNET_EXPERIMENTAL_METRICS_EXEMPLAR_FILTER_HISTOGRAMS` environment
variable. The supported values are: `always_off`, `always_on`, and
`trace_based`.
([#5611](https://github.com/open-telemetry/opentelemetry-dotnet/pull/5611))

## 1.8.1

Released 2024-Apr-17
Expand Down
65 changes: 52 additions & 13 deletions src/OpenTelemetry/Metrics/MeterProviderSdk.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ internal sealed class MeterProviderSdk : MeterProvider
internal const string EmitOverFlowAttributeConfigKey = "OTEL_DOTNET_EXPERIMENTAL_METRICS_EMIT_OVERFLOW_ATTRIBUTE";
internal const string ReclaimUnusedMetricPointsConfigKey = "OTEL_DOTNET_EXPERIMENTAL_METRICS_RECLAIM_UNUSED_METRIC_POINTS";
internal const string ExemplarFilterConfigKey = "OTEL_METRICS_EXEMPLAR_FILTER";
internal const string ExemplarFilterHistogramsConfigKey = "OTEL_DOTNET_EXPERIMENTAL_METRICS_EXEMPLAR_FILTER_HISTOGRAMS";

internal readonly IServiceProvider ServiceProvider;
internal readonly IDisposable? OwnedServiceProvider;
Expand All @@ -24,6 +25,7 @@ internal sealed class MeterProviderSdk : MeterProvider
internal bool EmitOverflowAttribute;
internal bool ReclaimUnusedMetricPoints;
internal ExemplarFilterType? ExemplarFilter;
internal ExemplarFilterType? ExemplarFilterForHistograms;
internal Action? OnCollectObservableInstruments;

private readonly List<object> instrumentations = new();
Expand Down Expand Up @@ -72,6 +74,9 @@ internal MeterProviderSdk(

this.viewConfigs = state.ViewConfigs;

OpenTelemetrySdkEventSource.Log.MeterProviderSdkEvent(
$"MeterProvider configuration: {{MetricLimit={state.MetricLimit}, CardinalityLimit={state.CardinalityLimit}, EmitOverflowAttribute={this.EmitOverflowAttribute}, ReclaimUnusedMetricPoints={this.ReclaimUnusedMetricPoints}, ExemplarFilter={this.ExemplarFilter}, ExemplarFilterForHistograms={this.ExemplarFilterForHistograms}}}.");

foreach (var reader in state.Readers)
{
Guard.ThrowIfNull(reader);
Expand All @@ -83,7 +88,8 @@ internal MeterProviderSdk(
state.CardinalityLimit,
this.EmitOverflowAttribute,
this.ReclaimUnusedMetricPoints,
this.ExemplarFilter);
this.ExemplarFilter,
this.ExemplarFilterForHistograms);

if (this.reader == null)
{
Expand Down Expand Up @@ -490,37 +496,70 @@ private void ApplySpecificationConfigurationKeys(IConfiguration configuration)
OpenTelemetrySdkEventSource.Log.MeterProviderSdkEvent("Reclaim unused metric point feature enabled via configuration.");
}

var hasProgrammaticExemplarFilterValue = this.ExemplarFilter.HasValue;

if (configuration.TryGetStringValue(ExemplarFilterConfigKey, out var configValue))
{
if (this.ExemplarFilter.HasValue)
if (hasProgrammaticExemplarFilterValue)
{
OpenTelemetrySdkEventSource.Log.MeterProviderSdkEvent(
$"Exemplar filter configuration value '{configValue}' has been ignored because a value '{this.ExemplarFilter}' was set programmatically.");
return;
}

ExemplarFilterType? exemplarFilter;
if (!TryParseExemplarFilterFromConfigurationValue(configValue, out var exemplarFilter))
{
OpenTelemetrySdkEventSource.Log.MeterProviderSdkEvent($"Exemplar filter configuration was found but the value '{configValue}' is invalid and will be ignored.");
return;
}

this.ExemplarFilter = exemplarFilter;

OpenTelemetrySdkEventSource.Log.MeterProviderSdkEvent($"Exemplar filter set to '{exemplarFilter}' from configuration.");
}

if (configuration.TryGetStringValue(ExemplarFilterHistogramsConfigKey, out configValue))
{
if (hasProgrammaticExemplarFilterValue)
{
OpenTelemetrySdkEventSource.Log.MeterProviderSdkEvent(
$"Exemplar filter histogram configuration value '{configValue}' has been ignored because a value '{this.ExemplarFilter}' was set programmatically.");
return;
}

if (!TryParseExemplarFilterFromConfigurationValue(configValue, out var exemplarFilter))
{
OpenTelemetrySdkEventSource.Log.MeterProviderSdkEvent($"Exemplar filter histogram configuration was found but the value '{configValue}' is invalid and will be ignored.");
return;
}

this.ExemplarFilterForHistograms = exemplarFilter;

OpenTelemetrySdkEventSource.Log.MeterProviderSdkEvent($"Exemplar filter for histograms set to '{exemplarFilter}' from configuration.");
}

static bool TryParseExemplarFilterFromConfigurationValue(string? configValue, out ExemplarFilterType? exemplarFilter)
{
if (string.Equals("always_off", configValue, StringComparison.OrdinalIgnoreCase))
{
exemplarFilter = ExemplarFilterType.AlwaysOff;
return true;
}
else if (string.Equals("always_on", configValue, StringComparison.OrdinalIgnoreCase))

if (string.Equals("always_on", configValue, StringComparison.OrdinalIgnoreCase))
{
exemplarFilter = ExemplarFilterType.AlwaysOn;
return true;
}
else if (string.Equals("trace_based", configValue, StringComparison.OrdinalIgnoreCase))

if (string.Equals("trace_based", configValue, StringComparison.OrdinalIgnoreCase))
{
exemplarFilter = ExemplarFilterType.TraceBased;
return true;
}
else
{
OpenTelemetrySdkEventSource.Log.MeterProviderSdkEvent($"Exemplar filter configuration was found but the value '{configValue}' is invalid and will be ignored.");
return;
}

this.ExemplarFilter = exemplarFilter;

OpenTelemetrySdkEventSource.Log.MeterProviderSdkEvent($"Exemplar filter set to '{exemplarFilter}' from configuration.");
exemplarFilter = null;
return false;
}
}
}
7 changes: 1 addition & 6 deletions src/OpenTelemetry/Metrics/Metric.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -125,12 +125,7 @@ internal Metric(
aggType = AggregationType.LongGauge;
this.MetricType = MetricType.LongGauge;
}
else if (instrumentIdentity.InstrumentType == typeof(Histogram<long>)
|| instrumentIdentity.InstrumentType == typeof(Histogram<int>)
|| instrumentIdentity.InstrumentType == typeof(Histogram<short>)
|| instrumentIdentity.InstrumentType == typeof(Histogram<byte>)
|| instrumentIdentity.InstrumentType == typeof(Histogram<float>)
|| instrumentIdentity.InstrumentType == typeof(Histogram<double>))
else if (instrumentIdentity.IsHistogram)
{
var explicitBucketBounds = instrumentIdentity.HistogramBucketBounds;
var exponentialMaxSize = instrumentIdentity.ExponentialHistogramMaxSize;
Expand Down
24 changes: 21 additions & 3 deletions src/OpenTelemetry/Metrics/MetricReaderExt.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ public abstract partial class MetricReader
private bool emitOverflowAttribute;
private bool reclaimUnusedMetricPoints;
private ExemplarFilterType? exemplarFilter;
private ExemplarFilterType? exemplarFilterForHistograms;

internal static void DeactivateMetric(Metric metric)
{
Expand Down Expand Up @@ -54,6 +55,11 @@ internal virtual List<Metric> AddMetricWithNoViews(Instrument instrument)
Debug.Assert(this.metrics != null, "this.metrics was null");

var metricStreamIdentity = new MetricStreamIdentity(instrument!, metricStreamConfiguration: null);

var exemplarFilter = metricStreamIdentity.IsHistogram
? this.exemplarFilterForHistograms ?? this.exemplarFilter
: this.exemplarFilter;

lock (this.instrumentCreationLock)
{
if (this.TryGetExistingMetric(in metricStreamIdentity, out var existingMetric))
Expand All @@ -72,7 +78,13 @@ internal virtual List<Metric> AddMetricWithNoViews(Instrument instrument)
Metric? metric = null;
try
{
metric = new Metric(metricStreamIdentity, this.GetAggregationTemporality(metricStreamIdentity.InstrumentType), this.cardinalityLimit, this.emitOverflowAttribute, this.reclaimUnusedMetricPoints, this.exemplarFilter);
metric = new Metric(
metricStreamIdentity,
this.GetAggregationTemporality(metricStreamIdentity.InstrumentType),
this.cardinalityLimit,
this.emitOverflowAttribute,
this.reclaimUnusedMetricPoints,
exemplarFilter);
}
catch (NotSupportedException nse)
{
Expand Down Expand Up @@ -114,6 +126,10 @@ internal virtual List<Metric> AddMetricWithViews(Instrument instrument, List<Met
var metricStreamConfig = metricStreamConfigs[i];
var metricStreamIdentity = new MetricStreamIdentity(instrument!, metricStreamConfig);

var exemplarFilter = metricStreamIdentity.IsHistogram
? this.exemplarFilterForHistograms ?? this.exemplarFilter
: this.exemplarFilter;

if (!MeterProviderBuilderSdk.IsValidInstrumentName(metricStreamIdentity.InstrumentName))
{
OpenTelemetrySdkEventSource.Log.MetricInstrumentIgnored(
Expand Down Expand Up @@ -150,7 +166,7 @@ internal virtual List<Metric> AddMetricWithViews(Instrument instrument, List<Met
metricStreamConfig?.CardinalityLimit ?? this.cardinalityLimit,
this.emitOverflowAttribute,
this.reclaimUnusedMetricPoints,
this.exemplarFilter,
exemplarFilter,
metricStreamConfig?.ExemplarReservoirFactory);

this.instrumentIdentityToMetric[metricStreamIdentity] = metric;
Expand All @@ -170,7 +186,8 @@ internal void ApplyParentProviderSettings(
int cardinalityLimit,
bool emitOverflowAttribute,
bool reclaimUnusedMetricPoints,
ExemplarFilterType? exemplarFilter)
ExemplarFilterType? exemplarFilter,
ExemplarFilterType? exemplarFilterForHistograms)
{
this.metricLimit = metricLimit;
this.metrics = new Metric[metricLimit];
Expand All @@ -179,6 +196,7 @@ internal void ApplyParentProviderSettings(
this.emitOverflowAttribute = emitOverflowAttribute;
this.reclaimUnusedMetricPoints = reclaimUnusedMetricPoints;
this.exemplarFilter = exemplarFilter;
this.exemplarFilterForHistograms = exemplarFilterForHistograms;
}

private bool TryGetExistingMetric(in MetricStreamIdentity metricStreamIdentity, [NotNullWhen(true)] out Metric? existingMetric)
Expand Down
8 changes: 8 additions & 0 deletions src/OpenTelemetry/Metrics/MetricStreamIdentity.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,14 @@ public MetricStreamIdentity(Instrument instrument, MetricStreamConfiguration? me

public bool HistogramRecordMinMax { get; }

public bool IsHistogram =>
this.InstrumentType == typeof(Histogram<long>)
|| this.InstrumentType == typeof(Histogram<int>)
|| this.InstrumentType == typeof(Histogram<short>)
|| this.InstrumentType == typeof(Histogram<byte>)
|| this.InstrumentType == typeof(Histogram<float>)
|| this.InstrumentType == typeof(Histogram<double>);

public static bool operator ==(MetricStreamIdentity metricIdentity1, MetricStreamIdentity metricIdentity2) => metricIdentity1.Equals(metricIdentity2);

public static bool operator !=(MetricStreamIdentity metricIdentity1, MetricStreamIdentity metricIdentity2) => !metricIdentity1.Equals(metricIdentity2);
Expand Down
Loading

0 comments on commit 1e065cb

Please sign in to comment.