Table of Contents
Exemplars are example data points for aggregated data. They provide specific context to otherwise general aggregations. One common use case is to gain ability to correlate metrics to traces (and logs). While OpenTelemetry .NET supports Exemplars, it is only useful if the telemetry backend also supports the capabilities. This tutorial uses well known open-source backends to demonstrate the concept. The following components are involved:
- Program.cs - this application is instrumented with OpenTelemetry, it sends metrics to Prometheus, and traces to Jaeger.
- Prometheus - Prometheus is used as the metrics backend.
- Jaeger - Jaeger is used as the distributed tracing backend.
- Grafana - UI to query metrics from Prometheus, traces from Jaeger, and to navigate between metrics and traces using Exemplars.
Download the latest binary distribution archive of Jaeger.
After finished downloading, extract it to a local location that's easy to
access. Run the jaeger-all-in-one(.exe) executable:
./jaeger-all-in-one --collector.otlp.enabledFollow the first steps to download the latest release of Prometheus.
After finished downloading, extract it to a local location that's easy to
access. Run the prometheus(.exe) server executable with feature flags
exemplars
storage
and
otlp-receiver
enabled:
./prometheus --enable-feature=exemplar-storage --enable-feature=otlp-write-receiverFollow the operating system specific instructions to download and install Grafana.
After installation, start the standalone Grafana server (grafana-server.exe or
./bin/grafana-server, depending on the operating system). Then, use a
supported web
browser
to navigate to http://localhost:3000/.
Follow the instructions in the Grafana getting started doc to log in.
After successfully logging in, hover on the Configuration icon on the panel at the left hand side, and click on Plugins.
Find and click on the Jaeger plugin. Next click on Create a Jaeger data source
button. Make the following changes:
- Set "URL" to
http://localhost:16686/. - At the bottom of the page click
Save & testto ensure the data source is working.
Find and click on the Prometheus plugin. Next click on
Create a Prometheus data source button. Make the following changes:
- Set "URL" to
http://localhost:9090. - Under the "Exemplars" section, enable "Internal link", set "Data source" to
Jaeger, and set "Label name" totrace_id. - At the bottom of the page click
Save & testto ensure the data source is working.
Create a new console application and run it:
dotnet new console --output exemplars
cd exemplars
dotnet runAdd reference to OTLP Exporter:
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocolNow copy the code from Program.cs and run the application again. The application will start sending metrics to Prometheus and traces to Jaeger.
The application is configured with trace-based exemplar filter, which enables the OpenTelemetry SDK to attach exemplars to metrics:
var meterProvider = Sdk.CreateMeterProviderBuilder()
...
.SetExemplarFilter(ExemplarFilterType.TraceBased)
...For more details about the SetExemplarFilter API see: Customizing
OpenTelemetry .NET SDK for Metrics >
ExemplarFilter.
Open Grafana, select Explore, and select Prometheus as the source. Select the
metric named MyHistogram_bucket, and plot the chart. Toggle on the "Exemplars"
option from the UI and hit refresh.
The Exemplars appear as special "diamond shaped dots" along with the metric charts in the UI. Select any exemplar to see the exemplar data, which includes the timestamp when the measurement was recorded, the raw value, and trace context when the recording was done. The "trace_id" enables jumping to the tracing backed (Jaeger in this case). Click on the "Query with Jaeger" button next to the "trace_id" field to open the corresponding trace in Jaeger.
