Skip to content

Commit 7fad5a1

Browse files
noahfalknoahfalk
authored
Update distributed tracing docs (#23313)
* Update distributed tracing docs We are about to announce support for OpenTelemetry .NET and there is likely to be an influx of developers reading these docs. They are already the #1 link for "distributed tracing" on bing.com so they are probably getting above average traffic/scrutiny too. - Reorganized the content into a landing page, conceptual docs, instrumentation walkthroughs and collection walkthroughs - Added some more intro and conceptual information assuming that many developers are completely new to distributed tracing and have no frame of reference. - Removed the example of doing distributed tracing with DiagnosticSource We might want to restore this in the future but hopefully it represents a niche usage scenario that we are migrating away from. Without context indicating when to use it it is likely to confuse new developers about recommended patterns. - Updated the examples to ensure they start with copy-pastable code that is fully runnable. - Added links for collecting distributed tracing instrumentation with Application Insights There are many oportunities for further improvement that we may want to triage: - Add examples and pictures of more realistic collected distributed trace information to help developers internalize why it might be useful to them. - Add a diagnostic guide showing how distributed tracing assists in resolving a realistic production issue. - Ensure the getting started guides for collection transfer smoothly to remote pages in the AppInsights and OpenTelemetry docs - There was a spot I wanted to link to supported OpenTelemetry exporters but there is no anchor that goes solely to that information. - Flesh out the instrumentation guidance to answer more questions developers may encounter such as: - naming conventions for Activity - when to use DisplayName - how to log exceptions - how to propagate IDs to and from network protocols - how to migrate pre-existing Activity instrumentation to use ActivitySource - Add performance analysis and guidance to reassure developers that Activity instrumentation + OpenTelemetry are serious about high performance workloads. - Discuss more conceptual topics and/or go into greater depth about some areas: - Hierarchical vs. W3C ID. - Format of a W3C ID - Describe sampling options in ActivityListener - Explain why some activities are uncapturable with ActivityListener and require workarounds with DiagnosticListener. - Roadmap for handling issues around custom propagators, supporting protocol updates, and suppressing undesired automatic Activity creation
1 parent 576cea4 commit 7fad5a1

File tree

7 files changed

+1021
-225
lines changed

7 files changed

+1021
-225
lines changed

‎docs/core/diagnostics/distributed-tracing-collection-walkthroughs.md‎

Lines changed: 368 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,368 @@
1+
---
2+
title: Collect a distributed trace - .NET
3+
description: Tutorials to collect distributed traces in .NET applications using OpenTelemetry, Application Insights, or ActivityListener
4+
ms.topic: tutorial
5+
ms.date: 03/14/2021
6+
---
7+
8+
# Collect a distributed trace
9+
10+
**This article applies to: ✔️** .NET Core 2.1 and later versions **and** .NET Framework 4.5 and later versions
11+
12+
Instrumented code can create <xref:System.Diagnostics.Activity> objects as part of a distributed trace, but the information
13+
in these objects needs to be collected into centralized storage so that the entire trace can be
14+
reviewed later. In this tutorial you will collect the distributed trace telemetry in different ways so that it is
15+
available to diagnose application issues when needed. See
16+
[the instrumentation tutorial](distributed-tracing-instrumentation-walkthroughs.md) if you need to add new instrumentation.
17+
18+
## Collect traces using OpenTelemetry
19+
20+
### Prerequisites
21+
22+
- [.NET Core 2.1 SDK](https://dotnet.microsoft.com/download/dotnet) or a later version
23+
24+
### Create an example application
25+
26+
Before any distributed trace telemetry can be collected we need to produce it. Often this instrumentation might be
27+
in libraries but for simplicity you will create a small app that has some example instrumentation using
28+
<xref:System.Diagnostics.ActivitySource.StartActivity%2A>. At this point no collection is happening yet,
29+
StartActivity() has no side-effect and returns null. See
30+
[the instrumentation tutorial](distributed-tracing-instrumentation-walkthroughs.md) for more details.
31+
32+
```dotnetcli
33+
dotnet new console
34+
```
35+
36+
Applications that target .NET 5 and later already have the necessary distributed tracing APIs included. For apps targeting older
37+
.NET versions add the [System.Diagnostics.DiagnosticSource NuGet package](https://www.nuget.org/packages/System.Diagnostics.DiagnosticSource/)
38+
version 5 or greater.
39+
40+
```dotnetcli
41+
dotnet add package System.Diagnostics.DiagnosticSource
42+
```
43+
44+
Replace the contents of the generated Program.cs with this example source:
45+
46+
```C#
47+
using System;
48+
using System.Diagnostics;
49+
using System.Threading.Tasks;
50+
51+
namespace Sample.DistributedTracing
52+
{
53+
class Program
54+
{
55+
static ActivitySource s_source = new ActivitySource("Sample.DistributedTracing");
56+
57+
static async Task Main(string[] args)
58+
{
59+
await DoSomeWork();
60+
Console.WriteLine("Example work done");
61+
}
62+
63+
static async Task DoSomeWork()
64+
{
65+
using (Activity a = s_source.StartActivity("SomeWork"))
66+
{
67+
await StepOne();
68+
await StepTwo();
69+
}
70+
}
71+
72+
static async Task StepOne()
73+
{
74+
using (Activity a = s_source.StartActivity("StepOne"))
75+
{
76+
await Task.Delay(500);
77+
}
78+
}
79+
80+
static async Task StepTwo()
81+
{
82+
using (Activity a = s_source.StartActivity("StepTwo"))
83+
{
84+
await Task.Delay(1000);
85+
}
86+
}
87+
}
88+
}
89+
```
90+
91+
Running the app does not collect any trace data yet:
92+
93+
```dotnetcli
94+
> dotnet run
95+
Example work done
96+
```
97+
98+
### Collect using OpenTelemetry
99+
100+
[OpenTelemetry](https://opentelemetry.io/) is a vendor neutral open source project supported by the
101+
[Cloud Native Computing Foundation](https://www.cncf.io/) that aims to standardize generating and collecting telemetry for
102+
cloud-native software. In this example you will collect and display distributed trace information on the console though
103+
OpenTelemetry can be reconfigured to send it elsewhere. See the
104+
[OpenTelemetry getting started guide](https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/trace/getting-started/README.md)
105+
for more information.
106+
107+
Add the [OpenTelemetry.Exporter.Console](https://www.nuget.org/packages/OpenTelemetry.Exporter.Console/) NuGet package.
108+
109+
```dotnetcli
110+
dotnet add package OpenTelemetry.Exporter.Console
111+
```
112+
113+
Update Program.cs with additional OpenTelemetry using statments:
114+
115+
```C#
116+
using OpenTelemetry;
117+
using OpenTelemetry.Resources;
118+
using OpenTelemetry.Trace;
119+
using System;
120+
using System.Diagnostics;
121+
using System.Threading.Tasks;
122+
```
123+
124+
Update Main() to create the OpenTelemetry TracerProvider:
125+
126+
```C#
127+
public static async Task Main()
128+
{
129+
using var tracerProvider = Sdk.CreateTracerProviderBuilder()
130+
.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("MySample"))
131+
.AddSource("Sample.DistributedTracing")
132+
.AddConsoleExporter()
133+
.Build();
134+
135+
await DoSomeWork();
136+
Console.WriteLine("Example work done");
137+
}
138+
```
139+
140+
Now the app collects distributed trace information and displays it to the console:
141+
142+
```dotnetcli
143+
> dotnet run
144+
Activity.Id: 00-7759221f2c5599489d455b84fa0f90f4-6081a9b8041cd840-01
145+
Activity.ParentId: 00-7759221f2c5599489d455b84fa0f90f4-9a52f72c08a9d447-01
146+
Activity.DisplayName: StepOne
147+
Activity.Kind: Internal
148+
Activity.StartTime: 2021-03-18T10:46:46.8649754Z
149+
Activity.Duration: 00:00:00.5069226
150+
Resource associated with Activity:
151+
service.name: MySample
152+
service.instance.id: 909a4624-3b2e-40e4-a86b-4a2c8003219e
153+
154+
Activity.Id: 00-7759221f2c5599489d455b84fa0f90f4-d2b283db91cf774c-01
155+
Activity.ParentId: 00-7759221f2c5599489d455b84fa0f90f4-9a52f72c08a9d447-01
156+
Activity.DisplayName: StepTwo
157+
Activity.Kind: Internal
158+
Activity.StartTime: 2021-03-18T10:46:47.3838737Z
159+
Activity.Duration: 00:00:01.0142278
160+
Resource associated with Activity:
161+
service.name: MySample
162+
service.instance.id: 909a4624-3b2e-40e4-a86b-4a2c8003219e
163+
164+
Activity.Id: 00-7759221f2c5599489d455b84fa0f90f4-9a52f72c08a9d447-01
165+
Activity.DisplayName: SomeWork
166+
Activity.Kind: Internal
167+
Activity.StartTime: 2021-03-18T10:46:46.8634510Z
168+
Activity.Duration: 00:00:01.5402045
169+
Resource associated with Activity:
170+
service.name: MySample
171+
service.instance.id: 909a4624-3b2e-40e4-a86b-4a2c8003219e
172+
173+
Example work done
174+
```
175+
176+
#### Sources
177+
178+
In the example code you invoked `AddSource("Sample.DistributedTracing")` so that OpenTelemetry would
179+
capture the Activities produced by the ActivitySource that was already present in the code:
180+
181+
```csharp
182+
static ActivitySource s_source = new ActivitySource("Sample.DistributedTracing");
183+
```
184+
185+
Telemetry from any ActivitySource can captured by calling AddSource() with the source's name.
186+
187+
#### Exporters
188+
189+
The console exporter is helpful for quick examples or local development but in a production deployment
190+
you will probably want to send traces to a centralized store. OpenTelemetry supports a variety
191+
of destinations using different
192+
[exporters](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/glossary.md#exporter-library).
193+
See the [OpenTelemetry getting started guide](https://github.com/open-telemetry/opentelemetry-dotnet#getting-started)
194+
for more information on configuring OpenTelemetry.
195+
196+
## Collect traces using Application Insights
197+
198+
Distributed tracing telemetry is automatically captured after configuring the Application Insights SDK
199+
([ASP.NET](https://docs.microsoft.com/azure/azure-monitor/app/asp-net), [ASP.NET Core](https://docs.microsoft.com/azure/azure-monitor/app/asp-net-core))
200+
or by enabling [code-less instrumentation](https://docs.microsoft.com/azure/azure-monitor/app/codeless-overview).
201+
202+
See the [Application Insights distributed tracing documentation](https://docs.microsoft.com/azure/azure-monitor/app/distributed-tracing) for more
203+
information.
204+
205+
> [!NOTE]
206+
> Currently Application Insights only supports collecting specific well-known Activity instrumentation and will ignore new user added Activities. Application
207+
> Insights offers [TrackDependency](https://docs.microsoft.com/azure/azure-monitor/app/api-custom-events-metrics#trackdependency) as a vendor
208+
> specific API for adding custom distributed tracing information.
209+
210+
## Collect traces using custom logic
211+
212+
Developers are free to create their own customized collection logic for Activity trace data. This example collects the
213+
telemetry using the <xref:System.Diagnostics.ActivityListener?displayProperty=nameWithType> API provided by .NET and prints
214+
it to the console.
215+
216+
### Prerequisites
217+
218+
- [.NET Core 2.1 SDK](https://dotnet.microsoft.com/download/dotnet) or a later version
219+
220+
### Create an example application
221+
222+
First you will create an example application that has some distributed trace instrumentation but no trace data is being collected.
223+
224+
```dotnetcli
225+
dotnet new console
226+
```
227+
228+
Applications that target .NET 5 and later already have the necessary distributed tracing APIs included. For apps targeting older
229+
.NET versions add the [System.Diagnostics.DiagnosticSource NuGet package](https://www.nuget.org/packages/System.Diagnostics.DiagnosticSource/)
230+
version 5 or greater.
231+
232+
```dotnetcli
233+
dotnet add package System.Diagnostics.DiagnosticSource
234+
```
235+
236+
Replace the contents of the generated Program.cs with this example source:
237+
238+
```C#
239+
using System;
240+
using System.Diagnostics;
241+
using System.Threading.Tasks;
242+
243+
namespace Sample.DistributedTracing
244+
{
245+
class Program
246+
{
247+
static ActivitySource s_source = new ActivitySource("Sample.DistributedTracing");
248+
249+
static async Task Main(string[] args)
250+
{
251+
await DoSomeWork();
252+
Console.WriteLine("Example work done");
253+
}
254+
255+
static async Task DoSomeWork()
256+
{
257+
using (Activity a = s_source.StartActivity("SomeWork"))
258+
{
259+
await StepOne();
260+
await StepTwo();
261+
}
262+
}
263+
264+
static async Task StepOne()
265+
{
266+
using (Activity a = s_source.StartActivity("StepOne"))
267+
{
268+
await Task.Delay(500);
269+
}
270+
}
271+
272+
static async Task StepTwo()
273+
{
274+
using (Activity a = s_source.StartActivity("StepTwo"))
275+
{
276+
await Task.Delay(1000);
277+
}
278+
}
279+
}
280+
}
281+
```
282+
283+
Running the app does not collect any trace data yet:
284+
285+
```dotnetcli
286+
> dotnet run
287+
Example work done
288+
```
289+
290+
### Add code to collect the traces
291+
292+
Update Main() with this code:
293+
294+
```C#
295+
static async Task Main(string[] args)
296+
{
297+
Activity.DefaultIdFormat = ActivityIdFormat.W3C;
298+
Activity.ForceDefaultIdFormat = true;
299+
300+
Console.WriteLine(" {0,-15} {1,-60} {2,-15}", "OperationName", "Id", "Duration");
301+
ActivitySource.AddActivityListener(new ActivityListener()
302+
{
303+
ShouldListenTo = (source) => true,
304+
Sample = (ref ActivityCreationOptions<ActivityContext> options) => ActivitySamplingResult.AllDataAndRecorded,
305+
ActivityStarted = activity => Console.WriteLine("Started: {0,-15} {1,-60}", activity.OperationName, activity.Id),
306+
ActivityStopped = activity => Console.WriteLine("Stopped: {0,-15} {1,-60} {2,-15}", activity.OperationName, activity.Id, activity.Duration)
307+
});
308+
309+
await DoSomeWork();
310+
Console.WriteLine("Example work done");
311+
}
312+
```
313+
314+
The output now includes logging:
315+
316+
```dotnetcli
317+
> dotnet run
318+
OperationName Id Duration
319+
Started: SomeWork 00-bdb5faffc2fc1548b6ba49a31c4a0ae0-c447fb302059784f-01
320+
Started: StepOne 00-bdb5faffc2fc1548b6ba49a31c4a0ae0-a7c77a4e9a02dc4a-01
321+
Stopped: StepOne 00-bdb5faffc2fc1548b6ba49a31c4a0ae0-a7c77a4e9a02dc4a-01 00:00:00.5093849
322+
Started: StepTwo 00-bdb5faffc2fc1548b6ba49a31c4a0ae0-9210ad536cae9e4e-01
323+
Stopped: StepTwo 00-bdb5faffc2fc1548b6ba49a31c4a0ae0-9210ad536cae9e4e-01 00:00:01.0111847
324+
Stopped: SomeWork 00-bdb5faffc2fc1548b6ba49a31c4a0ae0-c447fb302059784f-01 00:00:01.5236391
325+
Example work done
326+
```
327+
328+
Setting <xref:System.Diagnostics.Activity.DefaultIdFormat> and
329+
<xref:System.Diagnostics.Activity.ForceDefaultIdFormat> is optional
330+
but helps ensure the sample produces similar output on different .NET runtime versions. .NET 5 uses
331+
the W3C TraceContext ID format by default but earlier .NET versions default to using
332+
<xref:System.Diagnostics.ActivityIdFormat.Hierarchical> ID format. See
333+
[Activity IDs](distributed-tracing-concepts.md#activity-ids) for more details.
334+
335+
<xref:System.Diagnostics.ActivityListener?displayProperty=nameWithType> is used to receive callbacks
336+
during the lifetime of an Activity.
337+
338+
- <xref:System.Diagnostics.ActivityListener.ShouldListenTo> - Each
339+
Activity is associated with an ActivitySource which acts as its namespace and producer.
340+
This callback is invoked once for each ActivitySource in the process. Return true
341+
if you are interested in performing sampling or being notified about start/stop events
342+
for Activities produced by this source.
343+
- <xref:System.Diagnostics.ActivityListener.Sample> - By default
344+
<xref:System.Diagnostics.ActivitySource.StartActivity%2A> does not
345+
create an Activity object unless some ActivityListener indicates it should be sampled. Returning
346+
<xref:System.Diagnostics.ActivitySamplingResult.AllDataAndRecorded>
347+
indicates that the Activity should be created,
348+
<xref:System.Diagnostics.Activity.IsAllDataRequested> should be set
349+
to true, and <xref:System.Diagnostics.Activity.ActivityTraceFlags>
350+
will have the <xref:System.Diagnostics.ActivityTraceFlags.Recorded>
351+
flag set. IsAllDataRequested can be observed by the instrumented code as a hint that a listener
352+
wants to ensure that auxilliary Activity information such as Tags and Events are populated.
353+
The Recorded flag is encoded in the W3C TraceContext ID and is a hint to other processes
354+
involved in the distributed trace that this trace should be sampled.
355+
- <xref:System.Diagnostics.ActivityListener.ActivityStarted> and
356+
<xref:System.Diagnostics.ActivityListener.ActivityStopped> are
357+
called when an Activity is started and stopped respectively. These callbacks provide an
358+
oportunity to record relevant information about the Activity or potentially to modify it.
359+
When an Activity has just started much of the data may still be incomplete and it will
360+
be populated before the Activity stops.
361+
362+
Once an ActivityListener has been created and the callbacks are populated, calling
363+
<xref:System.Diagnostics.ActivitySource.AddActivityListener(System.Diagnostics.ActivityListener)?displayProperty=nameWithType>
364+
initiates invoking the callbacks. Call
365+
<xref:System.Diagnostics.ActivityListener.Dispose?displayProperty=nameWithType> to
366+
stop the flow of callbacks. Beware that in multi-threaded code callback notifications in
367+
progress could be received while Dispose() is running or even very shortly after it has
368+
returned.

0 commit comments

Comments
 (0)