From 43079cd944ddc782a8eede1423dd2d70e9daf62f Mon Sep 17 00:00:00 2001 From: Carlos Alberto Cortez Date: Mon, 8 Jan 2024 10:58:16 -0600 Subject: [PATCH 1/5] Automatically propagate peer.service --- text/trace/0247-peer-service-propagation.md | 157 ++++++++++++++++++++ 1 file changed, 157 insertions(+) create mode 100644 text/trace/0247-peer-service-propagation.md diff --git a/text/trace/0247-peer-service-propagation.md b/text/trace/0247-peer-service-propagation.md new file mode 100644 index 000000000..6c9cc0d2e --- /dev/null +++ b/text/trace/0247-peer-service-propagation.md @@ -0,0 +1,157 @@ +# Automatic Peer Service Name propagation + +Automatic propagation of `peer.service` through `TraceState``. + +## Motivation + +Knowing the service name on the other side of a remote call is valuable +troubleshooting information. The semantic conventions represent this via +`peer.service`, which needs to be manually populated. In a deployment scenario, +when a new service is added, all the existing services interacting with it +need to update `peer.service`, which is error-prone and may +become unreliable, making it eventually obsolete. + +This information can be effectively derived in the backend using the +`Resource` of the parent `Span`, but is otherwise not available +at Collector processing time, where it could be used for sampling +and tranformation purposes. + +As metrics and logs do not have defined a parent-child relationship, using +`peer.service` could help gaining insight into the remote service as well. + +Defining (optional) automated population of `peer.service` will greatly help +adoption of this attribute by users and vendors explictly interested in this +scenario. + +## Explanation + +SDKs will define an optional feature, disabled by default, +to read the `service.name` attribute of the related `Resource` and set it +in the spans' `TraceState` as described in +[trace state handling](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/tracestate-handling.md) +specifically using the `sn` subkey (denoting **service name**): + +``` +ot=sn:MyService +``` + +Instrumentation and processors are then free to use this information to set +`peer.service` and perform other related processing. + +## Internal details + +SDKs will disable by default this option, maintaining the current behavior. +When the feature is explicitly enabled by the user, spans will include +an additional entry in `TraceState` as described above. By doing this, +the user acknowledges the additional cost in memory and bandwidth. + +Span creation will be updated like this: + +```java +// +// SpanBuilder.startSpan() +// +if (tracerSharedState.propagateServiceName) { + String serviceName = tracerSharedState.getResource().getAttribute(PEER_SERVICE); + traceState = addServiceNameToTracerState(traceState, serviceName); +} +// Use the updated `traceState` to create the new SpanContext. +``` + +Server-side instrumentation (e.g. http servers, gRPC on the receiver side) +can then be updated to use the propagated context to look for the `sn` subkey +in `TraceState`, and if it exists, use it to set `peer.service` on the local `Span`: + +```java +// +// Incoming request handling. +// +try (Scope scope = extractRemoteContext(headers).makeCurrent()) { + SpanBuilder spanBuilder = tracer.spanBuilder("server-span"); + + TraceState remoteTraceState = Span.current() + .getSpanContext() + .getTraceState(); + String peerServiceName = getPropagatedServiceName(remoteTraceState); + if (peerServiceName != null) { + spanBuilder.setAttribute("peer.service", peerServiceName); + } +} +``` + +With `peer.service` present in server spans, further processing, filtering and sampling can +then be accomplished in the Collector, e.g. a preview of the dependency map of a service, +similar in spirit to zPages could be created. + +## Trade-offs and mitigations + +Given the `TraceState` [lenght contrains](https://www.w3.org/TR/trace-context/#tracestate-header) +we may decide to trim the service name up to a given length. + +In case propagating `peer.service` ever represents a privacy or security concern, +consider hashing the `peer.service` values, and provide a dictionary to interpret them +by the Collector and backends. + +## Prior art and alternatives + + Using `Baggage` to **automatically** propagate `service.name` was explored. +It would consist of two parts: + +* Explicit `Resource`'s `service.name` propagation using `Baggage` + at **request** time. Instrumentation libraries would need to include + an option to perform such propagation (potentially false by default, + in order to keep the current behavior). Caveat is that `Resource` is an SDK item, while + instrumentation is expected to solely rely on the API. +* Either explicit handling on the server-side instrumentation (similar to how + it's proposed using `TraceState` above, but relying on `Baggage ` instead), or + specialized processors that automatically enrich `Spans` with `Baggage` values, + as shown below: + +```java +public class BaggageDecoratingSpanProcessor implements SpanProcessor { + public BaggageDecoratingSpanProcessor(SpanProcessor processor, Predicate predicate, Set keys) { + this.processor = processor; + this.predicate = predicate; + this.keys = keys; + } + + public void onStart(Context context, ReadWriteSpan span) { + this.processor.onStart(context, span); + if (predicate.test(span)) { + Baggage baggage = Baggage.current(); + keys.forEach(key -> { + String value = baggage.getEntryValue(key); + if (value != null) { + span.setAttribute(key, baggage.getEntryValue(key)) + } + }) + } + } +} +``` + +The `TraceState` alternative was preferred as `Baggage` has general, +application-level propagation purposes, whereas `TraceState` can be used +by observability purposes, along with the fact that accessing `Resource` +from instrumentation is not feasible. + +## Open questions + +* At the moment of writing this OTEP only `peer.service` is defined (which + relies on `service.name`). However, semantic conventions also define + `service.version`, `service.instance.id` and `service.namespace`, + which may provide additional details. Given the contraints of memory + and bandwidth (for both `TraceState` and `Baggage`) we will decide + in the future whether to propagate these additional values or not. + +## Future possibilities + +Logging and metrics can be augmented by using automatic `peer.service` propagation, +in order to hint at a parent-child (or client-server) relationship, given they do not +include such information as part of their data models: + +* Logs can optionally be converted to traces if a hierarchy or dependency map is desired, + but augmenting them with `peer.service` could be done as an intermediate step. +* Metrics could use `peer.service` as an additional dimension that helps performing filtering + based on related services, for example. + From 10ad05d2c74c920bd65d47ac718048ad654767f1 Mon Sep 17 00:00:00 2001 From: Carlos Alberto Cortez Date: Fri, 19 Jan 2024 16:19:08 +0100 Subject: [PATCH 2/5] Use 'us' instead of 'sn' in the tracestate subkey. --- text/trace/0247-peer-service-propagation.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/text/trace/0247-peer-service-propagation.md b/text/trace/0247-peer-service-propagation.md index 6c9cc0d2e..a5df722ef 100644 --- a/text/trace/0247-peer-service-propagation.md +++ b/text/trace/0247-peer-service-propagation.md @@ -20,7 +20,7 @@ As metrics and logs do not have defined a parent-child relationship, using `peer.service` could help gaining insight into the remote service as well. Defining (optional) automated population of `peer.service` will greatly help -adoption of this attribute by users and vendors explictly interested in this +adoption of this attribute by users and vendors explicitly interested in this scenario. ## Explanation @@ -29,10 +29,10 @@ SDKs will define an optional feature, disabled by default, to read the `service.name` attribute of the related `Resource` and set it in the spans' `TraceState` as described in [trace state handling](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/tracestate-handling.md) -specifically using the `sn` subkey (denoting **service name**): +specifically using the `us` subkey (denoting **upstream service**): ``` -ot=sn:MyService +ot=us:MyService ``` Instrumentation and processors are then free to use this information to set @@ -59,7 +59,7 @@ if (tracerSharedState.propagateServiceName) { ``` Server-side instrumentation (e.g. http servers, gRPC on the receiver side) -can then be updated to use the propagated context to look for the `sn` subkey +can then be updated to use the propagated context to look for the `us` subkey in `TraceState`, and if it exists, use it to set `peer.service` on the local `Span`: ```java @@ -72,9 +72,9 @@ try (Scope scope = extractRemoteContext(headers).makeCurrent()) { TraceState remoteTraceState = Span.current() .getSpanContext() .getTraceState(); - String peerServiceName = getPropagatedServiceName(remoteTraceState); + String peerServiceName = getUpstreamServiceName(remoteTraceState); if (peerServiceName != null) { - spanBuilder.setAttribute("peer.service", peerServiceName); + spanBuilder.setAttribute(PEER_SERVICE, peerServiceName); } } ``` From c8b500a67bc09592f3c798ea65ce1a516da8416e Mon Sep 17 00:00:00 2001 From: Carlos Alberto Cortez Date: Mon, 22 Jan 2024 17:47:15 +0100 Subject: [PATCH 3/5] Add sampling examples. --- text/trace/0247-peer-service-propagation.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/text/trace/0247-peer-service-propagation.md b/text/trace/0247-peer-service-propagation.md index a5df722ef..f0e92cddb 100644 --- a/text/trace/0247-peer-service-propagation.md +++ b/text/trace/0247-peer-service-propagation.md @@ -13,8 +13,8 @@ become unreliable, making it eventually obsolete. This information can be effectively derived in the backend using the `Resource` of the parent `Span`, but is otherwise not available -at Collector processing time, where it could be used for sampling -and tranformation purposes. +at Collector processing time, where it could be used for transformation +purposes or sampling (e.g. adaptive sampling based on the calling service). As metrics and logs do not have defined a parent-child relationship, using `peer.service` could help gaining insight into the remote service as well. @@ -83,6 +83,20 @@ With `peer.service` present in server spans, further processing, filtering and s then be accomplished in the Collector, e.g. a preview of the dependency map of a service, similar in spirit to zPages could be created. +### Sampling scenarios + +A specially interesting case is sampling depending on the calling service, specifically: + +* An adaptive sampler may decide to sample or not based on the calling service, e.g. + given Service A amounting to 98% of requests, and Service B amounting to 2% only, + more traces could be sampled for the latter. +* In cases where a parent `Span ` is **not** sampled **but** its child (or linked-to `Span`) + wants to sample, knowing the calling service **may** help with the sampling decision. +* In deployment scenarios where context is properly propagated through all the services, + but not all of them are actually traced, it would be helpful to know what services + were part of the request, even if no traces/spans exist for them, see + https://github.com/w3c/trace-context/issues/550 as an example. + ## Trade-offs and mitigations Given the `TraceState` [lenght contrains](https://www.w3.org/TR/trace-context/#tracestate-header) From 5b8cc6a429593470d1dd8f7b81d1ad9dc7b2dafe Mon Sep 17 00:00:00 2001 From: Carlos Alberto Cortez Date: Mon, 22 Jan 2024 17:51:36 +0100 Subject: [PATCH 4/5] Details. --- text/trace/0247-peer-service-propagation.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/text/trace/0247-peer-service-propagation.md b/text/trace/0247-peer-service-propagation.md index f0e92cddb..e63c273bb 100644 --- a/text/trace/0247-peer-service-propagation.md +++ b/text/trace/0247-peer-service-propagation.md @@ -92,10 +92,11 @@ A specially interesting case is sampling depending on the calling service, speci more traces could be sampled for the latter. * In cases where a parent `Span ` is **not** sampled **but** its child (or linked-to `Span`) wants to sample, knowing the calling service **may** help with the sampling decision. + Right now only a parent span id is available in such case. * In deployment scenarios where context is properly propagated through all the services, but not all of them are actually traced, it would be helpful to know what services - were part of the request, even if no traces/spans exist for them, see - https://github.com/w3c/trace-context/issues/550 as an example. + were part of the request, even if no traces/spans exist for them. See + https://github.com/w3c/trace-context/issues/550 . ## Trade-offs and mitigations From 21ca221445d07af69f73dbcee4a652ab282a747b Mon Sep 17 00:00:00 2001 From: Carlos Alberto Cortez Date: Tue, 23 Jan 2024 16:53:01 +0100 Subject: [PATCH 5/5] Clarify. --- text/trace/0247-peer-service-propagation.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/text/trace/0247-peer-service-propagation.md b/text/trace/0247-peer-service-propagation.md index e63c273bb..ad4f5b97a 100644 --- a/text/trace/0247-peer-service-propagation.md +++ b/text/trace/0247-peer-service-propagation.md @@ -83,20 +83,24 @@ With `peer.service` present in server spans, further processing, filtering and s then be accomplished in the Collector, e.g. a preview of the dependency map of a service, similar in spirit to zPages could be created. -### Sampling scenarios +### Use scenarios -A specially interesting case is sampling depending on the calling service, specifically: +Sampling can benefit from knowing the calling service, specifically: * An adaptive sampler may decide to sample or not based on the calling service, e.g. given Service A amounting to 98% of requests, and Service B amounting to 2% only, - more traces could be sampled for the latter. + with a default sampling rate of 50%, more traces of the latter service could be preserved, + as opposed to running the risk of no traces at all. * In cases where a parent `Span ` is **not** sampled **but** its child (or linked-to `Span`) wants to sample, knowing the calling service **may** help with the sampling decision. Right now only a parent span id is available in such case. -* In deployment scenarios where context is properly propagated through all the services, - but not all of them are actually traced, it would be helpful to know what services - were part of the request, even if no traces/spans exist for them. See - https://github.com/w3c/trace-context/issues/550 . + +In deployment scenarios where context is properly propagated through **all** the services, +but not all of them are traced (i.e. use the no-op implementation), it would be helpful to know what services +were actually part of the request, even if not observed. This could help with confusion and false +positives. Observe this cannot be currently +computed at the backend with OTel, as non-traced systems will simply send no telemetry +whatsoever. See https://github.com/w3c/trace-context/issues/550 ## Trade-offs and mitigations