AWS X-Ray traces without span info

[jphjsoares]

Component

OpenTelemetry.Extensions.AWS

Package Version

Package Name	Version
OpenTelemetry	1.9.0
OpenTelemetry.Contrib.Extensions.AWSXRay	1.2.0
OpenTelemetry.Instrumentation.AspNetCore	1.8.1

Runtime Version

net8.0

Description

I currently have an application that is collecting metrics and traces using Open Telemetry. Locally, I have an Aspire board that shows me the correct traces with the correct span info. I'm trying to see the span info of my traces on AWS.

Steps to Reproduce

This is a snippet of the code I'm using for this example:

(...)
var builder = WebApplication.CreateBuilder(args);
ActivitySource activitySource = new("OTEL");
builder.Services.AddOpenTelemetry()
        .ConfigureResource(resource => resource
            .AddService(
                serviceName: builder.Environment.ApplicationName,
                serviceVersion: appVersion)
            .AddDetector(new ContainerResourceDetector()))
        .WithMetrics(metrics => metrics
            .AddAspNetCoreInstrumentation()
            .AddOtlpExporter()
        .WithTracing(tracing => tracing
            .AddXRayTraceId()
            .AddAWSInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddSource("OTEL")
            .AddOtlpExporter()
            .AddConsoleExporter());

Sdk.SetDefaultTextMapPropagator(new AWSXRayPropagator());

(...)

app.MapGet("/", (HttpContext context) =>
{
  using var slow = activitySource.StartActivity("Slow span", ActivityKind.Server);
  slow?.SetTag("type", "slow");
  Thread.Sleep(500);

  context.Response.WriteAsync($"Hello World!");

  using var fast = activitySource.StartActivity("Fast span", ActivityKind.Server);
  fast?.SetTag("type", "fast");
  Thread.Sleep(100);
});

Expected Result

I was expecting something similar to what my aspire board is showing, but on AWS side:

(Note that the delays here were different from the following evidence)

As seen above, the aspire board is showing me the GET /, Fastttt and the Slowwww info.

Actual Result

When testing it in AWS, even though the traces are received and so is the metadata, there is no span info displayed on the waterfall chart. Not even the response code is showing:

And a snippet of the corresponding metadata:

{
  "default": {
    (...)
    "url.scheme": "http",
    "otel.resource.service.name": "xxxx",
    "otel.resource.telemetry.sdk.version": "1.9.0",
    "url.path": "/",
    "http.request.method": "GET",
    "http.route": "/",
    "http.response.status_code": 200,
    "otel.resource.telemetry.sdk.language": "dotnet"
  }
}

Additional Context

No response

[fracampit]

I had the same problem.

I was able to fix it by adding some custom tags to the trace:

                    .AddAspNetCoreInstrumentation(o =>
                    {
                        o.EnrichWithHttpRequest = (activity, httpRequest) =>
                        {
                            activity.SetTag("http.method", httpRequest.Method);
                            activity.SetTag("http.url", $"{httpRequest.Scheme}://{httpRequest.Host}{httpRequest.Path}{httpRequest.QueryString}");
                            activity.SetTag("http.target", httpRequest.Path);
                            activity.SetTag("http.host", httpRequest.Host);
                            activity.SetTag("http.scheme", httpRequest.Scheme);
                            activity.SetTag("http.client_ip", httpRequest.HttpContext.Connection.RemoteIpAddress?.ToString());
                            activity.SetTag("http.user_agent", httpRequest.Headers.UserAgent.ToString());
                        };
                        o.EnrichWithHttpResponse = (activity, httpResponse) =>
                        {
                            activity.SetTag("http.status_code", httpResponse.StatusCode);
                            activity.SetTag("http.response_content_length", httpResponse.ContentLength);
                        };
                        o.EnrichWithException = (activity, exception) =>
                        {
                            var exceptionTags = new ActivityTagsCollection
                            {
                                { "exception.type", exception.GetType().FullName },
                                { "exception.message", exception.Message },
                                { "exception.stacktrace", exception.ToString() }
                            };
                            var exceptionEvent = new ActivityEvent("exception", DateTime.UtcNow, exceptionTags);
                            activity.AddEvent(exceptionEvent);

                            // Optionally set the Activity status to error
                            activity.SetStatus(ActivityStatusCode.Error, exception.Message);
                        };
                    })

I was also able to populate the origin of the trace with this configuration:

            .ConfigureResource(resourceBuilder =>
            {
                resourceBuilder
                    .AddService("######")
                    .AddTelemetrySdk()
                    .AddAttributes(new Dictionary<string, object>
                    {
                        ["cloud.provider"] = "aws",
                        ["cloud.platform"] = "aws_app_runner"
                    });
            })

Hope this helps!

[Kielek]

@ppittle, FYI

[AsakerMohd]

I would suggest looking into using Application Signals. It's built on top of the Opentelemetry Auto Instrumentation agent and it enriches the spans with AWS specific attributes to be able to see them on the CloudWatch console. AWS Otel Instrumentation package for reference and it works the same way that the Opentelemetry Auto Instrumentation agent does so you could follow the instructions there but use the zip files from the AWS Instrumentation release page.

[AsakerMohd]

The current version of Opentelemetry.Instrumentation.AWS should be doing that for you though since you are manually instrumenting your application. Do you mind sharing the version that you were using?