How can I generate RPC traces from my websocket server?

[cjgatto]

My app is using

• .Net core 3.1
• SignalR 1.1.0
• OpenTelemetry instrumentation 1.0.0-rc9.12

For trace consumption, I am using

• OpenTelemetry Collector 0.70.0
• Elastic (APM, Elasticsearch, Kibana) 8.2.0

I have a .Net server that runs a SignalR Hub to serve UI messages and perform RPC requests. I'm setting up my services like so:

public void ConfigureServices(IServiceCollection services)
{
    ...
    services.AddHttpContextAccessor().AddSignalR(hubOptions =>
        {
            ...
        });

    services.AddOpenTelemetry()
        .WithTracing(builder => builder
            .SetResourceBuilder(ResourceBuilder
                .CreateDefault()
                .AddService(serviceName))
            .AddSource(serviceName)
            .AddAspNetCoreInstrumentation()
            .AddConsoleExporter()
            .AddOtlpExporter(options =>
            {
                options.Endpoint = new Uri(OtelCollectorUrl);
                options.Protocol = OtlpExportProtocol.HttpProtobuf;
            }))
        .StartWithHost();
    ...
}

In my Hub class I'm injecting a Tracer and calling it during my RPC methods like so:

private readonly Tracer _tracer;

public SignalHub(..., Tracer tracer)
{
    _tracer = tracer;
}

// This is my RPC method
public Task HealthRequest(int id)
{
    using var span = _tracer.StartActiveSpan("health-request");
    
    // Do work
}

The issue I'm having is that spans created to measure health-request are nested under the trace for the persistent websocket connection. OTEL exports these spans immediately with the correct trace.id, but the trace itself isn't exported until the connection is closed. As such, my APM server will not render a trace and the corresponding spans until the entire trace is complete.

I understand why it is working this way, the RPC requests are children of the initial HTTP request that was upgraded to a websocket. However, doing it this way isn't as helpful for distributed tracing. The RPC request is more contextually relevant as a single open-and-close request that I can associate with traces from other applications.

What I really need
It would be optimal if .AddAspNetCoreInstrumentation() and .AddHttpClientInstrumentation() had a way to filter on websocket connections that allow for unique traces during RPC method handling. In lieu of that, I think I'm looking for a way to create discrete Traces for any method. I'd like to build a trace each time my RPC method is called so I can export and render them quickly. It would be nice if I could relate these RPC method traces to the underlying websocket connection trace.

--

The following screencap is the result of opening my websocket connection, processing a few "health-request" requests, and shutting down the websocket connection. This trace and "health-request" data is only available when the connection has closed.

[cijothomas]

@rajkumar-rangaraj might be able to help. ?

[cjgatto]

👋 Hello, any chance someone might be able to provide guidance on this? Would really be appreciated