Dynamic env tag results in incorrectly tagged trace metrics

[burns47]

Describe the bug

We're running a multitenant API deployment and we're using the env tag to identify the customer that issued the request. As a result, we're manually specifying env using tracer.ActiveScope.Span.SetTag("env", ...), and everything seems to work just fine w.r.t the reported traces and their sub-spans.

However, certain automatically generated metrics (e.g. trace.aspnet_core.request.hits) seem to confuse their env tag values with other requests that are running at the same time.

To Reproduce
Use this endpoint and call it with different envs:

        [HttpGet]
        [AllowAnonymous]
        [Route("test")]
        public async Task<ActionResult> RunRequestWithDynamicTag(string env) {

            //setting env on the trace results in env appearing on trace.aspnet_core.request.hits,
            //but it appears to be incorrect if other requests in the process are also running with different envs
            var tracer = Tracer.Instance;
            tracer.ActiveScope.Span.SetTag("env", env);

            await Task.Delay(5000);

            //this metric works just fine, however, and reports different results
            DogStatsd.Increment("requests.total", 1, tags: new string[] { $"env:{env}" }); 

            return Ok();
        }

Expected behavior
Trace metrics respect the env tag set during the trace.

Screenshots

On the left is a custom metric where we report the env tag explicitly, on the right is trace.aspnet_core.request.hits.

Runtime environment (please complete the following information):

• Instrumentation mode: automatic
• Tracer version: multiple versions, tried with the latest as well
• OS: linux (mcr.microsoft.com/dotnet/aspnet:6.0)
• CLR: .NET 6.0

[burns47]

Looking around a bit more, it seems the source of these metrics is StatsAggregator and StatsBuffer, which keys hits/duration/etc based on a StatsAggregationKey, which doesn't include env. So it would seem traces for different envs might be mixed together w.r.t these metrics. I would expect these stats to be bucketed based on the incoming Span's Context.TraceContext.Environment as well, maybe?

I don't know much about this codebase but appreciate any help in advance!