spanprofiler: do less work on unsampled traces

[bboreham]

What this PR does:

Skip the work to examine the span and decorate it, if the whole trace is not sampled hence the data is going nowhere.
This is 4% of all garbage in one production Mimir installation I looked at, maybe 2% of all CPU.

BenchmarkTracer/unsampled-28             2866868               405.6 ns/op           979 B/op         10 allocs/op
BenchmarkTracer/sampled-28               1000000              1024 ns/op            2346 B/op         25 allocs/op

Checklist

• Tests updated
• CHANGELOG.md updated

[kolesnikovae]

I probably should have made this more clear, but there is a reason for collecting sampled traces: span_name attribution (aggregated span profiles):

dskit/spanprofiler/spanprofiler.go

Lines 66 to 71 in 5141842

	} else {
	// Even if the trace has not been sampled, we still need to keep track
	// of samples that belong to the span (all spans with the given name).
	ctx = pprof.WithLabels(parentCtx, pprof.Labels(
	spanNameLabelName, operationName))
	}

On the other hand, I suppose that probabilistic trace sampling might not impact the results of profiling sampling (structurally and proportionally), so that even a fraction of span profiles forms a trustworthy picture.

I guess it might make sense to add an option that controls the behaviour

[bboreham]

Sorry I missed that nuance.

It will affect the result: suppose a particular span name does 1x work in traces sampled 0.01 and 100x work in traces sampled 0.1.

But by this reasoning, the aggregate seems low-value. I need to know the cost in context.

[kolesnikovae]

I'm going to add an option that controls this behavior. By default, sampled traces will be accounted for in span aggregate profiles. Would you mind?