sample entire trace which contains span with specific tags

[seanyinx]

Feature:

To provide a way to sample entire trace which contains span with specific tags such as "error=true" or significant delay

Rational

Currently sampling is based on traceId. Normally we care much more about failed or slow requests instead of successful ones. With sampling on traceId, it's possible to miss important traces with error information; while it puts lots of pressure on underlying storage if we don't sample at all.

Example Scenario

if a trace contains no error or slow span, sample the trace before saving with a property xxx.ok-requests.sample=0.1
if a trace contains error or slow span, sample the trace before saving with a property xxx.ex-requests.sample=1.0

[jeqo]

Thanks for the request @seanyinx!

This is an interesting idea. I did a small poc a while ago https://github.com/jeqo/zipkin-storage-kafka-experiments/tree/master/sample-error-trace but could be better to have these extension modules as part of the project.

If you can take a look to this proof-of-concept and check if it solves your requirement, or you have something built already, please feel free to share it and we can see the best way to add it to the project.

[seanyinx]

@jeqo Thank you for providing your proof of concept! Yes, it solves my requirement. I added what I am trying to implement as a pull request (jeqo/zipkin-storage-kafka-experiments#2).

I am still getting familiar with this project and it seems possible to apply this sampling before saving traces to trace topic, so that we can save some space on Kafka. I am not sure if this idea fits with the project design though.