Adding custom attributes to traces at step runtime

[pfiaux]

Is your feature request related to a problem? Please describe.
Currently the attributes added to a trace (in open telemetry at least) by buildkite are static. For example a trace stored in tempo had the following attribute keys on the root span:

buildkite.agent	
buildkite.branch	
buildkite.build_id	
buildkite.build_number	
buildkite.build_url	
buildkite.job_id	
buildkite.job_key	
buildkite.job_label	
buildkite.job_url	
buildkite.org	
buildkite.parallel	
buildkite.pipeline	
buildkite.queue	
buildkite.rebuilt_from_id	
buildkite.retry	
buildkite.source	
buildkite.triggered_from_id	
buildkite.version	
cluster	
deployment.environment	
service.name	
service.version	
service_component	

There's no way to add another attribute dynamically at run time (or statically either currently).

Yet this could be useful to allow filtering by different dimensions:

• Commit hash
• Team of author (which might or might not be related to the buildkite team but come from another source)
• ...

This would give flexibility to extend traces as needed without having to cover too many cases statically in the agent and makes them very powerful.

Describe the solution you'd like
The CLI could allow setting an attribute (ideally at the root span level) dynamically at run time:

# Example cli interface:
buildkite-agent tracing set-attribute "attr-name" "json-encoded-attr-value"

This would then be propagated to the spans (except already closed ones, i.e. all the pre-run ones).

According to the open telemetry attribute docs the value can be primitives, or an array of primitives. Using json encoded would make it easy to tell if for example 1 is a string or a number and to make arrays as opposed to a plain string value. Or it could be another encoding.

Describe alternatives you've considered
There's lots of ways attributes could be added/requested and this could be done at different times of the lifecycle.

There would be other ways to explicitly or implicitly expose attributes:

• Static attributes could be added via ENV/Config/Startup flags like other settings
• Expose buildkite meta data variables as attributes
  • only if they match a default or customizable prefix (e.g. trace_*)
  • list keys that need to be exposed/forwarded as a global settings
• Expose env variables as attributes if they match a certain patter (e.g. TRACE_* or more backend specific OTEL_ATTR_*)
• Expose env variables from a list of exposed ones (e.g. `TRACE_FORWARD_ENV_VARS="custom1 custom2 custom.experiment*"

Additional context
I'm not sure how close or different datadog is from open telemetry, ideally this feature could be done in a way that supports both backends to avoid as much backend specific code as possible.

Would buildkite accept a PR allowing addition of trace attributes? If so what kind of solution/approach would be accepted?

[moskyb]

hi @pfiaux! this sounds like a great idea, and we'd be happy to accept PRs to this effect.

i think the CLI definition you've proposed looks good, though if we were bring it more in line with the current CLI tools it'd probably be more like:

buildkite-agent tracing attribute set --key foo --value bar

i think that adding attributes statically at agent start time should be relatively straightforward - we could just add another cli flag. adding them during a job run would be more complicated, but definitely still doable.

the best way forward would likely be the Job API, a REST API that allows hooks (like the command hook, which is what runs your job) to communicate with the state of the job as a whole. you can check out the Job API in action in the buildkite-agent lock and buildkite-agent env command groups. Note that it's currently experimental, but we're planning on graduating it soon.

[pfiaux]

We were interested in adding it at job time, it's a little bit less important for us now because we can't use the traces for the metrics we wanted to grab (the agent traces are unfortunately at the step rather than the pipeline level but that's another topic).

I agree that your proposal is more like the current interface, but I'm not sure how to handle the types using that signature.

# i.e. integer vs string:
buildkite-agent tracing attribute set --key int --value 1
# vs
buildkite-agent tracing attribute set --key str --value "1"

# or arrays of primitives:
buildkite-agent tracing attribute set --key int_arr --value 1,2,3

That's why I suggested JSON, which is supported by some commands like pipeline upload so it's not entirely strange... It would make it less straight forward to implement, but it could help with multiple values in 1 go:

# setting multiple attribues
buildkite-agent tracing upload-attributes "{
  "int": 1,
  "str": "1",
  "int_arr": [1,2,3]
}"

I'm not sure I got what you meant by using the Job API. This one right https://buildkite.com/docs/apis/rest-api/jobs? You mean instead of using the cli to set a the value we could use the API or the implementation of the CLI should use the API?