read resource attributes from annotations #3204

zeitlinger · 2024-08-07T15:04:54Z

Testing: Added integration test

Documentation: will be added to https://opentelemetry.io/docs/kubernetes/operator/automatic/ once released

jaronoff97 · 2024-08-07T17:23:12Z

per our discussion, @zeitlinger will rebase and expand on the capabilities introduced by #2330 :)

jaronoff97

Can you also update the README.md, explaining the changes and also explaining the precedence of deciding these variables?

pkg/instrumentation/sdk_test.go

jaronoff97

@zeitlinger last thing – can you update the README.md with some more information about the new behavior? Maybe also include it in the subtext of the chlog entry

pkg/instrumentation/sdk.go

TylerHelmuth · 2024-08-13T17:14:38Z

pkg/instrumentation/sdk.go

+	if name := chooseLabelOrAnnotation(pod, semconv.ServiceNameKey, constants.AnnotationAppName); name != "" {
+		return name
+	}


I am concerned that this would be a breaking change.

yes, I'll include that in the changelog

@jaronoff97 @pavolloffay I am pretty worried about this breaking change. Changes to service.name will change emitted telemetry, which could affect end-users alerts/SLOs/dashboards/etc. This change will (likely) not be detected with unit or e2e tests before a user updates their operator. Additionally, the operator's auto-instrumentation injection is already a huge obfuscation on top of already obfuscated auto-instrumentation, which means that our end-users are even less likely to understand the impact of this change.

I would much rather prefer this work be implemented in such a way that the annotation is used as the service name if no other service name is set OR if some other explicit configuration field is set that enables this behavior.

The reason I call this out as a breaking change in the current implementation is that it is now defaulting to the annotation value if it exists. Since it is a common annotation, it will probably be on most end-user's pods, resulting in the potential breaking change.

I would much rather this be the last option in this list to avoid the breaking change.

In addition, there is no stable semantic convention that ranks which value should be used as the service.name. I'd prefer avoiding breaking changes on this topic until there is a stable semantic convention, so that we can minimize the number of breaking changes.

open-telemetry/semantic-conventions#236

we could solve this problem if we let the user define the mapping they prefer - so it would be an opt-in feature

something like this - should be similar to https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#use-pod-fields-as-values-for-environment-variables

we can even use this public method (passing in a fieldRef): https://github.com/kubernetes/kubernetes/blob/a9593d634c6a053848413e600dadbf974627515f/staging/src/k8s.io/kubectl/pkg/cmd/set/env/env_resolve.go#L248

apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: my-instrumentation spec: resource: attributes: - key: service.name valueFrom: fieldRef: fieldPath: metadata.labels['test.example.com/key']

from SIG:
I think we should move forward with this by putting it within a new "defaults" section in the instrumentation CRD as so:

apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: my-instrumentation spec: defaults: useLabelsForResource: true

We also must then document in the readme the precedence for the automatic variables we set for users and how this boolean changes that.

I have useLabelsForResourceAttributes now

However, I've noticed that it could also make sense here:

opentelemetry-operator/apis/v1alpha1/instrumentation_types.go

Line 94 in 20c4bb7

AddK8sUIDAttributes bool `json:"addK8sUIDAttributes,omitempty"`

WDYT?

pkg/instrumentation/sdk.go

zeitlinger · 2024-08-14T08:45:49Z

@zeitlinger last thing – can you update the README.md with some more information about the new behavior? Maybe also include it in the subtext of the chlog entry

done

pavolloffay · 2024-08-30T13:01:27Z

CI failed

README.md

apis/v1alpha1/instrumentation_types.go

pkg/instrumentation/apachehttpd.go

pkg/instrumentation/sdk.go

zeitlinger · 2024-08-30T20:38:48Z

@jaronoff97 please check again

jaronoff97

another thought, also if we could add something in to an existing e2e test that would be great :) thank you! FWIW, i do think I will be taking on a refactor to make this logic a bit more straightforward in the future.

README.md

pkg/instrumentation/sdk.go

Co-authored-by: Jacob Aronoff <[email protected]>

zeitlinger · 2024-09-04T07:29:34Z

another thought, also if we could add something in to an existing e2e test that would be great :)

done @jaronoff97

pkg/instrumentation/sdk.go

jaronoff97 · 2024-09-06T14:17:53Z

pkg/instrumentation/sdk.go

+	for k, v := range pod.Annotations {
+		if strings.HasPrefix(k, constants.ResourceAttributeAnnotationPrefix) {
+			k = strings.TrimPrefix(k, constants.ResourceAttributeAnnotationPrefix)
+			if !existingRes[k] && k != string(semconv.ServiceNameKey) {


don't we want to let the override happen here?

we treat the user defined enviroment variables as the highest priority - otherwise it would be a breaking change.
Not sure if it would also be surprising.

https://github.com/zeitlinger/opentelemetry-operator/blob/c1966659f203516c87991715c95f1dc00a372aa1/README.md?plain=1#L783

zeitlinger · 2024-09-09T12:00:12Z

@jaronoff97 please check again

jaronoff97 · 2024-09-10T17:26:31Z

pkg/instrumentation/sdk_test.go

@@ -259,7 +529,7 @@ func TestSDKInjection(t *testing.T) {
 							Env: []corev1.EnvVar{
 								{
 									Name:  "OTEL_SERVICE_NAME",
-									Value: "explicitly_set",
+									Value: "explicit-name",


OOC – why change this for the test?

to make sure it's coming from the right place

jaronoff97

two minor questions, neither blocking. overall i think this looks good. Going to wait on @TylerHelmuth's review before merging.

zeitlinger requested a review from a team as a code owner August 7, 2024 15:04

jaronoff97 mentioned this pull request Aug 7, 2024

Use opentelemetry.io* namespace for extracing resource and attribute labels from Kubernetes manifests #2181

Closed

zeitlinger force-pushed the resource-attribute-from-annotations branch from cfe0591 to a2f5aa4 Compare August 7, 2024 17:27

jaronoff97 reviewed Aug 7, 2024

View reviewed changes

pkg/instrumentation/sdk_test.go Outdated Show resolved Hide resolved

zeitlinger force-pushed the resource-attribute-from-annotations branch from 5edb85e to 220650b Compare August 12, 2024 09:44

jaronoff97 approved these changes Aug 13, 2024

View reviewed changes

TylerHelmuth reviewed Aug 13, 2024

View reviewed changes

pavolloffay approved these changes Aug 14, 2024

View reviewed changes

zeitlinger force-pushed the resource-attribute-from-annotations branch 2 times, most recently from 8745d54 to baf151c Compare August 30, 2024 12:45

zeitlinger force-pushed the resource-attribute-from-annotations branch from 20c4bb7 to a273927 Compare August 30, 2024 15:16

jaronoff97 reviewed Aug 30, 2024

View reviewed changes

README.md Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

apis/v1alpha1/instrumentation_types.go Outdated Show resolved Hide resolved

pkg/instrumentation/apachehttpd.go Outdated Show resolved Hide resolved

pkg/instrumentation/sdk.go Outdated Show resolved Hide resolved

jaronoff97 reviewed Aug 30, 2024

View reviewed changes

pkg/instrumentation/sdk.go Show resolved Hide resolved

jaronoff97 reviewed Aug 30, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

pkg/instrumentation/sdk.go Show resolved Hide resolved

zeitlinger force-pushed the resource-attribute-from-annotations branch from a7c7322 to cc6338a Compare September 3, 2024 11:12

zeitlinger added 11 commits September 4, 2024 08:32

read resource attributes from annotations

7eb63af

use labels

407ea97

changelog entry

882a9c7

fix rebase

65d9aa7

change prefix to "resource.opentelemetry.io/"

47a7fec

change prefix to "resource.opentelemetry.io/"

7059349

fix rebase

930dd3d

update changelog and docs

ef827b2

changes from feedback

7733e72

changes from feedback

5b0fbde

generate

182ee4b

zeitlinger and others added 10 commits September 4, 2024 08:33

Update pkg/instrumentation/apachehttpd.go

34c51d6

Co-authored-by: Jacob Aronoff <[email protected]>

pr review

9bb63f0

comments

091c26f

lint

063eebe

lint

4bc7582

Update README.md

79b79ea

Co-authored-by: Jacob Aronoff <[email protected]>

change priority to avoid breaking change

976f68e

add e2e test

4366c12

add e2e test

e549a5b

add e2e test

2e9f6ec

zeitlinger force-pushed the resource-attribute-from-annotations branch from cc6338a to 2e9f6ec Compare September 4, 2024 06:34

Merge branch 'main' into resource-attribute-from-annotations

c196665

jaronoff97 reviewed Sep 6, 2024

View reviewed changes

pr review

f0b4c6a

jaronoff97 reviewed Sep 10, 2024

View reviewed changes

jaronoff97 approved these changes Sep 10, 2024

View reviewed changes

jaronoff97 requested a review from TylerHelmuth September 10, 2024 17:27

zeitlinger added 2 commits September 11, 2024 14:53

Merge branch 'main' into resource-attribute-from-annotations

247720e

use pod methods

1bc62e5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

read resource attributes from annotations #3204

read resource attributes from annotations #3204

zeitlinger commented Aug 7, 2024 •

edited

Loading

jaronoff97 commented Aug 7, 2024

jaronoff97 left a comment

jaronoff97 left a comment

TylerHelmuth Aug 13, 2024

zeitlinger Aug 14, 2024

TylerHelmuth Aug 28, 2024

TylerHelmuth Aug 28, 2024

TylerHelmuth Aug 28, 2024 •

edited

Loading

zeitlinger Aug 28, 2024 •

edited

Loading

jaronoff97 Aug 29, 2024

zeitlinger Aug 30, 2024

zeitlinger commented Aug 14, 2024

pavolloffay commented Aug 30, 2024

zeitlinger commented Aug 30, 2024

jaronoff97 left a comment

zeitlinger commented Sep 4, 2024

jaronoff97 Sep 6, 2024

zeitlinger Sep 9, 2024

zeitlinger commented Sep 9, 2024

jaronoff97 Sep 10, 2024

zeitlinger Sep 11, 2024

jaronoff97 left a comment

read resource attributes from annotations #3204

Are you sure you want to change the base?

read resource attributes from annotations #3204

Conversation

zeitlinger commented Aug 7, 2024 • edited Loading

jaronoff97 commented Aug 7, 2024

jaronoff97 left a comment

Choose a reason for hiding this comment

jaronoff97 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TylerHelmuth Aug 28, 2024 • edited Loading

Choose a reason for hiding this comment

zeitlinger Aug 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeitlinger commented Aug 14, 2024

pavolloffay commented Aug 30, 2024

zeitlinger commented Aug 30, 2024

jaronoff97 left a comment

Choose a reason for hiding this comment

zeitlinger commented Sep 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeitlinger commented Sep 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaronoff97 left a comment

Choose a reason for hiding this comment

zeitlinger commented Aug 7, 2024 •

edited

Loading

TylerHelmuth Aug 28, 2024 •

edited

Loading

zeitlinger Aug 28, 2024 •

edited

Loading