S3Storage EgressExtension cant load AWS token from the service account mounted in a EKS POD

[miguepintor]

Description

Given an EKS cluster v 1.29 with the EKS Pod Identity Agent installed and an OIDC identity provider created.

1. Create a K8s SA linked to an AWS IAM Role thru IRSA or EKS Pod Identity
2. Spawn a pod with a carside container like this:

apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  containers:
  - name: app
    image: my-dotnet-app:latest
    resources:
      requests:
        cpu: 500m
        memory: 500Mi
    env:
    - name: DOTNET_DiagnosticPorts
      value: /diag/dotnet-monitor.sock
    volumeMounts:
    - mountPath: /diag
      name: diag-vol
  - name: monitor
    image: mcr.microsoft.com/dotnet/monitor:8
    args: [ "collect", "--no-auth" ]
    imagePullPolicy: Always
    env:
     - name: DOTNETMONITOR_DiagnosticPort__ConnectionMode
       value: Listen
     - name: DOTNETMONITOR_Storage__DefaultSharedPath
       value: /diag
     - name: DOTNETMONITOR_Urls
       value: http://localhost:52323
     - name: DOTNETMONITOR_Metrics__Endpoints
       value: http://+:52325
     - name: DOTNETMONITOR_Egress__S3Storage__dump__bucketName
       value: my-bucket
     - name: DOTNETMONITOR_Egress__S3Storage__dump__regionName
       value: eu-west-1
    volumeMounts:
    - mountPath: /diag
      name: diag-vol
    resources:
      requests:
        cpu: 500m
        memory: 500Mi
  volumes:
  - name: diag-vol
    emptyDir: {}
  serviceAccountName: my-sa

3. Port forward the monitor pod port to your local machine

kubectl port-forward pod/test 52323:52323

4. Then when you try to upload a dump to the S3 bucket

curl -i "http://localhost:52323/dump?egressProvider=dump"

The dontet-monitor throws the following error;
{"Timestamp":"2024-08-25T17:50:51.9854761Z","EventId":91,"LogLevel":"Information","Category":"Microsoft.Diagnostics.Tools.Monitor.Egress.EgressExtension","Message":"Extension process 46 wrote output: {\u0022Succeeded\u0022:false,\u0022FailureMessage\u0022:\u0022Assembly AWSSDK.SecurityToken could not be found or loaded. This assembly must be available at runtime to use Amazon.Runtime.AssumeRoleAWSCredentials.\u0022,\u0022ArtifactPath\u0022:null}","State":{"Message":"Extension process 46 wrote output: {\u0022Succeeded\u0022:false,\u0022FailureMessage\u0022:\u0022Assembly AWSSDK.SecurityToken could not be found or loaded. This assembly must be available at runtime to use Amazon.Runtime.AssumeRoleAWSCredentials.\u0022,\u0022ArtifactPath\u0022:null}","pid":46,"message":"{\u0022Succeeded\u0022:false,\u0022FailureMessage\u0022:\u0022Assembly AWSSDK.SecurityToken could not be found or loaded. This assembly must be available at runtime to use Amazon.Runtime.AssumeRoleAWSCredentials.\u0022,\u0022ArtifactPath\u0022:null}","{OriginalFormat}":"Extension process {pid} wrote output: {message}"},"Scopes":[{"Message":"ArtifactType:dump ArtifactSource_ProcessId:1 ArtifactSource_RuntimeInstanceCookie:694f2cff7a5745e48b84ed75f3900ca3","ArtifactType":"dump","ArtifactSource_ProcessId":"1","ArtifactSource_RuntimeInstanceCookie":"694f2cff7a5745e48b84ed75f3900ca3"}]}

⚠️ Note:
The exact same SA works like a charm when mounted in another pod. e.g. in a pod with an aws-cli container:

apiVersion: v1
kind: Pod
metadata:
  name: dev-pod
spec:
  containers:
    - name: aws-cli
      image: amazon/aws-cli:latest
      command: ['sleep', '36000']
  restartPolicy: Never
  serviceAccountName: my-sa

Expected

Dotnet-monitor, thru AWSDK should be able to retrieve the mounted token.
Supposedly it reads the following env vars:

• AWS_REGION
• AWS_ROLE_ARN
• AWS_WEB_IDENTITY_TOKEN_FILE

Those env vars are populated automatically when adding a IRSA SA, BUT when adding a SA thru EKS pod identity, others are created (it would be great to have support for those as well). None of them seems to work. Even if you manually populate those env vars.

Configuration

• Is this related to a specific tool? Egress Providers of Dotnet monitor
• What OS and version, and what distro if applicable? The container used is the official one --> mcr.microsoft.com/dotnet/monitor:8
• What is the architecture (x64, x86, ARM, ARM64)? x64
• Are you running in any particular type of environment? (e.g. Containers, a cloud scenario, app you are trying to target is a different user) dotnet monitor container deployed in an EKS cluster

[github-actions]

Welcome to dotnet-monitor!

Thanks for creating your first issue; let us know what you think of dotnet-monitor by filling out our survey.

[miguepintor]

BTW, I saw this pr --> #6626

Definitely either is not included in the latest mcr.microsoft.com/dotnet/monitor:8 or the issue still persist

[miguepintor]

For those affected, I was able to "workaround" this issue by building my own monitor container using the 9.0 monitor base.
e.g. https://github.com/dotnet/dotnet-docker/blob/main/src/monitor/9.0/azurelinux-distroless/amd64/Dockerfile

[jander-msft]

BTW, I saw this pr --> #6626

Definitely either is not included in the latest mcr.microsoft.com/dotnet/monitor:8 or the issue still persist

The PR to support this is not part of .NET Monitor 8.0. The 8.0 version of .NET Monitor was released in November 2023 and the PR was complete in May 2024. Coupled with the fact that we largely observe SemVer 2 versioning scheme for .NET Monitor, this enhancement will not ever be included in an 8.0 update. You'll have to wait for either an 8.1 version (which we have no definitive plans for a release at this time) or use the 9.0 previews.

For those affected, I was able to "workaround" this issue by building my own monitor container using the 9.0 monitor base.
e.g. https://github.com/dotnet/dotnet-docker/blob/main/src/monitor/9.0/azurelinux-distroless/amd64/Dockerfile

I would not recommend building your .NET Monitor image unless you need to make custom alterations to .NET Monitor itself. While building it may not be difficult, we do that already for you and service the image on a regular basis. The 9.0 images (currently in preview but expected to GA later this year) already include the mentioned PR for several previews at this point.

You can get the preview version of 9.0 using mcr.microsoft.com/dotnet/monitor:9-preview. As soon as .NET Monitor 9.0 RC 1 is released, that tag will no longer be supported; you can use the release version tag e.g. mcr.microsoft.com/dotnet/monitor:9 instead, which will be supported for the lifetime of .NET Monitor 9.

[jander-msft]

I will update the document to be clear about which versions include this new capability.

[miguepintor]

@jander-msft thanks a lot for clarifying the releases schema. It is clear now.

I will then definitely use mcr.microsoft.com/dotnet/monitor:9-preview.

Thanks for your quick support.