AWS EKS 1.24 client is not respecting jenkins-master pod role

[bober2000]

Jenkins and plugins versions report

Environment

Jenkins: 2.375.2
OS: Linux - 5.4.226-129.415.amzn2.x86_64
---
ace-editor:1.1
allure-jenkins-plugin:2.30.3
antisamy-markup-formatter:155.v795fb_8702324
apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61
authentication-tokens:1.4
authorize-project:1.4.0
aws-credentials:191.vcb_f183ce58b_9
aws-java-sdk:1.12.397-362.v050e9394cf8e
aws-java-sdk-cloudformation:1.12.397-362.v050e9394cf8e
aws-java-sdk-codebuild:1.12.397-362.v050e9394cf8e
aws-java-sdk-ec2:1.12.397-362.v050e9394cf8e
aws-java-sdk-ecr:1.12.397-362.v050e9394cf8e
aws-java-sdk-ecs:1.12.397-362.v050e9394cf8e
aws-java-sdk-efs:1.12.397-362.v050e9394cf8e
aws-java-sdk-elasticbeanstalk:1.12.397-362.v050e9394cf8e
aws-java-sdk-iam:1.12.397-362.v050e9394cf8e
aws-java-sdk-logs:1.12.397-362.v050e9394cf8e
aws-java-sdk-minimal:1.12.397-362.v050e9394cf8e
aws-java-sdk-sns:1.12.397-362.v050e9394cf8e
aws-java-sdk-sqs:1.12.397-362.v050e9394cf8e
aws-java-sdk-ssm:1.12.397-362.v050e9394cf8e
aws-parameter-store:1.2.2
aws-secrets-manager-credentials-provider:1.202.ve0ec0c17611c
badge:1.9.1
basic-branch-build-strategies:1.3.2
bitbucket:1.1.30
blueocean:1.25.2
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.1
blueocean-commons:1.27.1
blueocean-config:1.27.1
blueocean-core-js:1.27.1
blueocean-dashboard:1.27.1
blueocean-display-url:2.4.1
blueocean-events:1.27.1
blueocean-git-pipeline:1.27.1
blueocean-github-pipeline:1.27.1
blueocean-i18n:1.27.1
blueocean-jwt:1.27.1
blueocean-personalization:1.27.1
blueocean-pipeline-api-impl:1.27.1
blueocean-pipeline-editor:1.27.1
blueocean-pipeline-scm-api:1.27.1
blueocean-rest:1.27.1
blueocean-rest-impl:1.27.1
blueocean-web:1.27.1
bootstrap5-api:5.2.1-3
bouncycastle-api:2.27
branch-api:2.1071.v1a_188a_562481
build-with-parameters:1.6
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
checks-api:1.8.1
cloudbees-bitbucket-branch-source:791.vb_eea_a_476405b
cloudbees-folder:6.800.v71307ca_b_986b
command-launcher:1.6
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-27.vb_fa_3896786a_7
conditional-buildstep:1.4.2
configuration-as-code:1569.vb_72405b_80249
credentials:1214.v1de940103927
credentials-binding:523.vd859a_4b_122e6
data-tables-api:1.12.1-4
display-url-api:2.3.7
durable-task:504.vb10d1ae5ba2f
echarts-api:5.4.0-1
favorite:2.4.1
font-awesome-api:6.2.0-3
generic-webhook-trigger:1.83
git:5.0.0
git-client:4.1.0
git-parameter:0.9.13
git-server:99.va_0826a_b_cdfa_d
github:1.36.1
github-api:1.303-400.v35c2d8258028
github-branch-source:1701.v00cc8184df93
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
htmlpublisher:1.31
http_request:1.12
instance-identity:142.v04572ca_5b_265
ionicons-api:31.v4757b_6987003
jackson2-api:2.14.2-319.v37853346a_229
jakarta-activation-api:2.0.1-2
jakarta-mail-api:2.0.1-2
javax-activation-api:1.2.0-5
javax-mail-api:1.6.2-5
jaxb:2.3.7-1
jdk-tool:1.5
jenkins-design-language:1.27.1
jjwt-api:0.11.5-77.v646c772fddb_0
job-dsl:1.78.1
jquery:1.12.4-1
jquery3-api:3.6.1-2
jsch:0.1.55.61.va_e9ee26616e7
junit:1166.va_436e268e972
kubernetes:3842.v7ff395ed0cf3
kubernetes-client-api:6.3.1-206.v76d3b_6b_14db_b
kubernetes-credentials:0.10.0
lockable-resources:1122.v14c3d52cb_1b_1
mailer:448.v5b_97805e3767
matrix-auth:2.6.8
matrix-project:785.v06b_7f47b_c631
mercurial:1260.vdfb_723cdcc81
metrics:4.2.13-420.vea_2f17932dd6
mina-sshd-api-common:2.9.2-50.va_0e1f42659a_a
mina-sshd-api-core:2.9.2-50.va_0e1f42659a_a
octopusdeploy:3.1.7
okhttp-api:4.9.3-108.v0feda04578cf
parameterized-scheduler:0.9.2
parameterized-trigger:2.43.1
pipeline-aws:1.43
pipeline-build-step:2.18
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:629.vb_5627b_ee2104
pipeline-input-step:466.v6d0a_5df34f81
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2118.v31fd5b_9944b_5
pipeline-model-definition:2.2118.v31fd5b_9944b_5
pipeline-model-extensions:2.2118.v31fd5b_9944b_5
pipeline-rest-api:2.30
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5
pipeline-stage-view:2.30
pipeline-utility-steps:2.10.0
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:2.20.0
popper2-api:2.11.6-2
prometheus:2.0.10
promoted-builds:3.11
pubsub-light:1.17
purge-job-history:1.6
role-strategy:3.2.0
run-condition:1.5
saml:2.0.9
scm-api:631.v9143df5b_e4a_a
script-security:1229.v4880b_b_e905a_6
slack:2.49
snakeyaml-api:1.33-90.v80dcb_3814d35
sse-gateway:1.26
ssh-agent:1.23
ssh-credentials:305.v8f4381501156
sshd:3.242.va_db_9da_b_26a_c3
stashNotifier:1.20
structs:324.va_f5d6774f3a_d
terraform:1.0.10
text-finder:1.17
token-macro:321.vd7cc1f2a_52c8
trilead-api:2.84.v72119de229b_7
uno-choice:2.5.7
variant:59.vf075fe829ccb
workflow-aggregator:2.6
workflow-api:1208.v0cc7c6e0da_9e
workflow-basic-steps:994.vd57e3ca_46d24
workflow-cps:3606.v0b_d8b_e512dcf
workflow-cps-global-lib:609.vd95673f149b_b
workflow-durable-task-step:1223.v7f1a_98a_8863e
workflow-job:1254.v3f64639b_11dd
workflow-multibranch:716.vc692a_e52371b_
workflow-scm-step:400.v6b_89a_1317c9a_
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c

What Operating System are you using (both controller, and any agents involved in the problem)?

[root@ip-10-128-1-192 /]# cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Reproduction steps

Have jenkins pod running with attached role
kubectl describe pod jenkins-0 -n jenkins | grep AWS_ROLE AWS_ROLE_ARN: arn:aws:iam::000000000:role/service/eks/eks-cluster-jenkins-master-role
But when I try to get access to Secrets Manager it is using node-group role
2023-02-01 20:50:31.880+0000 [id=330] WARNING i.j.p.c.s.AwsCredentialsProvider#getCredentials: Could not list credentials in Secrets Manager: message=[User: arn:aws:sts::000000000:assumed-role/eks-node-group-20230123082959723000000001/i-0259469284873172d is not authorized to perform: secretsmanager:ListSecrets because no identity-based policy allows the secretsmanager:ListSecrets action (Service: AWSSecretsManager; Status Code: 400; Error Code: AccessDeniedException; Request ID: f9393b18-b391-4ac7-bacd-8f9442fd7861; Proxy: null)]

Expected Results

Plugin is using jenkins-master role arn:aws:iam::000000000:role/service/eks/eks-cluster-jenkins-master-role

Actual Results

As node-group role don't have needed permissions plugin can't read secretes from Secrets Manager

Anything else?

No response

[bober2000]

Seems to be duplicate of #172 but that issue was closed and it seems not resolved

[chriskilding]

It looks like the role is injected from the AWS environment. I'm just wondering if somehow AWS might be changing the environment under the hood in some cases, which might change the role used.

If you wanted to debug this, you could potentially set the role to use explicitly in the plugin's configuration:

unclassified:
  awsCredentialsProvider:
    client:
      credentialsProvider:
        assumeRole:
          roleArn: "arn:aws:iam::111111111111:role/foo"
          roleSessionName: "jenkins"

This should hopefully keep it locked into the role you specify.

[bober2000]

@chriskilding I've tried that - but than I'm getting other error that arn:aws:sts::000000000:assumed-role/eks-node-group-20230123082959723000000001/i-0259469284873172d is not authorized to perform: assume role
I also tried to give needed permissions to node-group role - error message in logs are gone - but I cant see credentials in list

[chriskilding]

When this error occurs, how are you accessing the credentials? (It can make a difference where you do this from.)

E.g. are you viewing the credentials in the Web UI (i.e. the /jenkins/manage/credentials/ screen)? Or does this happen when you access a credential inside a job?

[bober2000]

@chriskilding I'm trying to access credentials in Web UI first.
But in general I want to use this plugin together with JCasc to create list of secrets and use them in Jobs