ClusterBuilder EOF error when pushing the builder image

[semmet95]

Hey everyone.
I've been trying to use IRSA to authenticate a ClusterBuilder to push builder images to an ECR repo and I keep running into the following error:

Post "https://<ecr-registry>/v2/kpack-builders/blobs/uploads/?from=paketobuildpacks%2Fbuild&mount=sha256%3A04519a37e0a828er368530248f886b2f85d7d4713e5a8a0ea1b419ter47663df&origin=index.docker.io": EOF

I couldn't find more details in kpack-controller logs either ☹️

I even tried using docker-registry type secret created as follows:

kubectl create secret docker-registry regcred --docker-server=<ecr-registry> --docker-username=AWS --docker-password=$(aws ecr get-login-password)

and mounted it on the serviceaccount but I still get the same error.

I have verified that the IAM role being used has all the necessary permissions and the target repo exists.
Has anyone else run into this same issue?
Any help would be appreciated 🙏🏻

[tomkennedy513]

Can you share how you are setting up the IAM roles to be used in kpack

[semmet95]

I'm following the instructions from AWS docs. I already had an IAM role with ECR permissions. I added the cluster's OIDC to its trusted entities and also made sure to explicitly mention the exact service account that would assume this role. Then I added the annotation eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/my-role to the serviceaccount to let it assume the IAM role.

I have one question regarding the permissions. Do I need to add ecr:GetAuthorizationToken to the roles's actions? Currently I have added ecr:Put* for pushing permissions.

[semmet95]

Update: I found that kpack uses the corresponding worker node's role instead of the role associated to the attached serviceaccount via IRSA, to push the ClusterBuilder image.
Is this behaviour expected?

[semmet95]

To anyone stumbling upon this thread, it seems kpack doesn't support IRSA for ClusterBuilder, the image push permissions are determined by permissions the node kpack-controller is running on has, and not the role you attach to the serviceaccount via IRSA.

[alexbarbato]

FWIW - I think that this works now and uses the SA for kpack-controller

[chenbh]

I think this should've always worked (at least I'm looking at some of our internal EKS tests and they haven't changed in at least a year). The key is that the role must be attached to the kpack-controller service account in the kpack namespace.

The service accounts specified in the [Cluster]Builder will only be used for the secrets/dockerconfig-based auth, there's no mechanism to retrieve OIDC tokens from it