[EKS] [request]: Support custom TargetGroupBinding resources in auto-mode

[aslatter]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
I would like to use a pre-existing NLB with a k8s service running in an EKS-cluster with auto-mode enabled.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Because I would like my front-end Network Load Balancer to have a lifetime separate from that of the EKS control-plane, I currently provision the NLB and Target Group myself, and then directly create a TargetGroupBinding resource to register appropriate services with the NLB. This is a documented feature of the aws-load-balancer-controler here: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.11/guide/targetgroupbinding/spec/

I would like to continue this pattern with an EKS auto-mode cluster.

Following the public documentation listed above does not work because the k8s resource-group elbv2.k8s.aws/v1beta1 is not available.

There is a similar resource in eks.amazonaws.com/v1 which appears to be compatible at first, but I haven't found any documentation that what I'm doing is supported.

If I try to create a TGB in eks.amazonaws.com/v1 I get the following errors in CloudTrail for the RegisterTargets action:

User: arn:aws:sts::<my account id>:assumed-role/<my cluster role name>/aws-go-sdk-1735434588572399672 is not authorized to perform: elasticloadbalancing:RegisterTargets on resource: <my target group> because no session policy allows the elasticloadbalancing:RegisterTargets action

Presumably the role-session-policy is enforcing some constraint I don't know about (and even if I did know how to satisfy this constraint I would want some documentation or signal that this is supported).

Are you currently working around this issue?
How are you currently solving this problem? Avoiding auto-mode.

Additional context
Anything else we should know?

Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[Viktor-Osika]

I had the same issue recently with EKS Auto Mode.

After inspecting the cluster IAM Role AmazonEKSLoadBalancingPolicy I have found this:
{ "Effect": "Allow", "Action": [ "elasticloadbalancing:RegisterTargets" ], "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*", "Condition": { "StringEquals": { "aws:ResourceTag/eks:eks-cluster-name": "${aws:PrincipalTag/eks:eks-cluster-name}" } } },

Which explains the authorization issue - the TargetGroup has to have eks:eks-cluster-name tag with the cluster name.
To solve this, you either have to tag the TargetGroup or attach a new policy that will allow elasticloadbalancing:RegisterTargets for your TargetGroup. I have it working as expected now.

It really is a shame that there is no proper documentation for this, nor any AWS Load Balancer Controller logs / events available other than a CloudTrail.

[lucdew]

I confirm the workaround and was just about to write on it.
Tagging the target group with the tag eks:eks-cluster-name as key and the cluster name as value made it work.

In cloudtrail filtering on the sts:AssumeRole events you can see that an aws-go-sdk client assumes your cluster role with a scoped down policy to arn:aws:iam::aws:policy/AmazonEKSLoadBalancingPolicy and a policy (irrelevant here). And in that policy there is a context condition on the requestTag with key = eks:eks-cluster-name and value the cluster value for the action RegisterTargets

As a side note, I also add permission issues on a Service of type LoadBalancer. It creates dynamically a NLB properly but if you have a look at the events for the service there are error messages with failed removeTags operation. This time adding the permission in the cluster role solved the issue. Even though my cluster role had the permissions documented here https://docs.aws.amazon.com/eks/latest/userguide/auto-learn-iam.html#tag-prop

[aslatter]

The work-around works for my scenario. It would be helpful to know if this behavior is intentional, and if user-created TGBs is something supported in auto-mode.