Add a default group for all OIDC users

[evandam]

Problem

When using some OIDC providers (such as Google) with Dex or something similar, it can be tricky to configure mapping groups. For many users, I'm sure granting all OIDC users one "hard-coded" group would suffice and we can create RoleBindings on that group.

Currently, if I can't map my OIDC groups, the only solution seems to be to create RoleBindings for every possible user ([email protected], [email protected], [email protected], etc.).

Solution

Add support something like oidcDefaultGroup=read-only in the oidc-auth secret. This should get passed when making impersonated calls.

Additional context

I may be looking at this wrong and there's a better way to do this (maybe on the Dex side?). Happy to hear alternatives or workaround! 🙌

[bigkevmcd]

How are you configuring your Kubernetes to handle this?

We have kept to the same nomenclature and functionality as described here https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

But that's no reason we couldn't do something like this, but one of the design goals was that debugging auth should be do able using kubectl auth.

So, can you say a bit more about how you're configuring your K8s api-server here?

[evandam]

Hi @bigkevmcd, thanks for the response - happy to elaborate more.

First, I'm using a Dex connector like this (with correct env vars passed). No groups are passed for my account:

connectors:
  - type: google
    id: google
    name: Google
    config:
      clientID: '{{ getenv "GOOGLE_CLIENT_ID" }}'
      clientSecret: '{{ getenv "GOOGLE_CLIENT_SECRET" }}'
      redirectURI: '{{ getenv "DEX_ISSUER_URL" }}/callback'

I'm also using oidUsernamePrefix and oidcGroupsPrefix, but I don't think it matters much either way - just so it's clear where the users/groups are coming from:

oidcUsernamePrefix: weave-gitops-
oidcGroupsPrefix: weave-gitops-

For the sake of simplicity, I'm then trying to give any user logged in through OIDC the cluster-admin ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: weave-gitops-oidc-cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  # NOTE: I'm able to log in and access everything, but painful doing it one user at a time.
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: [email protected]
  # TODO: How can I apply a ClusterRoleBinding to all users without a group to reference?
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: weave-gitops-<SOMETHING>

Hopefully it makes more sense! I feel like I'm probably missing something obvious but I'm not sure 😅

It would still be possible to use kubectl auth can-i ..., just passing a known group instead of what the user's upstream group memberships are.

[bigkevmcd]

Ahh...right, that is kinda what I expected, and yes, I can see that it's annoying.

I was wondering if you'd avoided the kubectl auth can-i ... bit because for most folks, that would be either annoying or confusing :-)

How do you actually deal with this at the kubectl level? are you configuring the client to send a special group in some way?

Where possible, I want this to fit within existing models, so I'm curious about how you configure your kubectl tooling? Am I missing a way to hard-code a group?

It wouldn't be particularly hard for us to pass through a configured group-override mechanism, and I'd definitely be open to a PR for this.

Alternatively, given that all users have the same permissions, we released a "no-auth" mechanism, in which case you can configure a fixed user that is impersonated, that might be interesting.

[evandam]

Ah, sure so we're an EKS shop, so auth is all done through AWS IAM for any kubectl commands run by hand. We have some separate Roles and ClusterRoles to lock down access that developers have (prevent tinkering in namespaces they shouldn't access, etc.).

The tricky thing maybe is that the OIDC users don't really map to our IAM users - I'm pretty much considering them totally different entities. None of this "hardcoding groups" comes into play with typical kubectl usage - it's just a possible workaround I see for the OIDC users.

I think it would be nice to still leverage the OIDC auth, so they at least need to be able to log in to access it, but disabling auth altogether doesn't sound like a terrible option either 😅

[evandam]

Hey @bigkevmcd - wondering if there are any thoughts/updates on this? Thanks!

[bigkevmcd]

Hi @evandam This isn't something we're going to get to any time soon, as I indicated above, I'd be happy to look at PRs, but this does feel like a pretty unusual option.

Generally folks want "integrated" auth, rather than separate auth for audit reasons etc, but as you say:

The tricky thing maybe is that the OIDC users don't really map to our IAM users - I'm pretty much considering them totally different entities.

This is definitely an unusual case.

You might consider an authenticating proxy in front of a the "no-auth" mechanism.

https://aws.amazon.com/blogs/containers/authenticate-to-amazon-eks-using-google-workspace/ might also be interesting.

[evandam]

I feel like it's not that unusual though 😅 for example an EKS cluster typically maps IAM roles to k8s groups in the aws-auth ConfigMap, not "native OIDC" at least as far as I know.

Sorry but can you point me in the direction for the "no-auth" mechanism though? I'm struggling to find it in docs. I think that plus a proxy in front would work though.

[bigkevmcd]

Hi @evandam This is the doc for "no-auth" https://docs.gitops.weave.works/docs/guides/anonymous-access/index.html

Sorry, this should be linked from the guides, and I'll open a PR to address this.

I will dig into this a bit more but I have a pile of things, can you link to more on the aws-auth bits and I can see if there's anything we can do.

[evandam]

Ah, thank you! This looks really helpful.

I think this should be a decent cherry-picked doc for the aws-auth CoonfigMap - https://aws.github.io/aws-eks-best-practices/security/docs/iam/#use-iam-roles-when-multiple-users-need-identical-access-to-the-cluster

The tldr is that the ConfigMap maps IAM roles to k8s users/groups, and you then authenticate with the cluster by running something like aws eks update-kubeconfig --name my-cluster as whichever IAM user/role you need.

[Cajga]

Note: EKS and EKS Anywhere can use OIDC for authentication (this is how we use these services with AzureAD) that can be used also with weave-gitops.

[evandam]

Sure I realize there are ways to do a lot of things, but I was just looking for a way to setup this dashboard without completely changing how user management is already done in my clusters as I don't think everyone out there is using OIDC for authentication 😅

No worries anyway, I think the link @bigkevmcd provided is the way forward for my use case at least. I appreciate the help and guidance on this!

[Cajga]

@evandam just took a look on the clusterrole that is in that doc... that will give read access to all secrets in the cluster...

[evandam]

Sure I'm taking things with a grain of salt 😂 I'm just hooking things up on a test cluster anyway for now just to work out all of the kinks exactly like this.

Good catch though! Is this something that should be updated in docs?

[bigkevmcd]

There are two issues:

1. We should document why those perms are right
2. We should document why you might not want them

At least in this case, this is to allow access to the Helm Release secrets, so we could document that if you don't use Helm, you don't need this.

[evandam]

Ah, makes sense. I removed secrets from the permissions anyway (such a big downside of Helm imho!). I suppose it could always be locked down more with individual RoleBindings in the namespaces you install HelmReleases in, but just a bit more work.

[bigkevmcd]

Yeah, it's hard to come up with RBAC that works for everybody.

[Cajga]

@evandam unfortunately, it is not that simple. We use flux in multi-tenancy mode and as such, users have only access to their own namespaces. It turned out that at the moment weave-gitops requires access to flux-system (ticket is here).

(I was following this ticket if it would solve our problem but the anonymous access is not an option).

@bigkevmcd, I believe, that weave-gitops should only require as much access as the users have with flux cli and ignore if it cannot fetch a resource... :)

[bigkevmcd]

@Cajga Broadly, that's the case, this is for the "anonymous" access which requires a single service-account.

If you want to restrict access, use the authenticated version, and it will use normal K8s RBAC for the authz.

[Cajga]

@bigkevmcd yeah but the problem is that weave-gitops still need much more than flux cli. Let's say that you are a user (authenticated through OIDC) with RBAC access to only a single namespace. You could use flux cli (and/or kubectl) to list/delete/reconcile your (flux) resources but if you login to weave-gitops, the dashboard is empty (as it requires access to flux-system namespace for some reason that the user does not have).

Anyway, this is off-topic for this ticket... See details in the other ticket...