[k8s] Robust service account and namespace support

[romilbhardwaj]

This PR has a few updates for k8s SA and namespace support:

Code:

1. Moves the skypilot-system namespace creation to happen only if the default SERVICE_ACCOUNT remote_identity is used. This is to reduce the scope of permissions required if a SA already exists.
2. Update namespace fetching logic to also account for the case when the cluster is running in a in-cluster auth config but in a namespace different than 'default'. Previously we had been testing on default namespace so this bug was not caught.

Docs:

1. Updated the minimum permissions for Kubernetes docs and confirms that the permissions listed in the docs work.
2. Added permissions required if storage mounting is used
3. Add comments if the user needs to specify their own namespace.

Ran the following tests for two scenarios on a GKE cluster:

1. User does not have a special service account and uses the default namespace. This is a base case to verify existing functionality does not break.
2. User creates a service account in a special myns namespace following the create-sky-sa.yaml example in the docs, creates a kubeconfig that uses this namespace and service account to authenticate then uses this kubeconfig to run SkyPilot in the myns namespace with remote_identity: sky-sa.

• Manual sky launch test both from laptop and from pod launched by SkyPilot
• Managed Job controller smoke test on Kubernetes using storage mounting: pytest -v tests/test_smoke.py::test_managed_jobs_storage --kubernetes

[Michaelvll]

Thanks for fixing this @romilbhardwaj! Left several comments : )

[Michaelvll]

Can we change the sky-sa name above the same as the default name in our codebase, so after creation, the user does not need to specify it in a ~/.sky/config.yaml (we can still mention that a user can change the name of the service account and specify the config yaml, but we should keep the service account creation with the same name as the hardcoded one)?

[romilbhardwaj]

Using the default service account skypilot-service-account requires additional permissions because our code is self-correcting - it automatically fixes any service account misconfigurations by creating/patching resources (e.g., when user is upgrading versions or has accidentally deleted k8s RBAC resources created by SkyPilot).

1. Our code will inspect and create/update the necessary roles and rolebindings. This requires additional "create", "patch" permissions on "clusterroles", "clusterrolebindings".
2. Our code will inspect and create the skypilot-system namespace if it doesn't exist. This requires additional "list", "create" permissions on namespaces.

These extra permissions on "clusterroles", "clusterrolebindings", "namespaces" may be considered too permissive in some environments (e.g., shared clusters). Perhaps we should keep the permissions here limited to minimal permissions required?

[Michaelvll]

How about:

Suggested change

	* A `Kubeconfig <kubeconfig>`_ file containing access credentials and namespace to be used. Refer to :ref:`required permissions <cloud-permissions-kubernetes>` guide for details.
	* A `Kubeconfig <kubeconfig>`_ file containing access credentials and namespace to be used. To reduce the permissions for a user, check :ref:`required permissions <cloud-permissions-kubernetes>` for details.

[romilbhardwaj]

Thanks! Updated. I'm also reworking this page in a separate PR.

[romilbhardwaj]

Thanks @Michaelvll. Addressed comments and updated generate_static_kubeconfig.sh to be a lot more friendlier to use.

[Michaelvll]

Thanks for fixing those permissions @romilbhardwaj! This is very important for our k8s users. : )

[Michaelvll]

Should we include the usage of this usage in the doc as well? So as to make it easier for the user to create the service account?

The doc for minimal permission above could serve as a deep dive into what this shell script do.

[romilbhardwaj]

absolutely - I'm working on it in a separate for revamping k8s cluster admin docs :)