[k8s] Fix mounting when launching from a service account

[romilbhardwaj]

Closes #3530.

Adds the appropriate permissions to create the skypilot-system namespace and create resources in it.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Smoke tests - enabled test_managed_jobs_storage for kubernetes pytest -v tests/test_smoke.py::test_managed_jobs_storage --kubernetes

[Michaelvll]

Thanks for the fix @romilbhardwaj! Mostly looks good to me : )

[Michaelvll]

Is it possible to check the config of the account first and only patch the service account if it does not have all the config?
Reason: previously, we only required read permission for the user's account, but now we require write permission as well. This is to make sure the following works: the admin create the service account for all users and create restrictive user accounts with only read permission (this is what we made sure for the other clouds). Wdyt?

[romilbhardwaj]

Good point! I've updated checks to patch only if the existing role does not match the required permissions.

[romilbhardwaj]

Thanks @Michaelvll! Running smoke tests now:

• Smoke tests - pytest tests/test_smoke.py --kubernetes --managed-jobs
• Smoke tests -pytest tests/test_smoke.py --kubernetes --serve

[romilbhardwaj]

Tests pass, merging now!