[k8s] Add docs on minimal permissions required on Kubernetes cluster

[romilbhardwaj]

Our docs should list the minimal Kubernetes RBAC Role and ClusterRole required for the user configured in the current kubecontext.

In addition, we should include a minimal example yamls for users to easily setup these roles. Here's some off the top of my head, we should verify if these work and possibly consolidate into fewer roles:

# r/w permissions on pods
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list", "create", "update", "delete"]

# Services
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: <your-namespace>
  name: service-manager-role
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "watch", "list", "create", "update", "delete"]


# we'd also need ability to query nodes for resource status
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "watch", "list"]

# Create ssh key secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: <your-namespace>
  name: secret-manager-role
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create"]

[github-actions]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[romilbhardwaj]

Being added in #3109.

[romilbhardwaj]

Added in #3377.