Support OpenShift

[ezbz]

Tasks

Preview Give feedback

1. Initial R&D for OpenShift support #1935
   kind/feature priority/important-soon triage/accepted +3
   
2. Perftune jobs should use dedicated serviceaccount #1975
   kind/feature priority/important-soon +2
   
3. Mount path /mnt/persistent-volumes is not canonical (contains a symlink). #1985
   kind/bug priority/important-soon +2
   
4. NodeConfig ignores /var which also break other folders when symlinked #2029
   kind/bug priority/important-soon +2
   
5. Grant scylla operator additional RBAC rules when running on OpenShift #1976
   kind/feature priority/important-soon triage/accepted +3
   
6. There is a cycle between helm chart CRD symlinks and the deploy manifests #2043
   kind/cleanup priority/important-soon +2
7. https://github.com/scylladb/scylla-operator-release/issues/210
8. E2E conformance suite needs to work with storage on EKS #1988
   kind/feature priority/important-soon triage/accepted +3
   
9. https://github.com/scylladb/scylla-operator-release/issues/189
10. https://github.com/scylladb/scylla-operator-ci-tools/issues/16
11. https://github.com/scylladb/scylla-operator-release/issues/371
12. E2E failures - NodeConfig Optimizations [It] should correctly project state for each scylla pod [Serial] #2310
    kind/failing-test priority/important-soon +2
    
13. E2E failures - Node Setup should propagate mount controller's degraded condition #2308
    kind/failing-test priority/important-soon +2
    

[cgruver]

I have an interim solution here, but it will need some work to conform to the higher security requirements of OpenShift.

1. The operator needs to run with anyuid privileges because it wants to bind to port 443.
2. Need to set scyllaclusters/finalizers with update privilege on the cluster member ROLE.
3. The cluster service account needs to run with the privileged SCC so that it can access SYS_NICE.

You can find a working example here: https://github.com/lab-monkeys/home-library-tutorial under the Scylla folder.

It still needs more work, so I'll update with progress.

[cgruver]

I have ScyllaDB clusters running in OpenShift 4.6.

The Pod security still needs some work so that the Pods don't have to run as privileged containers, but it's a start.

The working example is here:

https://github.com/lab-monkeys/home-library-tutorial/tree/main/Scylla

oc apply -f cert-manager.yaml

oc apply -f operator.yaml
oc apply -f cluster-cql.yaml
oc apply -f cluster-dynamo.yaml

I'm running on a bare-metal cluster, so eventually I should be able to tune it for all of the performance tweaks that ScyllaDB needs.

[tnozicka]

The operator needs to run with anyuid privileges because it wants to bind to port 443.

We shouldn't listen on 443, the service should handle the redirection and we should listen on unprivileged port. Feel free to file a separate issue there, I think we could fix that sooner than we support OCP.

[cortopy]

I don't use Openshift, but I have a hardened cluster with strict security policies. For those that are in the same situation, this was enough for me. There are two lines that are the key:

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: scylla
  labels:
    {{- include "scylla-extras.labels" . | nindent 4 }}
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: "docker/default,runtime/default"
    apparmor.security.beta.kubernetes.io/allowedProfileNames: "runtime/default"
    seccomp.security.alpha.kubernetes.io/defaultProfileName: "runtime/default"
    apparmor.security.beta.kubernetes.io/defaultProfileName: "runtime/default"
spec:
  allowPrivilegeEscalation: false
  allowedCapabilities:
    - SYS_NICE # Fix 1
  volumes:
    - "configMap"
    - "emptyDir"
    - "projected"
    - "secret"
    - "downwardAPI"
    - "persistentVolumeClaim"
  hostNetwork: false
  hostIPC: false
  hostPID: false
  runAsUser:
    rule: "RunAsAny" # Fix 2
  seLinux:
    rule: "RunAsAny"
  supplementalGroups:
    rule: "MustRunAs"
    ranges:
      - min: 1
        max: 65535
  fsGroup:
    rule: "MustRunAs"
    ranges:
      - min: 1

SYS_NICE (Fix 1) capability is understandable

However, I have to allow scylla pods to run as root (fix number 2), which feels like an overkill and a source of potential vulnerablities

[tnozicka]

I think the root privs are for tuning devices and setting sysctl which are done on scylla startup

[gautam-borkar]

I tried the step mentioned above and assigned privileged role for serviceaccount. But getting this error :-

Running the command gives below output :-

kubectl -n scylla get pods -l "app.kubernetes.io/name=scylla-incident-mgmt"
No resources found in scylla namespace.

[tnozicka]

Running the command gives below output :-

the label selector seems wrong - it should be "app.kubernetes.io/name=scylla", not "app.kubernetes.io/name=scylla-incident-mgmt"

Some actionable info would be say oc get events --sort-by=.metadata.creationTimestamp or the pod's yaml if it gets created or operator logs if there are failures.

But in this case, as it's a feature, this is primarily waiting for cycles from one of the team members to go and try when we decide to support the platform, I assume we'd be almost certain to hit all those issues as well.

[mykaul]

Depends on #713

[scylla-operator-bot]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out

/lifecycle stale

[tnozicka]

/remove-lifecycle stale
/triage accepted