Skip to content

Enabling CVMFS at coffea casa AF

Jump to bottom
Carl Lundstedt edited this page Jan 11, 2022 · 6 revisions

Enabling CVMFS at coffea-casa AF

A common request by users was to have a CVMFS repo mounted and readable from their notebook and workers. The Tier-2 condor workers at Nebraska bind mount CVMFS to the job slot and have CVMFS available out-of-the-box, what is needed is a procedure to attach CVMFS to the notebook and sidecar in the analysis instance without using escalating privileges.

We used the OSG's method of mounting CVMFS in a kubernetes environment K8S-CVMFS (https://github.com/opensciencegrid/osg-k8s-cvmfs). These charts have two purposes. First they create a daemonset of CVMFS daemons running on ALL the nodes of the K8s cluster to ensure that persistent volumes are available regardless of where K8s deploys the client container and a squid server in the K8s instance to reduce redundant traffic. Second, the charts provide a way of creating PVs for the K8s client container to mount (required for each namespace wishing to mount CVMFS).

Here we'll cover how to start the daemonset by hand and change the CMS-AF charts to allow for the notebook and sidecar to mount the CVMFS image. Automation of the CVMFS stack is left as an exercise for the administrators of the K8s site.

============

  1. Get the osg-k8s-cvmfs charts

git clone https://github.com/opensciencegrid/osg-k8s-cvmfs.git

  1. The current version of the k8s-cvmfs code has a hardcoded machine for the squid server. Likely this can be changed in a kustomization file but we changed that machine name by hand to a squid appropriate server in our k8s stack:

In osg-k8s-cvmfs/cvmfs/cvmfs-daemonset change: .spec.template.spec.nodeSelector.kubernetes.io/hostname:

This was all we needed to do to deploy the daemonset into Nebraska's AF instance. (Note: the daemonset runs in a top level namespace and is privileged but opaque to users.)

  1. deploy the daemonset into k8s

kubectl create namespace cvmfs

kubectl apply -k cvmfs-daemonset -n cvmfs

Check to make sure the containers spin up and that the squid server is created.

  1. Create the needed PVCs in the desired casa instance for the instance to mount. For us this required two steps, first the cms.cern.ch endpoint was not among the provided, stock pvc files. (But did otherwise appear to be configured, so your experience with a non-provided cvmfs endpoint may be more complicated). Second the namespace specific instance needs created.

If needed, construct the cvmfs-pv-.yaml file in osg-k8s-cvmfs/cvmfs-pvcs directory. We copied the -oasis .yaml file and replaced the oasis .spec.local.path.entry with /var/lib/cvmfs-k8s/cms.cern.ch es and changed all cvmfs-oasis naming entries to cvmfs-cms. This new file was named cvmfs-pv-cms.yaml

Include this yaml file into kustomization.yaml so that it is picked up on deploy.

To create the namespace specific instance (ie the namespace where the casa deploy lives) you need to create a directory in the top level of the osg-k8s-cvmfs charts. For our first attempt we wished to add CVMFS to the Casa deploy running in namespace cmsaf-dev.

mkdir ie.

mkdir cmsaf-dev

In that directory create a kustomization.yaml that points to the cvmfs-pvcs directory:

kustomization.yaml:

namespace: cmsaf-dev
namePrefix: cmsaf-dev-
bases:
  - ../cvmfs-pvcs

An experienced admin may be able to tailor this file/directory to not include unneeded CVMFS mounts but that task is again left to the reader.

Deploy the PVCS into your namepace. (in the top directory)

kubectl apply -k cmsaf-dev

There should now be persistent volume claims in that namespace ready to be mounted from the containers.

Mounting the CVMFS PVCs in the charts:

Our initial, crude (we hope to change this in the future, but this works) way of mounting the PVCs was to added the volume and volume mounts to the jhub helm file for that instance.

.spec.values.singleuser.storage looks like:

extraVolumes:
  - name: cmsaf-dev-cvmfs-cms
    persistentVolumeClaim:
      claimName: cmsaf-dev-cvmfs-cms
      readOnly: true
extraVolumeMounts:
  - mountPath: /cvmfs/cms.cern.ch
    name: cmsaf-dev-cvmfs-cms
    readOnly: true

and .spec.values.singleuser.extraContainers[0].volumeMounts has the entry:

  - mountPath: /cvmfs/cms.cern.ch
    name: cmsaf-dev-cvmfs-cms
    readOnly: true

We found the readOnly flag being set to true in both was crucial.

With these new charts installed in the K8s instance the user's environment should have cms.cern.ch mounted in /cvfms.

##NOTE: In this method you'll have to duplicate the mounts for each CVMFS endpoint you wish mounted.

Clone this wiki locally