Skip to content

WMCore services upgrade in kubernetes

Jump to bottom
Valentin Kuznetsov edited this page Oct 23, 2023 · 8 revisions

This wiki is meant to provide you with:

  1. a checklist for upgrading WMCore services in CMSWEB testbed and/or production;
  2. a simple step-by-step procedure.

For further material, please visit the other WMCore wiki pages:

or the detailed instructions provided by the CMSWEB team:

Checklist

NOTE: per agreement with CMSWEB group we should deploy on production k8s cluster only stable images. At the moment our CI/CD pipeline does not produce stable images, see #11762, therefore a manual step is required to create such image

When deploying WMCore services in the CMSWEB pre-production and/or production clusters, please make sure the following steps are performed:

  1. Ensure that all the relevant docker images have been successfully created and uploaded to the CERN registry. Also make sure the required docker tag is available.
  2. Check whether all the relevant services configuration have been merged in the services_config repository, under the correct branch.
  3. Log in to lxplus8, set the kubernetes token and load the required kubernetes environment (prod, preprod, testX).
  4. Check whether you are pointing to the correct cluster environment with kubectl config get-clusters | grep -v NAME
  5. Clone the services_config repository and checkout the required branch (prod, preprod, test).
  6. Clone the CMSKubernetes repository.
  7. Define the list of services to be upgraded. This deployment uses the variable named serviceList for that.
  8. IF required, update the services secrets with the script ./scripts/deploy-secrets.sh.
  9. Finally, update the WMCore services - under the dmwm namespace - with the script ./scripts/deploy-srv.sh.
  10. IF couchapp changes are required, you need to log in to each CouchDB VM and run the command below (update the WMCore tag name):
sudo -H -u _couchdb bashs -l -c '/data/srv/current/config/couchdb/manage updatecouchapps 2.1.4.pre6 "I did read documentation"'

The next section assumes that you are familiar with the environment requirements (token/cluster setup, configuration repository, kubernetes repository).

List of WMCore services

Here is an exhaustive list of all the current WMCore central services that we deploy in CMSWEB kubernetes clusters, which you should set in your terminal for the subsequent steps in this wiki. They are:

$ SERVICE_LIST="reqmgr2 \
                reqmgr2-tasks \
                reqmgr2ms-monitor \
                reqmgr2ms-transferor \
                reqmgr2ms-rulecleaner \
                reqmgr2ms-output \
                reqmgr2ms-pileup \
                reqmgr2ms-pileup-tasks \
                reqmgr2ms-unmerged-t1 \
                reqmgr2ms-unmerged-t2t3us \
                reqmgr2ms-unmerged-t2t3 \
                workqueue \
                reqmon \
                t0_reqmon \
                t0_reqmon-tasks \
                reqmon-tasks"

Deploying WMCore secrets

This step is only required when there are changes to the services configuration (made to the services_config repository, in the relevant branch). Once you have defined the list of services to deploy - called as SERVICE_LIST from the step above - and that you have cloned and checked out the correct services_config branch, you can deploy their secrets (service configuration and database credentials) with the following command:

WORKDIR=`pwd`
CFGBRANCH=preprod

git clone https://:@gitlab.cern.ch:8443/cmsweb-k8s/services_config.git  $WORKDIR/services_config --branch $CFGBRANCH
git clone https://github.com/dmwm/CMSKubernetes.git $WORKDIR/CMSKubernetes --branch master

cd $WORKDIR/CMSKubernetes/kubernetes/cmsweb

for service in $SERVICE_LIST
do
  echo "Deploying secrets for $service"
  ./scripts/deploy-secrets.sh dmwm $service $WORKDIR/services_config
  echo ""
done

Note that Kubernetes detects configuration changes. In case there are no new changes with relation to what has already been deployed, nothing will change and the AGE of your secrets will remain the same.

Deploying WMCore services

Now with the relevant WMCore service list defined (as SERVICE_LIST) and the proper secrets in place, you can deploy all of the WMCore services with the following command; WAIT, before that, make sure you define the correct WMCore tag below TAG:

TAG=X.Y.ZrcX
for service in $SERVICE_LIST
do
  echo "Deploying service $service with tag $TAG"
  ./scripts/deploy-srv.sh $service $TAG
  echo ""
done

This deploy-srv.sh script will automatically delete any outdated service POD and deploy the new tag for your service(s).

Lastly, you can check the status of these services and their PODs with the following command:

kubectl get pods -n dmwm

note that it takes quite a few seconds to properly terminate old PODs and load in new containers.

Clone this wiki locally