Compare APS changes in cmsweb with cmsweb-prod

[amaltaro]

Impact of the bug
WM central services

Describe the bug
In the past week, we have found wrong APS rules for workqueue/workqueue_inbox, and there could be additional complex urls (couchdb ones) with problems as well.

So we should take advantage that cmsweb-prod is still running with Apache, and compare urls/redirects betweeen the 2 environment (which are actually sharing the same backend database).

How to reproduce it
None

Expected behavior
Behavior and outcome of cmsweb vs cmsweb-prod urls must be exact the same.
Further details in https://gitlab.cern.ch/cmsweb-k8s/services_config/-/issues/3

Additional context and error message
None

[vkuznet]

I decided to write comprehensive details about how to compare APS rules and make it easy for everyone to understand their flow. Maybe we should put it as wmcore document (feel free to vote for it and we can transfer docs below over there).

The APS rules are located in cmsweb-k8s/services_config gitlab repository. The APS configuration file can be configured to use either direct rules as list of dictionaries or use ingress files. Currently on cmsweb we adopted use of ingress files where each files contains rules for a given service, e.g. ingress/ms-output.json contains all rules for MSOutput data-services, etc. Each individual rule is a dictionary of the following form (here we use tier0_wmstats end-point as an example):

    {
      "path": "/couchdb/tier0_wmstats/_design/WMStats/_view/cooledoffRequests",
      "service_url": "http://xxx.cern.ch:5984",
      "old_path": "/couchdb/tier0_wmstats/_design/WMStats/_view/cooledoffRequests",
      "new_path": "/tier0_wmstats/_design/WMStatsErl4/_view/cooledoffRequests"
    }

where:

• path represents API end-point of the service
• service_url points to service backend, it may include port if necessary
• old_path represents URI path which will be replaced by new_path In this case, we replace http://frontend.../couchdb/tier0_wmstats/_design/WMStats/_view/cooledoffRequests with http://backend.../tier0_wmstats/_design/WMStatsErl4/_view/cooledoffRequests. The old_path to new_path substitution is useful in cases when frontend and backend routes are different.

For complete details of APS and its structure please refer to this document.

Please note, that APS executable can be used to dump rules in human readable form using the following command:

# download APS executable for your architecture
curl -ksLO https://github.com/dmwm/auth-proxy-server/releases/download/0.3.14/auth-proxy-tools_amd64.tar.gz

# untar the archite
tar xvfz  auth-proxy-tools_amd64.tar.gz

# and locate your executable
./auth-proxy-tools_amd64/auth-proxy-server -help

# adjust your APS config to properly list your ingress files and dump them on stdout
./auth-proxy-tools_amd64/auth-proxy-server -config=config.json -rules
^/renew                                                            => http://auth-proxy-server.auth.svc.cluster.local:443/
^/token                                                            => http://auth-proxy-server.auth.svc.cluster.local:443/
....

The printed rules are shown in exact order as they will be traversed by APS server, and order of rules is important.

Now, let's back to the issue in question.

Here is my findings about APS rules between cmsweb and cmsweb-prod:

• the cmsweb rules comes from aps-cmsweb branch
• the cmsweb-produ rules comes from aps-cmsweb-prod branch
• I extracted existing rules from cmsweb cluster as following:

# setup proper KUBECONFIG environment to point to cmsweb k8s cluster
export KUBECONFIG=/path.../private/cmsweb/k8s_admin_config/config.prod/config.cmsweb

# extract rules from cmsweb (you need CMSKubeternetes repo for that)
mkdir -p secrets/cmsweb/auth-proxy-server
./scripts/extract_secrets.sh auth auth-proxy-server-secrets secrets/cmsweb/auth-proxy-server

and, then I compared rules from aps-cmsweb-prod branch and extracted rules from cmsweb via the following command:

# cd to area where your aps-cmsweb-prod branch is
ls *.json | egrep -v "c.json|config.json" | awk '{print "echo "$1"; diff -u "$1" /path.../private/CMSKubernetes/kubernetes/cmsweb/secrets/cmsweb/auth-proxy-server/"$1""}' | /bin/sh

• I performed the following comparison
  • compared each individual ingress json file between aps-cmsweb-prod and extracted rules from cmsweb, results can be found here
  • dumped rules into human readable form and compared them between aps-cmsweb-prod and cmsweb
  • sorted rules of human readable form and performed their comparison, results are here

Observation

The difference between cmsweb and cmsweb-prod should be only visible for DBS server redirect rules since on cmsweb-prod we use dedicated DBS cluster. Therefore, all other rules should looks alike. But I found the following:

• rules for couch end-points are spread in different files, i.e. in cmsweb the couchdb.json contains entries for "^/couchdb/wmstats_logdb.*", "^/couchdb/workqueue_inbox.*" and "^/couchdb/workqueue.*" in addition to "^/couchdb.*" (which is consistent between the two). But these rules are placed into corresponding ingress files, i.e. "^/couchdb/wmstats_logdb/.*" is placed into ingress/wmstats_logdb.json in cmsweb-prod branch. This re-arrangement falls into question which ingress files should be used to hold the rules
• dbs rules are differ as expected
• there is a difference between rules for "^/dqm/dqm-square" end-point

You can see all differences in sorted_rules.diff.