qa/cephfs: override testing kernel with -k option

Normally, the fs suite overrides the kernel branch whenever the kernel client
is used, according to the matrix of configs. This prevents easily testing a new
kernel with the -k option to teuthology-suite. So, using the base config passed
to the script, detect if an alternate testing kernel is desired and use that
instead.

The scheduler still needs to do some work:

    $ teuthology-suite ... --verbose -k wip-pdonnell-i66704 --filter k-testing
    ...
    2024-10-18 00:29:43,875.875 DEBUG:teuthology.suite.merge:base kernel {'branch': 'wip-pdonnell-i66704', 'kdb': 1, 'sha1': '745cacd8f31e50d7f3b6039bbd8c9a8dfc07bf03', 'flavor': 'default'}
    2024-10-18 00:29:43,875.875 DEBUG:teuthology.suite.merge:overriding testing kernel with {'branch': 'wip-pdonnell-i66704', 'kdb': 1, 'sha1': '745cacd8f31e50d7f3b6039bbd8c9a8dfc07bf03', 'flavor': 'default'}

vs.

    $ teuthology-suite ... --verbose --filter k-testing
    ...
    2024-10-17 20:04:52,265.265 DEBUG:teuthology.suite.merge:base kernel {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}

Which will only select jobs using the "testing" kernel.

See-also: ceph/teuthology#2008
Fixes: https://tracker.ceph.com/issues/68603
Signed-off-by: Patrick Donnelly <[email protected]>

qa/cephfs/begin/3-kernel.yaml

@@ -0,0 +1,17 @@
+# When the --kernel option is given to teuthology-suite, the "default" kernel
+# is set for all nodes. We don't generally want to use a custom kernel for all
+# tests, so unset it. The k-testing.yaml will set it, if given, for only the
+# client nodes.
+#
+# Allow overriding this by using a branch ending in "-all".
+
+teuthology:
+  postmerge:
+    - |
+      if not yaml.kernel.branch:find "-all$" then
+        log.debug("removing default kernel specification: %s", yaml.kernel)
+        py_attrgetter(yaml.kernel).pop('branch', nil)
+        py_attrgetter(yaml.kernel).pop('flavor', nil)
+        py_attrgetter(yaml.kernel).pop('kdb', nil)
+        py_attrgetter(yaml.kernel).pop('sha1', nil)
+      end

qa/cephfs/mount/kclient/overrides/distro/testing/k-testing.yaml

@@ -1,3 +1,12 @@
+teuthology:
+  premerge: |
+    log.debug("base kernel %s", base_config.kernel)
+    local kernel = base_config.kernel
+    if kernel.branch ~= "distro" then
+      log.debug("overriding testing kernel with %s", kernel)
+      yaml_fragment.kernel.client = kernel
+    end
+
 kernel:
   client:
     branch: testing