OpenQA Tests for Qubes OS

To install, clone this repo to /var/lib/openqa/tests/qubesos

Example command for posting new ISO for test:

    openqa-client --host https://openqa.qubes-os.org isos post iso=Qubes-DVD-x86_64-20180224.iso DISTRI=qubesos VERSION=4.0 FLAVOR=install-iso ARCH=x86_64 BUILD=20180224 UEFI_DIRECT=1

Example command for starting tests of updates in current-testing:

    openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.0 FLAVOR=update ARCH=x86_64 BUILD=$(date +%Y%m%d%H) UEFI_DIRECT=1

Updates test on R4.1:

    openqa-cli api isos -X post VERSION=4.1 BUILD=$(date +%Y%m%d%H) DISTRI=qubesos ARCH=x86_64 FLAVOR=update UPDATE=1 UPDATE_TEMPLATES=fedora-34-xfce GUIVM=1

Example command for starting tests for a template in templates-*-testing:

    openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.0 FLAVOR=templates ARCH=x86_64 BUILD=2020022706-4.0-fedora-31 UPDATE_TEMPLATES='fedora-31 fedora-31-minimal' TEST_TEMPLATES='fedora-31 fedora-31-minimal'

Job templates:

This is now edited via web UI. The templates.json file is git is a backup.
Dump the templates: openqa-dump-templates --json | jq -S > templates.json
Load the templates: openqa-load-templates templates.json

Variables:

 - `DISTRI` - always `qubesos`
 - `VERSION` - system version, currently not interpreted (but displayed in UI)
 - `FLAVOR` - `install-iso` for installation tests, `update` for updates system tests
 - `ARCH` - always `x86_64`
 - `BUILD` - build number, not interpreted (but displayed in UI)
 - `UEFI_DIRECT` - UEFI boots directly to Xen, no separate bootloader

Variables used in tests:
 - `INSTALL_TEMPLATES` - space separated list of template groups (`debian`, `whonix`), or `all`; can include also specific template versions (`debian-11`) - they they will be installed/reinstalled from online repo after the initial install process
 - `USBVM` - USB VM configuration: `none` (assert that already disabled by default), `disable` (disable explicitly), `sys-usb` (default), `sys-net` (combined with Net VM)
 - `KEYBOARD_LAYOUT` - install with non-default keyboard layout; currently only `us-colemak` value is supported
 - `LOCALE` - install with non-default locale; currently only `en_DK.utf8` value is supported
 - `SYSTEM_TESTS` - run system tests from those (space separated) modules, each module can be suffixed with ":TIMEOUT" (in seconds); tests are run using nose2
 - `VERBOSE_LIBVIRT` - increase libvirt log level, including libxl logging; slows down the system and may lead to huge logs
 - `REPO_1` - repository name to install updates (dom0 and vm) from - in addition to standard repos
 - `KEY_1` - repository key short fingerprint to use for REPO_1
 - `SALT_SYSTEM_TESTS` - run salt formula preparing for system tests (mostly install extra packages)
 - `UPDATE_TEMPLATES` - update listed template packages as a whole (space separated list of templates to update)
 - `TEST_TEMPLATES` - limit tests to listed templates only (space separated list of templates to test)
 - `DEFAULT_TEMPLATE` - choose default template - on install jobs it's chosen in initial-setup, on update jobs all qubes that used default template are switched to this one
 - `UPDATE` - install updates in dom0 and all templates
 - `QUBES_TEST_EXTRA_INCLUDE`, `QUBES_TEST_EXTRA_EXCLUDE` - passed directly to the test environment
 - `TEST_GUI_INTERACTIVE` - simple GUI interactive tests (start application from menu etc)
 - `TEST_WINDOWS_GUI_INTERACTIVE` - simple GUI interactive tests for Windows with QWT installed
 - `WINDOWS_VERSION` - create Windows VM of specific version and install QWT inside (argument as for qvm-create-windows-qube)
 - `QWT_DOWNLOAD` - download specific QWT build (can be URL to iso or rpm file); otherwise qubes-windows-tools dom0 packages is installed from standard repositories
 - `GUIVM` - create GUI VM (sys-gui)
 - `GUIVM_VNC` - create GUI VM (sys-gui-vnc)
 - `GUIVM_GPU` - create GUI VM (sys-gui-gpu)
 - `KEEP_SCREENLOCKER` - do not disable xscreensaver (or whatever screenlocker is there)
 - `PARTITIONING` - partition layout ('standard', 'xfs', 'btrfs', 'default', 'unencrypted'); on install - set via installer options, later - created manually on sdb, and all templates are migrated there
 - `HEADS` - when set to `1`, the test will handle Heads boot menu. Currently relevant only with `BACKEND=generalhw`. It's supposed to be sed in "machine" definition.
 - `RELEASE_UPGRADE_REPO` - relevant for release-upgrade flavor, download scripts from specific URL, like: `https://raw.githubusercontent.com/marmarek/qubes-dist-upgrade/<branch-name>`
 - `WHONIXCHECK` - run whonixcheck / systemcheck on all Whonix-related VMs (including templates)
 - `WHONIX_INTERACTIVE` - run interactive tests on Whonix-related VMs (mostly anon-whonix)
 - `KERNEL_VERSION` - which kernel flavor should be used - currently supported value is 'latest', which will switch to kernel-latest and kernel-latest-qubes-vm during update
 - `PIPEWIRE` - when set to `1`, VMs are set to use pipewire instead of
   pulseaudio, including switching native pulseaudio to pipewire-pulse.
 - `SUSPEND_MODE` - what suspend mode should be used; default is `S3`, set to `S0ix` to use S0ix suspend
 - `INSTALL_OEM` - do fully automated OEM installation
 - `INSTALL_OEM_STARTUP` - start OEM installation, but still expect interactive prompts during installation
 - `HID` - for generalhw tests, tell it whether `USB` hid is used (default) or `PS2` one; in the latter case, sys-usb is not supposed to have input-proxy allowed by default

SecureDrop variables used in tests:
 - `SECUREDROP_INSTALL` - when set to `1`, SecureDrop Workstation is installed
 - `SECUREDROP_TEST` - when set with a value, SecureDrop tests are run on top of an existing installation