Test artifacts installation

[thrix]

Introduction

Due to the multihost changes, tmt will be taking over the responsibility of preparing the testing environment from Testing Farm.

For details about the test environment preparation in Testing Farm see the documentation.

This document is focusing on the artifact-installation phase.

Note that tmt already supports installation of certain artifacts via the install prepare plugin.

Use Cases

tmt

As a user, I want the ability to perform package installation tests during test execution. I expect the test repositories to be correctly configured to ensure accurate results. However, it is essential that packages are not pre-installed before the test runs to guarantee a clean testing environment.

As a user, I want to install a selected set of packages (e.g. just tmt, no other tmt subpackage), possibly gathered from the test.require keys.

As a user, I want to be sure that the correct rpm version from the tested artifact (from the requested koji/brew build, copr build) is installed.

As a user, I want all binary rpm packages from the requested artifact to be installed.

As a user, I want to exclude given packages from the installation (e.g. skip some weak dependencies).

As a user, I do want to install the tested rpms during test execution, I just want to have the rpms available in the test environment in a repository.

Testing Farm

As a user, I want to install rpms from a specific RPM repository.

As a user, I want to add a repository file to the testing environment.

As a user, I want to install Brew or Koji builds identified by the build task IDs.

As a user, I want to install Fedora copr builds uniquely identified by <build-id>:<chroot-name>.

Artifact types

In the initial implementation we will be targeting the following artifacts which are already supported by Testing Farm and tmt.

Supported by tmt

Documented in the prepare plugin documentation.

• rpm files

  • local rpm files
    • listing of files required
  • remote rpms files
    • listing of urls required
  • local directory with rpm files
    • by default, only enable the copr repository
    • possibility to exclude certain packages

• copr repository

  • enabled by supplied repository name, i.e. psss/tmt
    • possibility to install certain packages only
    • possibility to skip missing packages

Supported by Testing Farm

Documented in the Testing Farm API documentation, in the environments[].artifacts field.

Testing Farm by default downloads all artifacts including the source rpm, except for the repository-file artifact and creates a repository called test-artifacts via a post-artifact installation playbook

• Fedora koji builds

  • identified by koji build task ID, e.g. 43054146
  • by default, all rpms are installed from the build
  • possibility to exclude rpms, with regex support
  • possibility to skip installation
  • implementation details

• Red Hat brew builds

  • identified by brew build task ID, e.g. 43924499
  • by default, all rpms are installed from the build
  • possibility to exclude rpms, with regex support
  • possibility to skip installation
  • implementation details

• Fedora copr builds

  • identified by string <build-id>:<chroot-name>, e.g. 1784470:fedora-32-x86_64
  • by default, all rpms are installed from the build
  • possiblity to install only certain rpms from the repository, with regex support
  • possibility to exclude rpms, with regex support
  • possibility to skip installation
  • implementation details

• RPM repository

  • identified by a baseurl to the repository
  • the repository is not added to the system
  • by default, all rpms are installed from the repository
  • possiblity to install only certain rpms from the repository, with regex support
  • possibility to exclude rpms, with regex support
  • possibility to skip installation
  • implementation details

• RPM repository file

  • identified by an url to a repository file
  • repository file is added to /etc/yum.repos.d
  • no installation support
  • implementation details

Testing Farm Artifact Installation Procedure

Battle tested for all supported RHEL and Fedora versions.

A mixture of downgrade, reinstall, and install commands,
with a few hacks around weird dnf and yum behavior.
See the code in the implementation details links for each of the artifact types.

Proposed Solution

Create new prepare plugins which will implement the use cases and be compatible with the current tmt and Testing Farm use cases.

The installation procedure will be divided into two plugins.

Note that the names of the plugins are subject to discussion.

Plugin prepare-artifacts

Plugin will use priority 40.

This plugin will take care of gathering all rpms to be installed, and creating a common repository for them.

The plugin will have these steps:

• Download all rpms to the testing environment
• Create a single common repository for all the rpms in the testing environment
  • Note that this is a change from the proposal from the hacking day
  • The name of the repository should be configurable
  • The repository will be added with priority 9, which is higher than the default priority 99.
• Extract package nvrs and store them in tmt
  • Skip this for the repository-file artifact
  • Make it possible to skip this step if requested

The list of artifacts to be installed will be possible to specify on the command line, or via a yaml file.
The command line functionality can be limited in certain cases.
The configuration file should support all the functionality.

Note that the plugin was renamed from the original name prepare-repositories to better reflect its purpose and lower confusion with the repository artifact type.

Plugin install-artifacts

Plugin will use:

• priority 70 for required packages
• priority 75 for recommended packages

This plugin will take care of the installation of:

• artifacts provided by the prepare-artifacts plugin
• required packages from the tests
• recommended packages from the tests

The plugin will have these steps:

• Create a list of files to be installed, with locked versions, i.e. lock file

  • The required and recommended packages, which are provided by artifacts repository, will be locked to a certain version
  • Packages from the artifacts will be added with specific versions

• Create a list of packages to be verified

  • The recommended packages must not be installed, but if they are, they need to match the locked version (if applicable)

• Install packages from the lock file

  • Note that it is currently unclear, if a simple dnf install will be sufficient here to cover all the use cases.

• Verify all packages were installed correctly, check by hash, not by NVR

Testing

We will be creating an extensive test suite to make sure the designed plugins for as expected for all use cases and for all the supported distributions:

• RHEL 7 and later
• Fedora 37 and later
• CentOS Stream 8 and later

Additional Artifacts Support

We might want to support the following artifacts in the future:

• Red Hat copr builds

Todo

This is a list of todo items with the next steps in this effort.

• ⏳️ manually validate the proposal
  • ⏳️ koji build
    • ⏳️ httpd (non-system)
      • Fedora Rawhide
      • RHEL7
    • ⏳️ systemd (system)
      • Fedora Rawhide
      • RHEL7
  • copr build
  • repository
  • repository-file
  • koji + copr build
  • copr build + repository
  • copr bulid + repository file
• review of the discussion topic by the tmt community
• finalize the proposal according to the given comments
• decide about the implementation of internal artifacts, i.e. brew
• design the test suite
• prepare intial set of tests
• design the plugins
• implement koji/brew build artifact
• implement repository and repository file artifact
• implement copr repository artifact
• implement rpm file artifact
• implement copr build artifact

[happz]

Ok, let's start the bikeshedding: if we have prepare-artifacts, can we get install-artifacts instead of install-verify? :) The verify part seems more like a feature of the plugin, I sort of expect it to verify things before moving on, which leaves me with just install and that's already taken :) *-artifacts would align nicely.

And for something more tasty:

Fedora copr builds

Are other Copr sources, namely the internal one, out of the scope completely or just something with a very low priority?

This plugin (prepare-artifacts) will take care of gathering all rpms to be installed, and creating a common repository for them.

Does this cover also packages from require and recommend keys, or just the artifacts under the test?

• Create a list of packages to be verified
  • The above list without the recommended packages

I'd recommend including recommended packages in the list of packages to verify: if they were part of the lock file, the plugin should verify that, if a recommended package is eventually installed, the correct version is installed. If my recommend says foo-1.2-3.el9 and foo-3.2-1.el9 is installed by the plugin, would it be safe to ignore it? It might be correct to proceed with testing, depending on what's the exact semantics of recommend, but I would definitely expect tmt at least to report such a discrepancy.

[thrix]

Ok, let's start the bikeshedding: if we have prepare-artifacts, can we get install-artifacts instead of install-verify? :) The verify part seems more like a feature of the plugin, I sort of expect it to verify things before moving on, which leaves me with just install and that's already taken :) *-artifacts would align nicely.

Agreed, install-artifacts edited.

And for something more tasty:

Fedora copr builds

Are other Copr sources, namely the internal one, out of the scope completely or just something with a very low priority?

We currently do not support these in Testing Farm, seems there is no use case.
Anyway, if that need arises, I believe adding these will be easy, as we will have the solution for the similar problem space with koji/brew builds and the code for interacting with the copr instance.

I added a note about possible additional artifacts.

This plugin (prepare-artifacts) will take care of gathering all rpms to be installed, and creating a common repository for them.

Does this cover also packages from require and recommend keys, or just the artifacts under the test?

Currently, that is undecided, the idea was to create it during the install-artifacts phase, but we could do it also here.
I believe we can decide during the implmentation.
As it is preparation, maybe it will be cleaner to have it in the first plugin.

• Create a list of packages to be verified

  • The above list without the recommended packages

I'd recommend including recommended packages in the list of packages to verify: if they were part of the lock file, the plugin should verify that, if a recommended package is eventually installed, the correct version is installed. If my recommend says foo-1.2-3.el9 and foo-3.2-1.el9 is installed by the plugin, would it be safe to ignore it? It might be correct to proceed with testing, depending on what's the exact semantics of recommend, but I would definitely expect tmt at least to report such a discrepancy.

Yes, you are right, I updated that section, if they are locked, we should check them for the correct version.

[thrix]

@lukaszachy @psss kindly asking for review

[psss]

Thanks much for putting this together! Very nicely written. I believe all the essential steps are covered. Found just two minor typos:

As a user, I do not want to install the tested rpms during test execution, I just want to have the rpms available in the test environment in a repository.

Did you mean I do not want to install the tested rpms before test execution here? The point is that I don't want anything to be installed before I start doing whatever magic during the test execution, right?

• The recommended packages must not be installed, but if they are, they need to match the locked version (if applicable)

Probably recommended packages may not be installed, or does not have to be installed?

[happz]

@thrix FYI, I'm starting to refactor the "package managers" functionality in #2557. It does not work yet, but it should mature in a week or two at max, hopefully.

[martinhoyer]

Allow me to add one use-case:

As a user, I want to install a kernel from a Koji/Brew or a specific RPM repository. The system should boot into the installed kernel before the test execution.

[happz]

I see, I thought there’s a crucial difference in installation of RT kernel.

Basically this bools down to “install me a given kernel, and reboot before moving on”.

[martinhoyer]

Pretty much. Might be also worth verifying the installed kernel is default with grubby and that the system booted into it as expected.

[thrix]

@martinhoyer adding to the requirements, feels like user could indicate to reboot after artifact installation? would that be indication be sufficient for you?

[martinhoyer]

@thrix It would, thanks.

Verifying running kernel version would be nice-to-have, but I'm guessing that's out of scope for this. I imagine this and efibootmgr stuff to be with tmt-reboot.

[bgoncalv]

Note, not always just installing the kernel will change the default kernel.
It would be better to:

1. install the kernel
2. run grubby --set-default to specify the new kernel will be used for boot
3. reboot

example:
install kernel-debug, it won't be set as default. Same happens with kernel-64k...