-
Notifications
You must be signed in to change notification settings - Fork 0
RegressionTestFarm
You have all upstream code, and you're wondering if the internal Red Hat testing of KVM has a lot of internal 'secret sauce'. No, it does not.
However, it is a complex endeavor, since there are lots of details involved. The farm setup and maintenance is not easy, given the large amounts of things that can fail (machines misbehave, network problems, git repos unavailable, so on and so forth). You have been warned.
With all that said, we'll share what we have been doing. We did clean up our config files and extensions and released them upstream, together with this procedure, that we hope it will be useful to you guys. Also, this will cover KVM testing on a single host, as tests involving multiple hosts and Libvirt testing are a work in progress.
The basic steps are:
- Install an autotest server.
- Add machines to the server (test nodes). Those machines are the virt hosts that will be tested.
- Prepare the virt test jobs and schedule them.
- Set up cobbler in your environment so you can install hosts.
- Lots of trial and error until you get all little details sorted out.
We took years repeating all the steps above and perfecting the process, and we are willing to document it all to the best extent possible. I'm afraid however, that you'll have to do your homework and adapt the procedure to your environment.
We are assuming you will install autotest to its default upstream location
/usr/local/autotest
Therefore a lot of paths referred here will have this as the base dir.
During this text, we'll use frequently the terms CLI and Web UI.
By CLI we mean specifically the program:
/usr/local/autotest/cli/autotest-rpc-client
That is located in the autotest code checkout.
By Web UI, we mean the web interface of autotest, that can be accessed through
http://your-autotest-server.com/afe
Provided that you have internet on your test lab, this should be the easiest step. Pick up either a VM accessible in your lab, or a bare metal machine (it really doesn't make a difference, we use a VM here). We'll refer it from now on as the "Server" box.
The hard drive of the Server should hold enough room for test results. We found out that at least 250 GB holds data for more than 6 months, provided that QEMU doesn't crash a lot.
You'll follow the procedure described on
https://github.com/autotest/autotest/wiki/AutotestServerInstallRedHat
for Red Hat derivatives (such as Fedora and RHEL), and
https://github.com/autotest/autotest/wiki/AutotestServerInstall
for Debian derivatives (Debian, Ubuntu).
Note that using the install script referred right in the beginning of the documentation is the preferred method, and should work pretty well if you have internet on your lab. In case you don't have internet there, you'd need to follow the instructions after the 'installing with the script' instructions. Let us know if you have any problems.
It should go without saying, but the machines you have to add have to be virtualization capable (support KVM).
You can add machines either by using the CLI or the Web UI, following the documentation:
https://github.com/autotest/autotest/wiki/ConfiguringHosts
If you don't want to read that, I'll try to write a quick howto.
Say you have two x86_64 hosts, one AMD and the other, Intel. Their hostnames are:
foo-amd.bazcorp.com foo-intel.bazcorp.com
I would create 2 labels, amd64 and intel64, I would also create a label to indicate the machines can be provisioned by cobbler. This is because you can tell autotest to run a job in any machine that matches a given label.
Logged as the autotest user:
$ /usr/local/autotest/cli/autotest-rpc-client label create -t amd64
Created label:
'amd64'
$ /usr/local/autotest/cli/autotest-rpc-client label create -t intel64
Created label:
'intel64'
$ /usr/local/autotest/cli/autotest-rpc-client label create hostprovisioning
Created label:
'hostprovisioning'
Then I'd create each machine with the appropriate labels
$ /usr/local/autotest/cli/autotest-rpc-client host create -t amd64 -b hostprovisioning foo-amd.bazcorp.com
Added host:
foo-amd.bazcorp.com
$ /usr/local/autotest/cli/autotest-rpc-client host create -t amd64 -b hostprovisioning foo-intel.bazcorp.com
Added host:
foo-amd.bazcorp.com
Now you have to copy the plugin we have developed to extend the CLI to parse additional information for the virt jobs:
cp /usr/local/autotest/contrib/virt/site_job.py /usr/local/autotest/cli/
This should be enough to enable all the extra functionality.
You also need to copy the site-config.cfg file that we published as a reference, to the qemu config module:
cp /usr/local/autotest/contrib/virt/site-config.cfg /usr/local/autotest/client/tests/virt/qemu/cfg
Be aware that you need to read this file well, and later, configure it to your testing needs. We specially stress that you might want to create private git mirrors of the git repos you want to test, so you tax the upstream mirrors less, and have increased reliability.
Right now it is able to run regression testing on Fedora 18, and upstream kvm, provided that you have a cobbler instance functional, with a profile called f18-autotest-kvm that can be properly installed on your machines. Having that properly set up may open another can of worms.
One simple way to schedule the jobs, that we does use at our server, is to use cron to schedule daily testing jobs of the things you want to test. Here is an example that should work 'out of the box'. Provided that you have an internal mailing list that you created with the purpose of receiving email notifications, called [email protected], you can stick that on the crontab of the user autotest in the Server:
07 00 * * 1-7 /usr/local/autotest/cli/autotest-rpc-client job create -B never -a never -s -e [email protected] -f "/usr/local/autotest/contrib/virt/control.template" -T --timestamp -m '1*hostprovisioning' -x 'only qemu-git..sanity' "Upstream qemu.git sanity" 15 00 * * 1-7 /usr/local/autotest/cli/autotest-rpc-client job create -B never -a never -s -e [email protected] -f "/usr/local/autotest/contrib/virt/control.template" -T --timestamp -m '1*hostprovisioning' -x 'only f18..sanity' "Fedora 18 koji sanity" 07 01 * * 1-7 /usr/local/autotest/cli/autotest-rpc-client job create -B never -a never -s -e [email protected] -f "/usr/local/autotest/contrib/virt/control.template" -T --timestamp -m '1*hostprovisioning' -x 'only qemu-git..stable' "Upstream qemu.git stable" 15 01 * * 1-7 /usr/local/autotest/cli/autotest-rpc-client job create -B never -a never -s -e [email protected] -f "/usr/local/autotest/contrib/virt/control.template" -T --timestamp -m '1*hostprovisioning' -x 'only f18..stable' "Fedora 18 koji stable"
That should be enough to have one sanity and stable job for:
- Fedora 18.
- qemu.git userspace and kvm.git kernel.
What does these 'stable' and 'sanity' jobs do? In short:
- Host OS (Fedora 18) installation through cobbler
- Latest kernel for the Host OS installation (either the last kernel update build for fedora, or check out, compile and install kvm.git).
- 'sanity' job:
** Install latest Fedora 18 qemu-kvm, or compiles the latest qemu.git ** Installs a VM with Fedora 18, boots, reboots, does simple, single host migration with all supported protocols ** Takes about two hours. In fact, internally we test more guests, but they are not widely available (RHEL 6 and Windows 7), so we just replaced them with Fedora 18.
- 'stable' job:
** Same as above, but many more networking, timedrift and other tests
Cobbler is an installation server, that control DHCP and/or PXE boot for your x86_64 bare metal virtualization hosts. You can learn how to set it up in the following resource:
https://github.com/cobbler/cobbler/wiki/Start%20Here
You will set it up for simple installations, and you probably just need to import a Fedora 18 DVD into it, so it can be used to install your hosts. Following the import procedure, you'll have a 'profile' created, which is a label that describes an OS that can be installed on your virtualization host. The label we chose, as already mentioned is f18-autotest-kvm. If you want to change that name, you'll have to change site-config.cfg accordingly.
Also, you will have to add your test machines to your cobbler server, and will have to set up remote control (power on/off) for them.
The following is important:
The hostname of your machine in the autotest server has to be the name of your system in cobbler.
So, for the hypothetical example you'll have to have set up systems with names foo-amd.bazcorp.com foo-intel.bazcorp.com in cobbler. That's right, the 'name' of the system has to be the 'hostname'. Otherwise, autotest will ask cobbler and cobbler will not know which machine autotest is taking about.
Other assumptions we have here:
1) We have a (read only, to avoid people deleting isos by mistake) NFS share that has the Fedora 18 DVD and other ISOS. The structure for the base dir could look something like:
.
|-- linux
| `-- Fedora-18-x86_64-DVD.iso
`-- windows
|-- en_windows_7_ultimate_x64_dvd_x15-65922.iso
|-- virtio-win.iso
`-- winutils.iso
This is just in case you are legally entitled to download and use Windows 7, for example.
2) We have another NFS share with space for backups of qcow2 images that got corrupted during testing, and you want people to analyze them. The structure would be:
. |-- foo-amd `-- bar-amd
That is, one directory for each host machine you have on your grid. Make sure they end up being properly configured in the kickstart.
Now here is one excerpt of kickstart with some of the gotchas we learned with experience. Some notes:
- This is not a fully formed, functional kickstart, just in case you didn't notice.
- This is provided in the hopes you read it, understand it and adapt things to your needs. If you paste this into your kickstart and tell me it doesn't work, I WILL silently ignore your email, and if you insist, your emails will be filtered out and go to the trash can.
install text reboot lang en_US keyboard us rootpw --iscrypted [your-password] firewall --disabled selinux --disabled timezone --utc America/New_York firstboot --disable services --enabled network --disabled NetworkManager bootloader --location=mbr ignoredisk --only-use=sda zerombr clearpart --all --drives=sda --initlabel autopart network --bootproto=dhcp --device=eth0 --onboot=on %packages @core @development-libs @development-tools @virtualization wget dnsmasq genisoimage python-imaging qemu-kvm-tools gdb iasl libvirt python-devel ntpdate gstreamer-plugins-good gstreamer-python dmidecode popt-devel libblkid-devel pixman-devel mtools koji tcpdump bridge-utils dosfstools %end %post echo "[nfs-server-that-holds-iso-images]:[nfs-server-that-holds-iso-images]/base_path/iso /var/lib/virt_test/isos nfs ro,defaults 0 0" >> /etc/fstab echo "[nfs-server-that-holds-iso-images]:[nfs-server-that-holds-iso-images]/base_path/steps_data /var/lib/virt_test/steps_data nfs rw,nosuid,nodev,noatime,intr,hard,tcp 0 0" >> /etc/fstab echo "[nfs-server-that-has-lots-of-space-for-backups]:/base_path/[dir-that-holds-this-hostname-backups] /var/lib/virt_test/images_archive nfs rw,nosuid,nodev,noatime,intr,hard,tcp 0 0" >> /etc/fstab mkdir -p /var/lib/virt_test/isos mkdir -p /var/lib/virt_test/steps_data mkdir -p /var/lib/virt_test/images mkdir -p /var/lib/virt_test/images_archive mkdir --mode=700 /root/.ssh echo 'ssh-dss [the-ssh-key-of-the-Server-autotest-user]' >> /root/.ssh/authorized_keys chmod 600 /root/.ssh/authorized_keys ntpdate [your-ntp-server] hwclock --systohc systemctl mask tmp.mount %end
After all that, you can start running your test jobs and see what things will need to be fixed. You can run your jobs easily by logging into your Server, with the autotest user, and use the command:
/usr/local/autotest/cli/autotest-rpc-client job create -B never -a never -s -e [email protected] -f "/usr/local/autotest/contrib/virt/control.template" -T --timestamp -m '1*hostprovisioning' -x 'only f18..sanity' "Fedora 18 koji sanity"
As you might have guessed, this will schedule a Fedora 18 sanity job. So go through it and fix things step by step. If anything, you can take a look at this:
https://github.com/autotest/autotest/wiki/DiagnosingFailures
And see if it helps. You can also ask on the mailing list, but please, pretty please do your homework before you ask us to guide you through all the process step by step. This is already a step by step procedure.
All right, good luck, and happy testing!