Releases: avocado-framework/avocado
Avocado 0.26.0: The Office
Avocado 0.26.0 released!
Hi guys, I'm here to announce avocado 0.26.0. This release was dedicated to
polish aspects of the avocado user experience, such as documentation and
behavior.
Changes
- Now avocado tests that raise exceptions that don't inherit from
avocado.core.exceptions.TestBaseExceptionnow will be marked as ERRORs.
This change was made to make avocado to have clearly defined test statuses.
A new decorator,avocado.fail_on_errorwas added to let arbitary exceptions
to raise errors, if users need a more relaxed behavior. - The avocado.Test() utility method skip() now can only be called from inside
the setUp() method. This was made because by definition, if we get to the
test execution step, by definition it can't be skipped anymore. It's important
to keep the concepts clear and well separated if we want to give users a good
experience. - More documentation polish and updates. Make sure you check out our documentation
website http://avocado-framework.readthedocs.org/en/latest/. - A number of avocado command line options and help text was reviewed and
updated. - A new, leaner and mobile friendly version of the avocado website is live.
Please check http://avocado-framework.github.io/ for more information. - We have the first version of the avocado dashboard! avocado dashboard is the
initial version of an avocado web interface, and will serve as the frontend
to our testing database. You can check out a screenshot here
https://cloud.githubusercontent.com/assets/296807/8536678/dc5da720-242a-11e5-921c-6abd46e0f51e.png
- And the usual bugfixes. You can take a look at the full list of 68 commits here:
0.25.0...0.26.0
Avocado 0.25.0: Blade
Avocado 0.25.0 released!
Hi guys, I'm here to announce the newest avocado release, 0.25.0. This is an
important milestone in avocado development, and we would like to invite
you to be a part of the development process, by contributing PRs, testing
and giving feedback on the test runner's usability and new plugins we came
up with.
What to expect
This is the first release aimed for general use. We did our best to deliver
a coherent and enjoyable experience, but keep in mind that it's a young
project, so please set your expectations accordingly. What is expected to work
well:
- Running avocado 'instrumented' tests
- Running arbitrary executables as tests
- Automatic test discovery and run of tests on directories
- xUnit/JSON report
Known Issues
- HTML report of test jobs with multiplexed tests has a minor naming display
issue that is scheduled to be fixed by next release. - avocado-vt might fail to load if virt-test was not properly bootstrapped.
Make sure you always run bootstrap in the virt-test directory on any
virt-test git updates to prevent the issue. Next release will have more
mechanisms to give the user better error messages on tough to judge
situations (virt-test repo with stale or invalid config files that need
update).
Changes
-
The Avocado API has been greatly streamlined. After a long discussion and
several rounds of reviews and planning, now we have a clear separation of
what is intended as functions useful for test developers and plugin/core
developers:- avocado.core is intended for plugin/core developers. Things are more fluid
on this space, so that we can move fast with development - avocado.utils is a generic library, with functions we found out to be
useful for a variety of tests and core code alike. - avocado has some symbols exposed at its top level, with the test API:
- our Test() class, derived from the unittest.TestCase() class
- a main() entry point, similar to unittest.main()
- VERSION, that gives the user the avocado version (eg 0.25.0).
Those symbols and classes/APIs will be changed more carefully, and release
notes will certainly contain API update notices. In other words, we'll be
a lot more mindful of changes in this area, to reduce the maintenance cost
of writing avocado tests.
We believe this more strict separation between the available APIs will help
test developers to quickly identify what they need for test development, and
reduce following a fast moving target, what usually happens when we have a
new project that does not have clear policies behind its API design. - avocado.core is intended for plugin/core developers. Things are more fluid
-
There's a new plugin added to the avocado project: avocado-vt. This plugin
acts as a wrapper for the virt-test test suite
(https://github.com/autotest/virt-test), allowing people to use avocado to
list and run the tests available for that test suite. This allows people to
leverage a number of the new cool avocado features for the virt tests
themselves:- HTML reports, a commonly asked feature for the virt-test suite. You can
see a screenshot of what the report looks like here:
https://cloud.githubusercontent.com/assets/296807/7406339/7699689e-eed7-11e4-9214-38a678c105ec.png - You can run virt-tests on arbitrary order, and multiple instances of a
given test, something that is also currently not possible with the virt
test runner (also a commonly asked feature for the suite. - System info collection. It's a flexible feature, you get to configure
easily what gets logged/recorded between tests.
- HTML reports, a commonly asked feature for the virt-test suite. You can
-
The avocado multiplexer (test matrix representation/generation system)
also received a lot of work and fixes during this
release. One of the most visible (and cool) features of 0.25.0 is the new,
improved --tree representation of the multiplexer file:$ avocado multiplex examples/mux-environment.yaml -tc ┗━━ run ┣━━ hw ┃ ┣━━ cpu ┃ ┃ ╠══ intel ┃ ┃ ║ → cpu_CFLAGS: -march=core2 ┃ ┃ ╠══ amd ┃ ┃ ║ → cpu_CFLAGS: -march=athlon64 ┃ ┃ ╚══ arm ┃ ┃ → cpu_CFLAGS: -mabi=apcs-gnu -march=armv8-a -mtune=arm8 ┃ ┗━━ disk ┃ ╠══ scsi ┃ ║ → disk_type: scsi ┃ ╚══ virtio ┃ → disk_type: virtio ┣━━ distro ┃ ╠══ fedora ┃ ║ → init: systemd ┃ ╚══ mint ┃ → init: systemv ┗━━ env ╠══ debug ║ → opt_CFLAGS: -O0 -g ╚══ prod → opt_CFLAGS: -O2We hope you find the multiplexer useful and enjoyable.
-
If an avocado plugin fails to load, due to factors such as missing
dependencies, environment problems and misconfiguration, in order to notify
users and make them mindful of what it takes to fix the root causes for the
loading errors, those errors are displayed in the avocado stderr stream.However, often we can't fix the problem right now and don't need
the constant stderr nagging. If that's the case, you can set in your local
config file:[plugins] # Suppress notification about broken plugins in the app standard error. # Add the name of each broken plugin you want to suppress the notification # in the list. The names can be easily seen from the stderr messages. Example: # avocado.core.plugins.htmlresult ImportError No module named pystache # add 'avocado.core.plugins.htmlresult' as an element of the list below. skip_broken_plugin_notification = [] -
Our documentation has received a big review, that led to a number of
improvements. Those can be seen online
(http://avocado-framework.readthedocs.org/en/latest/), but if you feel so
inclined, you can build the documentation for local viewing, provided that
you have the sphinx python package installed by executing:$ make -C docs html
Of course, if you find places where our documentation needs
fixes/improvements, please send us a PR and we'll gladly review it. -
As one would expect, many bugs were fixed. You can take a look at the full
list of 156 commits here:
0.24.0...0.25.0
0.24.0: A Beautiful Mind
We are proud to announce the new Avocado 0.24.0 release! This time, we had made a number of API changes to reflect some good practices in API development, and changes to the multiplexer, that is making large steps towards a production ready state. Let's break down the changes:
Core/General API isolation
In order to avoid user confusion, now the avocado API has a clear separation with regards to what it's supposed to be used by test developers, and what is supposed to be used by plugin/core application developers:
avocado.coreis restricted to plugin/core app developersavocadocan be used by both test developers and plugin/core app developers
Breakdown of changes:
avocado.core.data_dir->avocado.data_diravocado.plugins->avocado.core.pluginsavocado.remote->avocado.core.remoteavocado.job->avocado.core.jobavocado.job.main->avocado.mainavocado.loader->avocado.core.loaderavocado.output->avocado.core.outputavocado.loader->avocado.core.loaderavocado.parser->avocado.core.parseravocado.runner->avocado.core.runneravocado.sysinfo->avocado.core.sysinfoavocado.external.gdbmi_parser->avocado.core.gdbmi_parseravocado.external.spark->avocado.core.sparkavocado.cli.app->avocado.core.appavocado.exceptions->avocado.core.exceptions
Multiplexer
The multiplexer has gone through changes on the parameters retrieval API. The new way of retrieving params is through the get API, that has the form::
get(key, path, default)
The avocado.test.Test.params object is no longer a dictionary, and it's not possible to refer to the values through the shortcut avocado.test.Test.params.key anymore.
Example of old usage::
c_file = self.get_data_path(self.params.get('source', 'doublefree.c'))
This statement has to be updated to::
c_file = self.get_data_path(self.params.get('source', default='doublefree.c'))
Otherwise the default doublefree.c value will be taken as the retrieval path, then c_file would be None.
The multiplexer file format also has a new !mux tag, that means to recursively multiplex all nodes below the specified tag. By default now, we don't multiplex until the !mux tag is specified. Let's consider the following file::
hw:
cpu:
intel:
cpu_CFLAGS: '-march=core2'
amd:
cpu_CFLAGS: '-march=athlon64'
arm:
cpu_CFLAGS: '-mabi=apcs-gnu -march=armv8-a -mtune=arm8'
disk:
scsi:
disk_type: 'scsi'
virtio:
disk_type: 'virtio'
distro:
fedora:
init: 'systemd'
mint:
init: 'systemv'
env:
debug:
opt_CFLAGS: '-O0 -g'
prod:
opt_CFLAGS: '-O2'
Here, the nodes cpu, disk, distro and env are the ones we actually want to combine, while hw is just for organization purposes (we don't want to combine hw with distro, really).
New version, with !mux:
hw:
cpu: !mux
intel:
cpu_CFLAGS: '-march=core2'
amd:
cpu_CFLAGS: '-march=athlon64'
arm:
cpu_CFLAGS: '-mabi=apcs-gnu -march=armv8-a -mtune=arm8'
disk: !mux
scsi:
disk_type: 'scsi'
virtio:
disk_type: 'virtio'
distro: !mux
fedora:
init: 'systemd'
mint:
init: 'systemv'
env: !mux
debug:
opt_CFLAGS: '-O0 -g'
prod:
opt_CFLAGS: '-O2'
This way we can keep using hw as a organization label, not interfering with our final purpose of combining only cpu, disk, distro and env. The variants generated with the new tag arrangement are:
Variant 1: /hw/cpu/intel, /hw/disk/scsi, /distro/fedora, /env/debug
Variant 2: /hw/cpu/intel, /hw/disk/scsi, /distro/fedora, /env/prod
Variant 3: /hw/cpu/intel, /hw/disk/scsi, /distro/mint, /env/debug
Variant 4: /hw/cpu/intel, /hw/disk/scsi, /distro/mint, /env/prod
Variant 5: /hw/cpu/intel, /hw/disk/virtio, /distro/fedora, /env/debug
Variant 6: /hw/cpu/intel, /hw/disk/virtio, /distro/fedora, /env/prod
Variant 7: /hw/cpu/intel, /hw/disk/virtio, /distro/mint, /env/debug
Variant 8: /hw/cpu/intel, /hw/disk/virtio, /distro/mint, /env/prod
Variant 9: /hw/cpu/amd, /hw/disk/scsi, /distro/fedora, /env/debug
Variant 10: /hw/cpu/amd, /hw/disk/scsi, /distro/fedora, /env/prod
Variant 11: /hw/cpu/amd, /hw/disk/scsi, /distro/mint, /env/debug
Variant 12: /hw/cpu/amd, /hw/disk/scsi, /distro/mint, /env/prod
Variant 13: /hw/cpu/amd, /hw/disk/virtio, /distro/fedora, /env/debug
Variant 14: /hw/cpu/amd, /hw/disk/virtio, /distro/fedora, /env/prod
Variant 15: /hw/cpu/amd, /hw/disk/virtio, /distro/mint, /env/debug
Variant 16: /hw/cpu/amd, /hw/disk/virtio, /distro/mint, /env/prod
Variant 17: /hw/cpu/arm, /hw/disk/scsi, /distro/fedora, /env/debug
Variant 18: /hw/cpu/arm, /hw/disk/scsi, /distro/fedora, /env/prod
Variant 19: /hw/cpu/arm, /hw/disk/scsi, /distro/mint, /env/debug
Variant 20: /hw/cpu/arm, /hw/disk/scsi, /distro/mint, /env/prod
Variant 21: /hw/cpu/arm, /hw/disk/virtio, /distro/fedora, /env/debug
Variant 22: /hw/cpu/arm, /hw/disk/virtio, /distro/fedora, /env/prod
Variant 23: /hw/cpu/arm, /hw/disk/virtio, /distro/mint, /env/debug
Variant 24: /hw/cpu/arm, /hw/disk/virtio, /distro/mint, /env/prod
The tree representation also changed to print the !mux points of branching, to further help the user to understand the structure of the tree::
$ scripts/avocado multiplex --tree examples/mux-environment.yaml
Config file tree structure:
/-intel
|
/cpu-<>--amd
| |
/hw \-arm
| |
| | /-scsi
| \disk-<>
| \-virtio
-|
| /-fedora
|-distro-<>
| \-mint
|
| /-debug
\env-<>
\-prod
The more detailed breakdown of the 145 commits can be seen here:
Happy testing!
0.23.0: Mad Max: Fury Road
Hello everyone! I'm here to announce the new Avocado 0.23.0 release! This time around, we have a few changes worth of note:
- Now avocado looks more like nose and other unittest based test runners: You can define a number of test methods prefixed with
test, and the runner will simply run them. Also, for compatibility with the base unittest class, nowsetupandcleanuparesetUpandtearDown.
Let's see how an old avocado test looked like:
#!/usr/bin/python
from avocado import test
from avocado import job
class OldTest(test.Test):
def setup(self):
self.hello = "Hi there!"
def action(self):
self.assertEqual(self.hello, "Hi there!")
def cleanup(self):
self.log.debug('Cleanup')
if __name__ == '__main__':
job.main()
The new avocado tests look like this:
#!/usr/bin/python
from avocado import test
from avocado import job
class MultipleTests(test.Test):
def setUp(self):
self.hello = "Hi there!"
def test_hello(self):
self.assertEqual(self.hello, "Hi there!")
def testIdentity(self):
self.assertTrue(1, 1)
def tearDown(self):
self.log.debug('Cleanup')
if __name__ == '__main__':
job.main()
avocado run: Simplified command line options, cutting off a number of rarely used options and moving them to config file entriesavocado run: Added the--job-timeoutoption, that allows users to specify the maximum time an avocado test job might take.avocado run: Consolidated the options--remote-timeoutand--vm-timeoutinto--job-timeout.- Sysinfo feature: The sysinfo feature went through refactoring and simplification - Now it is possible to specify what information from the system is to be collected through config files
See how to configure your collectibles:
[sysinfo.collectibles]
# File with list of commands that will be executed and have their output collected
commands = /etc/avocado/sysinfo/commands
# File with list of files that will be collected verbatim
files = /etc/avocado/sysinfo/files
# File with list of commands that will run alongside the job/test
profilers = /etc/avocado/sysinfo/profilers
The default commands file looks like:
df -mP
dmesg -c
uname -a
lspci -vvnn
gcc --version
ld --version
mount
hostname
uptime
dmidecode
ifconfig -a
brctl show
ip link
numactl --hardware show
lscpu
fdisk -l
- Multiplexer: A new params retrieval system is in place. Now, instead of
self.params.keyyou must use theself.params.get(key, path, default)API. A compatibility layer with the old dict structure was made, but we encourage you to migrate your param retrieval calls fromself.params.keyorself.params['key'] toself.params.get('key').
The full list of commits can be seen here:
Happy testing!
Clean and Sober
- Avocado is now compatible with python 2.6 and distros that ship 2.6
as their default runtime, such as RHEL 6 and CentOS 6. Our CI
environment was updated to test under 2.6 and 2.7 targets - Avocado rest client application (lets you talk to the avocado server
application) - A new option was added to disable colored output in the config file,
runner.output.colored - A new option was added to add commands that will profile new test job
runs, sysinfo.collect.profiler_commands - The documentation was updated with GDB instrumentation examples
(besides the normal docs update on new features/patchsets) - API Change: avocado.utils.io -> avocado.utils.genio
- API Change: avocado.utils.process.find_command ->
avocado.utils.path.find_command - API Change: avocado.utils.process.CmdNotFoundError ->
avocado.utils.path.CmdNotFoundError - BUGFIX: remote machine could get stuck waiting on human input on
paginator - BUGFIX: Avocado crashes on certain test defined exceptions
- BUGFIX: Internal HTML report links when using remote/vm plugins
- BUGFIX: Job directory generated even when 'avocado run' had option
validation errors - BUGFIX: When installing avocado as an RPM, some example tests stop
working - BUGFIX: Avocado crashes if a remote test is missing
The complete list with the 130 commits can be seen here:
Lovers on the Sun
New features:
avocado:
- HTML job report plugin, that gives users the ability to generate a nice
and good looking report of the testing activities. - --wrapper feature, that allows users to write scripts to wrap the execution
of test commands. You can use it to profile your QEMU instance during the
test with perf/ltrace, for example. - Support for the GDB remote protocol inside avocado.
- A new config plugin, that allows users to see the current configuration
keys set for their particular instances of avocado, and help you to modify
values if you wish so. - Tweaks and documentation improvements in the requirements.txt file, and the
split of the file into other requirements files, with more specific purpose. - Fix a deadlock that occurred when the main test code has an early abort.
- Fix a temporary dir issue, that had potential security implications.
avocado-virt plugin:
- Introduction of QEMU command line templates, to make it easier for people to
provide specific command lines to be run inside avocado virt specific tests. - Fix a bug in migration process end detection, the code is now more robust and
reliable.
avocado-server:
- Introduction of software components and test environment data models,
preparing the server for a big feature: Automated bisection of failed tests.
infrastructure:
- Avocado has now a web page, courtesy of github pages: