Stateful objects as test requirements and state composition

[pevogam]

Hi all,

This is a daring proposal that will no doubt raise a lot of questions and needs a full scale clarification on my side as well but I have some ideas for features that could improve the applications of Avocado to testing entire OS distributions. As our team uses Avocado together with our plugin extensively for testing our own Linux distribution, we have collected some years of experience in the QA of our own products and as our own end-to-end test coverage has grown to thousands of tests our needs for parallel testing have grown as well. We still do parallel runs at present but through LXC containers managed by external bash scripts and would like to eventually see them turned into LXC spawners to be managed by a single avocado run.

Our use cases however are more elaborate than this as we need to test entire OS releases during few months cycles so we have additional (nested) environments within the original container environment as virtual machines. The container (outer) environment is thus equivalent to an entire network of virtual machines (our vmnet subpackage) that we also fully automate for a given test and eventually wish to be able to freeze and reuse.

So before turning this RFC into an entire blueprint itself and overextending on details here and to keep it to the point, the containers and virtual machines we use are stateful objects where we could also switch between container/vm states (an example of such state manipulation would be QCOW2 snapshots or e.g. LVM snapshots for the vm images), a set of vms is a requirement for a test, and ultimately there is a state composition where some stateful objects may contain other stateful objects (e.g. containers might contain entire networks of vms). Such functionality could also be implemented as a separate plugin as usual (it currently is at least partially through our plugin) and it doesn't have to touch core avocado code as long as the core code remains compatible with such generalizations. The idea of this RFC is to ask what you think about this, do you like the idea for a potential blueprint, and what are the major questions or feedback you have at this stage.

Some final minimal elaboration to answer minimal potential questions:

Q) Why stateful objects (containers/vms/proccesses) rather than simple downloads of some cloudinit-prepared images?
A) The reason is space spent on storing and time spent on copying (locally or remotely) the images. A normal image download and some minimal local (each time after loading) or remote (once before storing in the cloud) setup would be fine if this customization is short enough but complex test scenarios are in a constant trade-off between (1) performing longer and longer customization to reach the preconditions of a certain test and (2) storing a separate image for all these customizations and spending alternative time loading these. The solution then is to condense a lot of these separate setup steps (and test variants that involve the test environment in some way) into states and store these together "differentiably", i.e. taking only the state diff for extra storage space and loading penalty.

Q) Why state composition then?
A) People might want to do larger scale simulations with macroscopic objects where setup reuse and speed of reruns plays the same major role. We recently had to debug an issue in StrongSwan that required us to simulate 30+ VPN tunnels at the same time and having to set all of them up takes significant amount of time even if we reuse vms at certain previous states. This is of course kept in mind as something general (whenever possible) to states of images within a vm, states of vms within a vm network, or other compositions. And such composition is just as usable for further speeding up the customizations by reusing different components to set up and store a composite state.

Alright, I will leave it to that at the moment to see some of your comments about what parts of this proposal made sense to you and what would need more explanation.

[beraldoleal]

Hi, @pevogam, happy new year, and thanks for bringing this up. IIUC, this seems a legit requirement and I think that would be welcome.

But just one question for your case: what about having an Ansible playbook (that would bootstrap your env) as requirements? Would be helpful somehow? Or it will still need some extra work to make it work?

[pevogam]

Hi @beraldoleal, happy new year to you too! Indeed Ansible is developed to make the life of DevOps easier through lots of networking automation, however the networks (as examples of nested environments and state composition) are fully virtualized and can have different topology for each test. This means different topology down to the "cabling" and the way we connect any participating vms, not just automating their roles (web servers or other servers with an SSH-based pushes like the ones form Ansible). From what I know Ansible also doesn't have the notion of "states" above and just as cloudinit or other initialization software it will redo the setup for each machine with a given role. This is ok for cases where DevOps might perform one such configuration push per update or network extension but in our case we are talking about QA and tests of a very high number of variants and only partial overlapping of setup where such reruns are not optimal for performance. A lot of the concepts above are the result of optimizing the process of running large numbers of end-to-end tests and reusing as much setup as possible to cover all their variants.

We also configure clients with aexpect and basic SSH sessions but while Ansible's playbooks and configuration are adapted for DevOps efficiency, we need more of the flexibility of more programming (incl. conditionals) on top of the same configurations which are a mere test config to define the test variants.

I could explain more and possibly in (pseudo)code in case you want, perhaps this would illustrate the points here better.

[pevogam]

Attaching some notes here with examples and additional explanations on terms I have used above. Some of them might not be clear but I am attaching them here for visibility and so that I am able to use them for a potential blueprint if there is sufficient interest.

[beraldoleal]

Hi @pevogam I quite hard to understand some words, if you don't mind, could you convert this to an SVG or other digital format?

[pevogam]

Hehe, no worries this is just some initial note taking that I store here where I specified the ideas, everything will be converted to something more formal pretty soon. 👍

[pevogam]

I converted/formalized this into an implementation candidate in our plugin since there doesn't seem to be much interest on the avocado side for the time being. Thanks for the input @beraldoleal, if you think anything here could be useful for avocado even if it is a small feature let me know and I could assist with blueprints, PR-s and so on.

[pevogam]

There is some progress in this direction because LXC supports nesting of containers (containers within containers) which could easily become the backbone of such state composition.