Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

User Guide: improve section on Avocado's basic concepts #5887

Merged
merged 3 commits into from
Apr 4, 2024
Merged

User Guide: improve section on Avocado's basic concepts #5887

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
520c3f6
Docs/User Guide: cover all test types
clebergnu Mar 26, 2024
84753a7
Docs/User guide: avoid bitrot by using references
clebergnu Mar 26, 2024
7387648
Docs/User Guide: document the concept about the Test Resolver
clebergnu Mar 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 26 additions & 17 deletions docs/source/guides/user/chapters/concepts.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,20 @@ location::
JOB LOG : $HOME/avocado/job-results/job-2015-06-10T10.44-49ec339/job.log


Test Resolver
~~~~~~~~~~~~~

A test resolver is Avocado's component that will take a reference you
know about and will turn it into an actual test that can be run. This
reference, explained next, can be pretty much any string, but it'll
usually be some form of text containing the path to the file that
contains the test.

For more information please refer to :ref:`finding_tests`.

Test References
~~~~~~~~~~~~~~~

.. warning:: TODO: We are talking here about Test Resolver, but the reader was
not introduced to this concept yet.

A Test Reference is a string that can be resolved into
(interpreted as) one or more tests by the Avocado Test Resolver.
A given resolver plugin is free to interpret a test reference,
Expand Down Expand Up @@ -149,22 +157,23 @@ Test types
Avocado at its simplest configuration can run three different types of tests:

* Executable tests (``exec-test``)
* python unittest
* instrumented
* Python unittest tests (``python-unittest``)
* Avocado Instrumented tests (``avocado-instrumented``)
* TAP producing tests (``tap``)

You can mix and match those in a single job.

Avocado plugins can also introduce additional test types.

Executable Tests
Executable tests
~~~~~~~~~~~~~~~~

Any executable file can serve as a test. The criteria for PASS/FAIL is
the return code of the executable. If it returns 0, the test PASSes,
if it returns anything else, it FAILs.

Python unittest
~~~~~~~~~~~~~~~
Python unittest tests
~~~~~~~~~~~~~~~~~~~~~

The discovery of classical Python unittest is also supported, although unlike
Python unittest we still use static analysis to get individual tests so
Expand All @@ -173,8 +182,8 @@ no surprises when running unittests via Avocado.

.. _Instrumented:

Instrumented
~~~~~~~~~~~~
Avocado Instrumented tests
~~~~~~~~~~~~~~~~~~~~~~~~~~

These are tests written in Python or BASH with the Avocado helpers that use the
Avocado test API.
Expand All @@ -189,8 +198,8 @@ including logging, test result status and other more sophisticated test APIs.
Test statuses ``PASS``, ``WARN`` and ``SKIP`` are considered
successful. The ``ERROR``, ``FAIL`` and ``INTERRUPTED`` signal failures.

TAP
~~~
TAP producing tests
~~~~~~~~~~~~~~~~~~~

TAP tests are pretty much like executable tests in the sense that they are
programs (either binaries or scripts) that will executed. The
Expand Down Expand Up @@ -227,11 +236,11 @@ can have a good idea on what happened to the job.

The single individual exit codes are:

* AVOCADO_ALL_OK (0)
* AVOCADO_TESTS_FAIL (1)
* AVOCADO_JOB_FAIL (2)
* AVOCADO_FAIL (4)
* AVOCADO_JOB_INTERRUPTED (8)
* :data:`AVOCADO_ALL_OK <avocado.core.exit_codes.AVOCADO_ALL_OK>`
* :data:`AVOCADO_TESTS_FAIL <avocado.core.exit_codes.AVOCADO_TEST_FAIL>`
* :data:`AVOCADO_JOB_FAIL <avocado.core.exit_codes.AVOCADO_JOB_FAIL>`
* :data:`AVOCADO_FAIL <avocado.core.exit_codes.AVOCADO_FAIL>`
* :data:`AVOCADO_JOB_INTERRUPTED <avocado.core.exit_codes.AVOCADO_JOB_INTERRUPTED>`

If a job finishes with exit code `9`, for example, it means we had at least one
test that failed and also we had at some point a job interruption, probably due
Expand Down
Loading