From 520c3f63dd886fd9ba5a7fd561f48b23daa9d2a9 Mon Sep 17 00:00:00 2001 From: Cleber Rosa Date: Tue, 26 Mar 2024 11:31:10 -0400 Subject: [PATCH 1/3] Docs/User Guide: cover all test types TAP is also a basic type of test supported on standard Avocado, and it's currently missing in the list. While at it, let's use more meaningful names and add the types as the plugins list them. Signed-off-by: Cleber Rosa --- docs/source/guides/user/chapters/concepts.rst | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/source/guides/user/chapters/concepts.rst b/docs/source/guides/user/chapters/concepts.rst index 540cb4ecf2..c52d9266d9 100644 --- a/docs/source/guides/user/chapters/concepts.rst +++ b/docs/source/guides/user/chapters/concepts.rst @@ -149,22 +149,23 @@ Test types Avocado at its simplest configuration can run three different types of tests: * Executable tests (``exec-test``) -* python unittest -* instrumented +* Python unittest tests (``python-unittest``) +* Avocado Instrumented tests (``avocado-instrumented``) +* TAP producing tests (``tap``) You can mix and match those in a single job. Avocado plugins can also introduce additional test types. -Executable Tests +Executable tests ~~~~~~~~~~~~~~~~ Any executable file can serve as a test. The criteria for PASS/FAIL is the return code of the executable. If it returns 0, the test PASSes, if it returns anything else, it FAILs. -Python unittest -~~~~~~~~~~~~~~~ +Python unittest tests +~~~~~~~~~~~~~~~~~~~~~ The discovery of classical Python unittest is also supported, although unlike Python unittest we still use static analysis to get individual tests so @@ -173,8 +174,8 @@ no surprises when running unittests via Avocado. .. _Instrumented: -Instrumented -~~~~~~~~~~~~ +Avocado Instrumented tests +~~~~~~~~~~~~~~~~~~~~~~~~~~ These are tests written in Python or BASH with the Avocado helpers that use the Avocado test API. @@ -189,8 +190,8 @@ including logging, test result status and other more sophisticated test APIs. Test statuses ``PASS``, ``WARN`` and ``SKIP`` are considered successful. The ``ERROR``, ``FAIL`` and ``INTERRUPTED`` signal failures. -TAP -~~~ +TAP producing tests +~~~~~~~~~~~~~~~~~~~ TAP tests are pretty much like executable tests in the sense that they are programs (either binaries or scripts) that will executed. The From 84753a76a37add3766b97373cdd8ba5534a9d352 Mon Sep 17 00:00:00 2001 From: Cleber Rosa Date: Tue, 26 Mar 2024 11:31:10 -0400 Subject: [PATCH 2/3] Docs/User guide: avoid bitrot by using references Instead of having copies of the valus of the return codes, let's point at the actual values. Signed-off-by: Cleber Rosa --- docs/source/guides/user/chapters/concepts.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/guides/user/chapters/concepts.rst b/docs/source/guides/user/chapters/concepts.rst index c52d9266d9..05d80e1bb9 100644 --- a/docs/source/guides/user/chapters/concepts.rst +++ b/docs/source/guides/user/chapters/concepts.rst @@ -228,11 +228,11 @@ can have a good idea on what happened to the job. The single individual exit codes are: -* AVOCADO_ALL_OK (0) -* AVOCADO_TESTS_FAIL (1) -* AVOCADO_JOB_FAIL (2) -* AVOCADO_FAIL (4) -* AVOCADO_JOB_INTERRUPTED (8) +* :data:`AVOCADO_ALL_OK ` +* :data:`AVOCADO_TESTS_FAIL ` +* :data:`AVOCADO_JOB_FAIL ` +* :data:`AVOCADO_FAIL ` +* :data:`AVOCADO_JOB_INTERRUPTED ` If a job finishes with exit code `9`, for example, it means we had at least one test that failed and also we had at some point a job interruption, probably due From 73876480d8c2c3441628ffa86cea54e683a7db93 Mon Sep 17 00:00:00 2001 From: Cleber Rosa Date: Tue, 26 Mar 2024 11:31:10 -0400 Subject: [PATCH 3/3] Docs/User Guide: document the concept about the Test Resolver Instead of having a TODO, let's add a brief intro to the Test Resolver. Signed-off-by: Cleber Rosa --- docs/source/guides/user/chapters/concepts.rst | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/docs/source/guides/user/chapters/concepts.rst b/docs/source/guides/user/chapters/concepts.rst index 05d80e1bb9..b32048d3c3 100644 --- a/docs/source/guides/user/chapters/concepts.rst +++ b/docs/source/guides/user/chapters/concepts.rst @@ -24,12 +24,20 @@ location:: JOB LOG : $HOME/avocado/job-results/job-2015-06-10T10.44-49ec339/job.log +Test Resolver +~~~~~~~~~~~~~ + +A test resolver is Avocado's component that will take a reference you +know about and will turn it into an actual test that can be run. This +reference, explained next, can be pretty much any string, but it'll +usually be some form of text containing the path to the file that +contains the test. + +For more information please refer to :ref:`finding_tests`. + Test References ~~~~~~~~~~~~~~~ -.. warning:: TODO: We are talking here about Test Resolver, but the reader was - not introduced to this concept yet. - A Test Reference is a string that can be resolved into (interpreted as) one or more tests by the Avocado Test Resolver. A given resolver plugin is free to interpret a test reference,