Tap runner hides failures fix

This commit adds fixes for TAP runner to not hide failures when some
tests are skipped. Also, it updates Avocado documentation to respect
newest changes in TAP runner.

Reference: #6007
Signed-off-by: Jan Richter <[email protected]>

avocado/plugins/runners/tap.py

@@ -38,7 +38,7 @@ class TAPRunner(ExecTestRunner):
     @staticmethod
     def _get_tap_result(stdout):
         parser = TapParser(io.StringIO(stdout.decode()))
-        result = "error"
+        result = ""
         fail_reason = None
         for event in parser.parse():
             if isinstance(event, TapParser.Bailout):
@@ -50,16 +50,18 @@ def _get_tap_result(stdout):
                 break
             elif isinstance(event, TapParser.Plan):
                 if event.skipped:
-                    result = "skip"
-                    break
+                    if not result:
+                        result = "skip"
+                    continue
             elif isinstance(event, TapParser.Test):
                 if event.result == TestResult.FAIL:
                     result = "fail"
                     fail_reason = event.explanation
                     break
                 elif event.result == TestResult.SKIP:
-                    result = "skip"
-                    break
+                    if not result:
+                        result = "skip"
+                    continue
                 elif event.result == TestResult.XPASS:
                     result = "warn"
                     if event.name:

docs/source/guides/user/chapters/concepts.rst

@@ -207,6 +207,21 @@ difference is that the test result will be decided based on the
 produced output, that should be in `Test Anything Protocol
 <https://testanything.org>`_ format.
 
+Even though such executable can be seen as test suite from avocado point of view,
+it will be considered as one standalone test. If you want to get result of each
+test of such executable, you can get generated tap output in debug.log file.
+:ref:`avocado-log-files`
+
+.. note::
+  The result of Tap test is based on the importance of individual results types
+  like this:
+
+  `SKIP -> PASS -> FAIL`.  
+
+  This means if one of the tests in TAP output
+  is `not ok` the TAP test result is `FAIL`. If all tests are `ok` or `skip`
+  the results is `PASS` and if all results are `skip` the result is `SKIP`.
+
 Test statuses
 -------------
 

docs/source/guides/user/chapters/logging.rst

@@ -50,6 +50,8 @@ from all tests in one place.
  built-in streams in this function.
 
 
+.. _avocado-log-files:
+
 Avocado log files
 -----------------
 

selftests/unit/plugin/runners/tap.py

@@ -69,7 +69,7 @@ def test_skip(self):
         results = [status for status in runner.run(runnable)]
         last_result = results[-1]
         self.assertEqual(last_result["status"], "finished")
-        self.assertEqual(last_result["result"], "skip")
+        self.assertEqual(last_result["result"], "pass")
         self.assertEqual(last_result["returncode"], 0)
 
     @skipUnlessPathExists("/bin/sh")