Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tap runner hides failures #6007

Closed
richtja opened this issue Aug 22, 2024 · 1 comment
Closed

Tap runner hides failures #6007

richtja opened this issue Aug 22, 2024 · 1 comment
Assignees
Labels

Comments

@richtja
Copy link
Contributor

richtja commented Aug 22, 2024

Describe the bug
If one of the tests in tap output is skipped before some failure, avocado will mark the whole output as skip instead of fail and users are not notified about the failure.

Steps to reproduce
TAP output generated by tap test stored in example.tap:

TAP version 13

# MUST be treated as a SKIP test
ok 1 - must be skipped test # SKIP

not ok 2 - actual failure
ok 3 - passed test

bash script for reading this file as test.sh:

#!/bin/bash - 

cat example.tap

JSON for running the tap test as test.json:

[
    {"kind": "tap", "uri": "./test.sh"}
]

Avocado command:

$ avocado run test.json

Expected behavior
A clear and concise description of what you expected to happen.

JOB ID     : f9305c1fa08da1d0a2e0a668e76905e23182e46a
JOB LOG    : /home/janrichter/avocado/job-results/job-2024-08-22T11.41-f9305c1/job.log
 (1/1) /home/janrichter/Avocado/testing/tap/test.sh: STARTED
 (1/1) /home/janrichter/Avocado/testing/tap/test.sh: FAIL
RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/janrichter/avocado/job-results/job-2024-08-22T11.41-f9305c1/results.html
JOB TIME   : 3.51 s

Current behavior
What you are getting?

JOB ID     : f9305c1fa08da1d0a2e0a668e76905e23182e46a
JOB LOG    : /home/janrichter/avocado/job-results/job-2024-08-22T11.41-f9305c1/job.log
 (1/1) /home/janrichter/Avocado/testing/tap/test.sh: STARTED
 (1/1) /home/janrichter/Avocado/testing/tap/test.sh: SKIP
RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 1 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/janrichter/avocado/job-results/job-2024-08-22T11.41-f9305c1/results.html
JOB TIME   : 3.51 s

System information (please complete the following information):

  • OS:
LSB Version:    :core-5.0-amd64:core-5.0-noarch
Distributor ID: Fedora
Description:    Fedora release 40 (Forty)
Release:        40
Codename:       Forty
  • Avocado version: master
  • Avocado installation method: github
@richtja richtja added the bug label Aug 22, 2024
@clebergnu clebergnu added this to the 107 - Codename TBD milestone Aug 26, 2024
@clebergnu clebergnu assigned clebergnu and richtja and unassigned clebergnu Aug 26, 2024
richtja added a commit to richtja/avocado that referenced this issue Aug 26, 2024
This commit adds fixes for TAP runner to not hide failures when some
tests are skipped. Also, it updates Avocado documentation to respect
newest changes in TAP runner.

Reference: avocado-framework#6007
Signed-off-by: Jan Richter <[email protected]>
@richtja
Copy link
Contributor Author

richtja commented Sep 16, 2024

Resolved in #6010

@richtja richtja closed this as completed Sep 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done 108
Development

No branches or pull requests

2 participants