Skip to content

Allow skip and xfail to both be applied to test cases #19552

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 4 additions & 6 deletions mypy/test/data.py
Original file line number Diff line number Diff line change
Expand Up @@ -312,8 +312,7 @@ def __init__(
def runtest(self) -> None:
if self.skip:
pytest.skip()
# TODO: add a better error message for when someone uses skip and xfail at the same time
elif self.xfail:
if self.xfail:
self.add_marker(pytest.mark.xfail)
parent = self.getparent(DataSuiteCollector)
assert parent is not None, "Should not happen"
Expand Down Expand Up @@ -659,8 +658,7 @@ def pytest_pycollect_makeitem(collector: Any, name: str, obj: object) -> Any | N
r"(?P<only_when>-only_when_cache|-only_when_nocache)?"
r"(?P<skip_path_normalization>-skip_path_normalization)?"
r"(-(?P<platform>posix|windows))?"
r"(?P<skip>-skip)?"
r"(?P<xfail>-xfail)?"
r"(?:(?P<skip>-skip)?(?P<xfail>-xfail)?|(?P<xfail2>-xfail)?(?P<skip2>-skip)?)?"
)


Expand Down Expand Up @@ -699,8 +697,8 @@ def split_test_cases(
writescache=bool(m.group("writescache")),
only_when=m.group("only_when"),
platform=m.group("platform"),
skip=bool(m.group("skip")),
xfail=bool(m.group("xfail")),
skip=bool(m.group("skip") or m.group("skip2")),
xfail=bool(m.group("xfail") or m.group("xfail2")),
normalize_output=not m.group("skip_path_normalization"),
data=data,
line=line_no,
Expand Down
20 changes: 20 additions & 0 deletions test-data/unit/check-skipfail.test
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
-- Because this file tests both "-skip" and "-xfail" themselves its results are not very meaningful, except that they should not fail.
-- If you ever want to stop getting s and x results for these, you will have to figure out how to test these features instead.

-- What does it mean to both skip and xfail a test? Well, since the two are semantically different, according to https://docs.pytest.org/en/stable/how-to/skipping.html, there may be a meaningful answer to that question. For instance, "skipping tests that depend on an external resource which is not available at the moment", "for a feature not yet implemented". Or, as another example, if a feature hasn't been completed, but the test for it is also taking an enormous amount of time, you might xfail it and also skip it.
-- You won't be able to tell from the test cases below, but if skip and xfail are both applied, the test case is skipped, not xfailed (which makes sense). The order does not matter.

[case skipfail1-skip-xfail]
x: int = 1

[case skipfail2-xfail-skip]
x: int = 1

[case skipfail0]
x: int = 1

[case skipfail3-skip]
x: int = 1

[case skipfail4-xfail]
x: int = "1"