Testing that expressions fail type-checking: a xfail counterpart to reveal_type

[rsokl]

This post is highly relevant to the Q&A at #2163, but that discussion gets at capabilities that are beyond what I am looking for; hopefully it is still appropriate for me to post this here. Please let me know if you'd like me to move this post to that thread.

I am hoping to check that an improper use of a function gets flagged by pyright, and to include this in my automated testing process.

Currently, I am only testing that my functions are producing the right types; I do so using reveal_type:

from my_library import sqrt

# valid input and expected output type
out: Literal["float"]  = reveal_type(sqrt(22))

This has worked great for me thus far. I love the ergonomics of reveal_type: getting to see if things pass right in VSCode, while also being able to leverage pyright's CLI in my automated tests has been great. I am especially grateful for this in light of the heavy machinery other libraries have stood up to validate their types (e.g. Hypothesis' infrastructure for mypy tests).

What I would like is a utility comparable to reveal_type, but something that says "I expect pyright to mark this as invalid". Perhaps it would be called xfail. And I would use it as

out = xfail(sqrt("hello"))  # pyright CLI reports error if `sqrt("hello")` is seen as valid

I realize that this sort of function would not cover the full range of functionality that was requested in #2163 , nor does it enable users to verify what the reported error is. That being said, it would make this sort of contrapositive testing possible and exceedingly simple.

I'd love to hear peoples' thoughts. Again, I am happy to move this to #2163 if it would be appropriate.

[erictraut]

Is it sufficient to test for "some error was reported", or do want to be more specific and verify that the correct error was reported and for the correct text range? I would think that you'd want the latter. If so, then my recommendation is to run the pyright CLI with --outputjson option and write a script to verify that the JSON output (which includes a full list of errors) matches the expected output.

[jakebailey]

If it's useful, I could add an option to the action that makes it not fail/comment on errors, and then puts the JSON output as an action output.

FWIW TS has @ts-expect-error.

[rsokl]

@jakebailey I appreciate the offer to implement that. I need to think it over for a bit to know if I would end up using that feature - I wouldn't want you to use your time on it if I didn't.

[jakebailey]

It wouldn't be all that difficult; I had already been thinking about making pyright-action able to run the type verifier and write its output to a file or something, so this is in the same vein. Let me know, no problem.

[rsokl]

That would be great, then! I do think I would leverage that.

[jakebailey]

Sorry for the delay on this, but I don't know if I'll have time to get to this in short order, given holidays and the fact that I switched teams (so this would be in my off time).

If you're still interested and haven't come up with an alternative in the meantime, feel free to file an issue on https://github.com/jakebailey/pyright-action so I don't forget; I'm still assuming that what you want is the action to not fail the step and instead output to a file (though the former can be done with continue-on-error, I think). That'd still be a unique mode, though, so I'm not 100% sure what difference it'd have with a plain npx pyright --outputjson > something.json if the step has to be special anyway; maybe you are wanting it to still show the errors on the PR or something...

[rsokl]

A follow up to this for people reading this threads looking for solutions to this. It turns out that you can configure pyright to raise on unnecessary # type: ignore comments:

Add to pyrightconfig.json:

"reportUnnecessaryTypeIgnoreComment": true,

This provides the exact mechanism that I am looking for to do contrapositive testing (i.e., testing guard against false negatives). Given the above configuration

out = sqrt("hello")  # type: ignore

This will pass only if the type-checker sees a problem with this line.