Merge branch 'main' into enum_spec

.pre-commit-config.yaml

@@ -0,0 +1,24 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
+    hooks:
+      - id: trailing-whitespace
+        exclude: conformance/results/.*/.*\.toml
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-toml
+      - id: check-merge-conflict
+      - id: mixed-line-ending
+        args: [--fix=lf]
+        exclude: docs/make\.bat
+      - id: check-case-conflict
+  - repo: meta
+    hooks:
+      - id: check-hooks-apply
+
+ci:
+  autofix_commit_msg: "[pre-commit.ci] auto fixes from pre-commit.com hooks"
+  autofix_prs: true
+  autoupdate_commit_msg: "[pre-commit.ci] pre-commit autoupdate"
+  autoupdate_schedule: quarterly
+  submodules: false

conformance/README.md

@@ -20,6 +20,7 @@ This project contains test cases for behaviors defined in the Python typing spec
 * [literals](https://typing.readthedocs.io/en/latest/spec/literal.html)
 * [protocols](https://typing.readthedocs.io/en/latest/spec/protocol.html)
 * [callables](https://typing.readthedocs.io/en/latest/spec/callables.html)
+* [constructors](https://typing.readthedocs.io/en/latest/spec/constructors.html)
 * [overloads](https://typing.readthedocs.io/en/latest/spec/overload.html)
 * [dataclasses](https://typing.readthedocs.io/en/latest/spec/dataclasses.html)
 * [typeddicts](https://typing.readthedocs.io/en/latest/spec/typeddict.html)
@@ -40,6 +41,17 @@ Test cases are meant to be human readable. They should include comments that hel
 
 The test suite focuses on static type checking not general Python semantics. Tests should therefore focus on static analysis behaviors, not runtime behaviors.
 
+Test cases use the following conventions:
+
+* Lines that are expected to produce a type checker error should have a comment starting with # E",
+  either by itself or followed by an explanation after a colon (e.g., "# E: int is not a subtype
+  of str"). Such explanatory comments are purely for human understanding, but type checkers are not
+  expected to use their exact wording.
+* Lines that may produce an error (e.g., because the spec allows multiple behaviors) should be
+  marked with "# E?" instead of "# E".
+* If a test case tests conformance with a specific passage in the spec, that passage should be
+  quoted in a comment prefixed with "# > ".
+
 ## Running the Conformance Test Tool
 
 To run the conformance test suite:
@@ -54,7 +66,7 @@ Note that some type checkers may not run on some platforms. For example, pytype
 
 Different type checkers report errors in different ways (with different wording in error messages and different line numbers or character ranges for errors). This variation makes it difficult to fully automate test validation given that tests will want to check for both false positive and false negative type errors. Some level of manual inspection will therefore be needed to determine whether a type checker is fully conformant with all tests in any given test file. This "scoring" process is required only when the output of a test changes — e.g. when a new version of that type checker is released and the tests are rerun. We assume that the output of a type checker will be the same from one run to the next unless/until a new version is released that fixes or introduces a bug. In this case, the output will need to be manually inspected and the conformance results re-scored for those tests whose output has changed.
 
-Conformance results are reported and summarized for each supported type checker. Initially, results will be reported for mypy and pyright. It is the goal and desire to add additional type checkers over time.
+Conformance results are reported and summarized for each supported type checker. Currently, results are reported for mypy, pyre, pyright, and pytype. It is the goal and desire to add additional type checkers over time.
 
 ## Adding a New Test Case
 
@@ -68,6 +80,22 @@ If a test is updated (augmented or fixed), the process is similar to when adding
 
 If a new version of a type checker is released, re-run the test tool with the new version. If the type checker output has changed for any test cases, the tool will supply the old and new outputs. Examine these to determine whether the conformance status has changed. Once the conformance status has been updated, re-run the test tool again to regenerate the summary report.
 
+## Automated Conformance Checking
+
+In addition to manual scoring, we provide an experimental tool that automatically checks type checkers for conformance. This tool relies on the "# E" comments present in the stubs and on parsing type checker output. This logic is run automatically as part of the conformance test tool. It produces the following fields in the `.toml` output files:
+
+* `errors_diff`: a string describing all issues found with the type checker's behavior: either expected errors that were not emitted, or extra errors that the conformance test suite does not allow.
+* `conformance_automated`: either "Pass" or "Fail" based on whether there are any discrepancies with the expected behavior.
+
+This tool does not yet work reliably on all test cases. The script `conformance/src/unexpected_fails.py` can be run to find all test cases where the automated tool's conformance judgment differs from the manual judgment entered in the `.toml` files.
+
+Some common problems with automated checks:
+
+* Sometimes the spec is imprecise or allows multiple options. In this case, use "# E?" to mark an error as optional.
+* Type checkers may produce additional errors for issues unrelated to the topic being tested. In this case, add an extra field `ignore_errors` in the type checker's `.toml` file that contains the text of the irrelevant errors. Any error message that contains a substring in the `ignore_errors` list is ignored. For example, if `ignore_errors = ["Too many arguments"]`, then a mypy error `dataclasses_usage.py:127: error: Too many arguments for "DC7"  [call-arg]` will be ignored.
+* Type checkers may differ in the line on which they report an error. In this case, on each of the lines where an error could
+  reasonably be shown, write `# E[<tag>]`, where `<tag>` is an arbitrary string that is unique in the file. The test will be marked as passing if the type checker produces an error on exactly one of the lines where this tag appears.
+
 ## Contributing
 
 Contributions are welcome!

conformance/requirements.txt

@@ -5,4 +5,3 @@ pyright
 mypy
 pyre-check
 pytype; platform_system != "Windows"
-

conformance/results/mypy/aliases_explicit.toml

@@ -3,10 +3,10 @@ notes = """
 Does not reject specialization of type alias that has already been implicitly specialized.
 """
 output = """
-aliases_explicit.py:67: error: Bad number of arguments for type alias, expected: 0, given: 1  [type-arg]
-aliases_explicit.py:68: error: Bad number of arguments for type alias, expected: 0, given: 1  [type-arg]
-aliases_explicit.py:69: error: Bad number of arguments for type alias, expected: 1, given: 2  [type-arg]
-aliases_explicit.py:70: error: Bad number of arguments for type alias, expected: 1, given: 2  [type-arg]
+aliases_explicit.py:67: error: Bad number of arguments for type alias, expected 0, given 1  [type-arg]
+aliases_explicit.py:68: error: Bad number of arguments for type alias, expected 0, given 1  [type-arg]
+aliases_explicit.py:69: error: Bad number of arguments for type alias, expected 1, given 2  [type-arg]
+aliases_explicit.py:70: error: Bad number of arguments for type alias, expected 1, given 2  [type-arg]
 aliases_explicit.py:71: error: Can only replace ParamSpec with a parameter types list or another ParamSpec, got "int"  [misc]
 aliases_explicit.py:79: error: Invalid type alias: expression is not a valid type  [valid-type]
 aliases_explicit.py:80: error: Bracketed expression "[...]" is not valid as a type  [valid-type]
@@ -24,5 +24,12 @@ aliases_explicit.py:90: error: Invalid type alias: expression is not a valid typ
 aliases_explicit.py:90: error: Function "list" could always be true in boolean context  [truthy-function]
 aliases_explicit.py:91: error: Invalid type alias: expression is not a valid type  [valid-type]
 aliases_explicit.py:101: error: "<typing special form>" not callable  [operator]
-aliases_explicit.py:102: error: Bad number of arguments for type alias, expected: 0, given: 1  [type-arg]
+aliases_explicit.py:102: error: Bad number of arguments for type alias, expected 0, given 1  [type-arg]
 """
+conformance_automated = "Fail"
+errors_diff = """
+Line 100: Expected 1 errors
+"""
+ignore_errors = [
+    'Function "list" could always be true in boolean context',
+]

conformance/results/mypy/aliases_implicit.toml

@@ -1,9 +1,9 @@
 conformant = "Pass"
 output = """
-aliases_implicit.py:76: error: Bad number of arguments for type alias, expected: 0, given: 1  [type-arg]
-aliases_implicit.py:77: error: Bad number of arguments for type alias, expected: 0, given: 1  [type-arg]
-aliases_implicit.py:78: error: Bad number of arguments for type alias, expected: 1, given: 2  [type-arg]
-aliases_implicit.py:79: error: Bad number of arguments for type alias, expected: 1, given: 2  [type-arg]
+aliases_implicit.py:76: error: Bad number of arguments for type alias, expected 0, given 1  [type-arg]
+aliases_implicit.py:77: error: Bad number of arguments for type alias, expected 0, given 1  [type-arg]
+aliases_implicit.py:78: error: Bad number of arguments for type alias, expected 1, given 2  [type-arg]
+aliases_implicit.py:79: error: Bad number of arguments for type alias, expected 1, given 2  [type-arg]
 aliases_implicit.py:80: error: Can only replace ParamSpec with a parameter types list or another ParamSpec, got "int"  [misc]
 aliases_implicit.py:81: error: Type argument "str" of "GoodTypeAlias12" must be a subtype of "float"  [type-var]
 aliases_implicit.py:100: error: Function "list" could always be true in boolean context  [truthy-function]
@@ -36,5 +36,9 @@ aliases_implicit.py:118: note: See https://mypy.readthedocs.io/en/stable/common_
 aliases_implicit.py:119: error: Variable "aliases_implicit.BadTypeAlias14" is not valid as a type  [valid-type]
 aliases_implicit.py:119: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
 aliases_implicit.py:133: error: "<typing special form>" not callable  [operator]
-aliases_implicit.py:135: error: Bad number of arguments for type alias, expected: 0, given: 1  [type-arg]
+aliases_implicit.py:135: error: Bad number of arguments for type alias, expected 0, given 1  [type-arg]
 """
+conformance_automated = "Pass"
+errors_diff = """
+"""
+ignore_errors = ["Function \"list\" could always be true in boolean context"]

conformance/results/mypy/aliases_newtype.toml

@@ -6,8 +6,6 @@ aliases_newtype.py:20: error: Cannot use isinstance() with NewType type  [misc]
 aliases_newtype.py:23: error: Cannot subclass "NewType"  [misc]
 aliases_newtype.py:32: error: String argument 1 "BadName" to NewType(...) does not match variable name "GoodName"  [misc]
 aliases_newtype.py:38: error: "GoodNewType1" expects no type arguments, but 1 given  [type-arg]
-aliases_newtype.py:41: error: Cannot redefine "GoodNewType2" as a NewType  [misc]
-aliases_newtype.py:41: error: Name "GoodNewType2" already defined on line 36  [no-redef]
 aliases_newtype.py:44: error: Argument 2 to NewType(...) must be subclassable (got "int | str")  [valid-newtype]
 aliases_newtype.py:47: error: Type variable "aliases_newtype.T" is unbound  [valid-type]
 aliases_newtype.py:47: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class)
@@ -18,3 +16,6 @@ aliases_newtype.py:58: error: Argument 2 to NewType(...) must be subclassable (g
 aliases_newtype.py:60: error: NewType(...) expects exactly two positional arguments  [misc]
 aliases_newtype.py:62: error: Argument 2 to NewType(...) must be subclassable (got "Any")  [valid-newtype]
 """
+conformance_automated = "Pass"
+errors_diff = """
+"""

conformance/results/mypy/aliases_recursive.toml

@@ -6,9 +6,12 @@ aliases_recursive.py:38: error: Incompatible types in assignment (expression has
 aliases_recursive.py:39: error: Name "t6" already defined on line 38  [no-redef]
 aliases_recursive.py:50: error: Dict entry 0 has incompatible type "str": "list[int]"; expected "str": "str | int | Mapping[str, RecursiveMapping]"  [dict-item]
 aliases_recursive.py:51: error: Dict entry 2 has incompatible type "str": "list[int]"; expected "str": "str | int | Mapping[str, RecursiveMapping]"  [dict-item]
-aliases_recursive.py:55: error: Dict entry 2 has incompatible type "str": "list[int]"; expected "str": "str | int | Mapping[str, RecursiveMapping]"  [dict-item]
-aliases_recursive.py:67: error: List item 0 has incompatible type "float"; expected "GenericTypeAlias1[str] | str"  [list-item]
-aliases_recursive.py:73: error: List item 0 has incompatible type "float"; expected "GenericTypeAlias2[str, int] | str | int"  [list-item]
-aliases_recursive.py:76: error: Invalid recursive alias: a union item of itself  [misc]
-aliases_recursive.py:81: error: Invalid recursive alias: a union item of itself  [misc]
+aliases_recursive.py:52: error: Dict entry 2 has incompatible type "str": "list[int]"; expected "str": "str | int | Mapping[str, RecursiveMapping]"  [dict-item]
+aliases_recursive.py:63: error: List item 0 has incompatible type "float"; expected "GenericTypeAlias1[str] | str"  [list-item]
+aliases_recursive.py:69: error: List item 0 has incompatible type "float"; expected "GenericTypeAlias2[str, int] | str | int"  [list-item]
+aliases_recursive.py:72: error: Invalid recursive alias: a union item of itself  [misc]
+aliases_recursive.py:75: error: Invalid recursive alias: a union item of itself  [misc]
+"""
+conformance_automated = "Pass"
+errors_diff = """
 """

conformance/results/mypy/aliases_type_statement.toml

@@ -74,3 +74,24 @@ aliases_type_statement.py:90: error: Cannot resolve name "RecursiveTypeAlias7" (
 aliases_type_statement.py:90: error: Name "RecursiveTypeAlias7" is used before definition  [used-before-def]
 aliases_type_statement.py:91: error: PEP 695 type aliases are not yet supported  [valid-type]
 """
+conformance_automated = "Fail"
+errors_diff = """
+Line 17: Expected 1 errors
+Line 19: Expected 1 errors
+Line 26: Expected 1 errors
+Line 31: Expected 1 errors
+Line 8: Unexpected errors ['aliases_type_statement.py:8: error: PEP 695 type aliases are not yet supported  [valid-type]']
+Line 9: Unexpected errors ['aliases_type_statement.py:9: error: PEP 695 type aliases are not yet supported  [valid-type]', 'aliases_type_statement.py:9: error: Name "S3" is not defined  [name-defined]', 'aliases_type_statement.py:9: error: Name "S1" is not defined  [name-defined]', 'aliases_type_statement.py:9: error: Name "S2" is not defined  [name-defined]']
+Line 10: Unexpected errors ['aliases_type_statement.py:10: error: PEP 695 type aliases are not yet supported  [valid-type]', 'aliases_type_statement.py:10: error: Value of type "UnionType" is not indexable  [index]']
+Line 14: Unexpected errors ['aliases_type_statement.py:14: error: PEP 695 type aliases are not yet supported  [valid-type]']
+Line 21: Unexpected errors ['aliases_type_statement.py:21: error: "type[int]" has no attribute "__value__"  [attr-defined]']
+Line 54: Unexpected errors ['aliases_type_statement.py:54: error: PEP 695 type aliases are not yet supported  [valid-type]']
+Line 72: Unexpected errors ['aliases_type_statement.py:72: error: PEP 695 type aliases are not yet supported  [valid-type]', 'aliases_type_statement.py:72: error: Name "T" is not defined  [name-defined]', 'aliases_type_statement.py:72: error: Variable "aliases_type_statement.RecursiveTypeAlias1" is not valid as a type  [valid-type]']
+Line 74: Unexpected errors ['aliases_type_statement.py:74: error: Variable "aliases_type_statement.RecursiveTypeAlias1" is not valid as a type  [valid-type]']
+Line 75: Unexpected errors ['aliases_type_statement.py:75: error: Variable "aliases_type_statement.RecursiveTypeAlias1" is not valid as a type  [valid-type]']
+Line 77: Unexpected errors ['aliases_type_statement.py:77: error: PEP 695 type aliases are not yet supported  [valid-type]', 'aliases_type_statement.py:77: error: Name "P" is not defined  [name-defined]', 'aliases_type_statement.py:77: error: Name "T" is not defined  [name-defined]', 'aliases_type_statement.py:77: error: Name "S" is not defined  [name-defined]', 'aliases_type_statement.py:77: error: Variable "aliases_type_statement.RecursiveTypeAlias2" is not valid as a type  [valid-type]']
+Line 80: Unexpected errors ['aliases_type_statement.py:80: error: Unexpected "..."  [misc]', 'aliases_type_statement.py:80: error: Variable "aliases_type_statement.RecursiveTypeAlias2" is not valid as a type  [valid-type]']
+Line 82: Unexpected errors ['aliases_type_statement.py:82: error: Bracketed expression "[...]" is not valid as a type  [valid-type]', 'aliases_type_statement.py:82: error: Variable "aliases_type_statement.RecursiveTypeAlias2" is not valid as a type  [valid-type]']
+Line 88: Unexpected errors ['aliases_type_statement.py:88: error: PEP 695 type aliases are not yet supported  [valid-type]', 'aliases_type_statement.py:88: error: Name "T" is not defined  [name-defined]', 'aliases_type_statement.py:88: error: Variable "aliases_type_statement.RecursiveTypeAlias5" is not valid as a type  [valid-type]']
+Line 91: Unexpected errors ['aliases_type_statement.py:91: error: PEP 695 type aliases are not yet supported  [valid-type]']
+"""