Add a command for running complete type checking with nox -s typecheck (including vendored dependencies)

[pelson]

In #13476 @notatallshaw did a lot of work to improve the typing situation in the codebase. Unfortunately that PR got quite big, and was ultimately closed.

During the development of that PR, I proposed an additional change which would trick mypy into learning about the full type information of the vendored dependencies (notatallshaw#4). That merge request meant that much of the if TYPE_CHECKING tricks from #13476 would not be necessary, reducing the size of required change as well as reducing the duplication that the approach presented (and the risk of the two paths diverging).

This PR is a targeted change to allow a nox -s typecheck command to run a complete mypy call with full dependency context. I have intentionally not tried to change anything in the codebase, including the existing (very limited) mypy config nor actual type problems. I figure that knowing about the type issues is the first step to resolving them.

There are a good number of follow-on activities from this PR, including making use of dependency groups, reusing the vendored dependency definitions from vendor.txt, fixing the typing issues, and adapting the CI systematically validate against this. I would propose not doing that in this PR to allow us to move forwards more easily (and ultimately, this PR doesn't change anything about the existing repository infra).

The trick that I'm playing in this PR is to make a second pip package containing the _vendor stubs which I show to mypy along with the real ./src/pip via MYPYPATH. With the appropriate exclusions in the mypy config it then combines the two together to build the full context of the types used in ./src/pip.

To demonstrate that this is doing what we expect/want, with this change I include the complete output with and without building requests stubs for pip._vendor.requests. There is a lot of output, but the key observation is that with requests stubs, we are informed of potentially important information, for example:

src/pip/_internal/network/auth.py:440: error: Return type "Request" of "__call__" incompatible with return type "PreparedRequest" in supertype "pip._vendor.requests.auth.AuthBase"  [override]

The line in question:

pip/src/pip/_internal/network/auth.py

Line 440 in ed05518

def __call__(self, req: Request) -> Request:

Is a genuine issue with the type annotation. It is entirely missed if we disable the generation of hints for pip._vendor.requests:

Result when running against ./src/pip/_internal/network/session.py

Running: nox -s typecheck -- ./src/pip/_internal/network/session.py

nox > Running session typecheck
nox > Re-using existing virtual environment at .nox/typecheck.
nox > python tools/protected_pip.py install mypy types-PyYAML keyring nox packaging pytest httpx rich
nox > python tools/protected_pip.py install --target=tests/typing/untracked-vendored-stubs types-docutils types-requests 'types-urllib3==1.*' types-setuptools types-six types-PyYAML
nox > stubgen src/pip/__init__.py tests/typing/untracked-vendored-stubs/pip_vendor.py --output tests/typing/untracked-vendored-stubs
Processed 2 modules
Generated files under tests/typing/untracked-vendored-stubs/
nox > mypy --config-file=tests/typing/pyproject.toml ./src/pip/_internal/network/session.py
src/pip/_internal/utils/compat.py:24: error: Module "pip._vendor.urllib3.util" has no attribute "IS_PYOPENSSL"  [attr-defined]
src/pip/_internal/utils/compat.py:26: error: Returning Any from function declared to return "bool"  [no-any-return]
src/pip/_internal/exceptions.py:314: error: Incompatible types in assignment (expression has type "PreparedRequest", variable has type "Request | None")  [assignment]
src/pip/_internal/locations/_sysconfig.py:71: error: Returning Any from function declared to return "str"  [no-any-return]
src/pip/_internal/locations/_sysconfig.py:90: error: Returning Any from function declared to return "str"  [no-any-return]
src/pip/_internal/locations/_sysconfig.py:105: error: Returning Any from function declared to return "str"  [no-any-return]
src/pip/_internal/locations/__init__.py:139: error: Returning Any from function declared to return "bool"  [no-any-return]
src/pip/_internal/utils/misc.py:551: error: Returning Any from function declared to return "bool"  [no-any-return]
src/pip/_internal/metadata/pkg_resources.py:68: error: Returning Any from function declared to return "Iterable[str]"  [no-any-return]
src/pip/_internal/metadata/pkg_resources.py:128: error: Argument "metadata" to "DistInfoDistribution" has incompatible type "InMemoryMetadata"; expected "IResourceProvider | None"  [arg-type]
src/pip/_internal/metadata/pkg_resources.py:149: error: Argument "metadata" to "DistInfoDistribution" has incompatible type "InMemoryMetadata"; expected "IResourceProvider | None"  [arg-type]
src/pip/_internal/metadata/pkg_resources.py:171: error: Returning Any from function declared to return "str | None"  [no-any-return]
src/pip/_internal/metadata/pkg_resources.py:179: error: Item "IResourceProvider" of "IResourceProvider | EmptyProvider" has no attribute "path"  [union-attr]
src/pip/_internal/metadata/pkg_resources.py:179: error: Item "EmptyProvider" of "IResourceProvider | EmptyProvider" has no attribute "path"  [union-attr]
src/pip/_internal/metadata/pkg_resources.py:193: error: Returning Any from function declared to return "str"  [no-any-return]
src/pip/_internal/metadata/pkg_resources.py:196: error: Returning Any from function declared to return "bool"  [no-any-return]
src/pip/_internal/metadata/pkg_resources.py:208: error: Returning Any from function declared to return "str"  [no-any-return]
src/pip/_internal/metadata/pkg_resources.py:244: error: Returning Any from function declared to return "Iterable[Requirement]"  [no-any-return]
src/pip/_internal/req/req_file.py:142: error: Returning Any from function declared to return "str | None"  [no-any-return]
src/pip/_internal/network/auth.py:440: error: Return type "Request" of "__call__" incompatible with return type "PreparedRequest" in supertype "pip._vendor.requests.auth.AuthBase"  [override]
src/pip/_internal/network/auth.py:440: error: Argument 1 of "__call__" is incompatible with supertype "pip._vendor.requests.auth.AuthBase"; supertype defines the argument type as "PreparedRequest"  [override]
src/pip/_internal/network/auth.py:440: note: This violates the Liskov substitution principle
src/pip/_internal/network/auth.py:440: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
src/pip/_internal/commands/__init__.py:125: error: Returning Any from function declared to return "Command"  [no-any-return]
src/pip/_internal/commands/__init__.py:137: error: Returning Any from function declared to return "str | None"  [no-any-return]
src/pip/_internal/operations/install/wheel.py:424: error: Returning Any from function declared to return "list[str]"  [no-any-return]
src/pip/_internal/cli/autocompletion.py:152: error: Returning Any from function declared to return "str | None"  [no-any-return]
src/pip/_internal/network/session.py:54: error: Cannot find implementation or library stub for module named "pip._vendor.urllib3.proxymanager"  [import-not-found]
src/pip/_internal/network/session.py:54: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
src/pip/_internal/network/session.py:215: error: Argument 3 of "send" is incompatible with supertype "pip._vendor.requests.adapters.BaseAdapter"; supertype defines the argument type as "float | tuple[float, float] | tuple[float, None] | None"  [override]
src/pip/_internal/network/session.py:215: note: This violates the Liskov substitution principle
src/pip/_internal/network/session.py:215: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
src/pip/_internal/network/session.py:217: error: Argument 5 of "send" is incompatible with supertype "pip._vendor.requests.adapters.BaseAdapter"; supertype defines the argument type as "bytes | str | tuple[bytes | str, bytes | str] | None"  [override]
src/pip/_internal/network/session.py:217: note: This violates the Liskov substitution principle
src/pip/_internal/network/session.py:217: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
src/pip/_internal/network/session.py:220: error: Argument 1 to "url_to_path" has incompatible type "str | None"; expected "str"  [arg-type]
src/pip/_internal/network/session.py:224: error: Incompatible types in assignment (expression has type "str | None", variable has type "str")  [assignment]
src/pip/_internal/network/session.py:238: error: Argument 1 to "CaseInsensitiveDict" has incompatible type "dict[str, object]"; expected "Mapping[str, str] | Iterable[tuple[str, str]] | None"  [arg-type]
src/pip/_internal/network/session.py:246: error: Cannot assign to a method  [method-assign]
src/pip/_internal/network/session.py:280: error: Returning Any from function declared to return "PoolManager"  [no-any-return]
src/pip/_internal/network/session.py:280: note: Error code "no-any-return" not covered by "type: ignore" comment
src/pip/_internal/network/session.py:353: error: Incompatible types in assignment (expression has type "MultiDomainBasicAuth", variable has type "tuple[str, str] | Callable[[PreparedRequest], PreparedRequest] | None")  [assignment]
src/pip/_internal/network/session.py:397: error: Incompatible types in assignment (expression has type "HTTPAdapter", variable has type "CacheControlAdapter")  [assignment]
src/pip/_internal/network/session.py:398: error: Incompatible types in assignment (expression has type "InsecureHTTPAdapter", variable has type "InsecureCacheControlAdapter")  [assignment]
src/pip/_internal/network/session.py:414: error: Item "tuple[str, ...]" of "tuple[str, str] | Callable[[PreparedRequest], PreparedRequest] | None" has no attribute "index_urls"  [union-attr]
src/pip/_internal/network/session.py:414: error: Item "function" of "tuple[str, str] | Callable[[PreparedRequest], PreparedRequest] | None" has no attribute "index_urls"  [union-attr]
src/pip/_internal/network/session.py:414: error: Item "None" of "tuple[str, str] | Callable[[PreparedRequest], PreparedRequest] | None" has no attribute "index_urls"  [union-attr]
src/pip/_internal/network/session.py:521: error: Signature of "request" incompatible with supertype "pip._vendor.requests.sessions.Session"  [override]
src/pip/_internal/network/session.py:521: note:      Superclass:
src/pip/_internal/network/session.py:521: note:          def request(self, method: str | bytes, url: str | bytes, params: SupportsItems[str | bytes | int | float, str | bytes | int | float | Iterable[str | bytes | int | float] | None] | tuple[str | bytes | int | float, str | bytes | int | float | Iterable[str | bytes | int | float] | None] | Iterable[tuple[str | bytes | int | float, str | bytes | int | float | Iterable[str | bytes | int | float] | None]] | str | bytes | None = ..., data: Iterable[bytes] | str | bytes | SupportsRead[str | bytes] | list[tuple[Any, Any]] | tuple[tuple[Any, Any], ...] | Mapping[Any, Any] | None = ..., headers: Mapping[str, str | bytes | None] | None = ..., cookies: RequestsCookieJar | MutableMapping[str, str] | None = ..., files: Mapping[str, SupportsRead[str | bytes] | str | bytes | tuple[str | None, SupportsRead[str | bytes] | str | bytes] | tuple[str | None, SupportsRead[str | bytes] | str | bytes, str] | tuple[str | None, SupportsRead[str | bytes] | str | bytes, str, Mapping[str, str]]] | Iterable[tuple[str, SupportsRead[str | bytes] | str | bytes | tuple[str | None, SupportsRead[str | bytes] | str | bytes] | tuple[str | None, SupportsRead[str | bytes] | str | bytes, str] | tuple[str | None, SupportsRead[str | bytes] | str | bytes, str, Mapping[str, str]]]] | None = ..., auth: tuple[str, str] | AuthBase | Callable[[PreparedRequest], PreparedRequest] | None = ..., timeout: float | tuple[float | None, float | None] | None = ..., allow_redirects: bool = ..., proxies: MutableMapping[str, str] | None = ..., hooks: Mapping[str, Iterable[Callable[[Response], Any]] | Callable[[Response], Any]] | None = ..., stream: bool | None = ..., verify: bool | str | None = ..., cert: str | tuple[str, str] | None = ..., json: Any | None = ...) -> Response
src/pip/_internal/network/session.py:521: note:      Subclass:
src/pip/_internal/network/session.py:521: note:          def request(self, method: str, url: str, *args: Any, **kwargs: Any) -> Response
src/pip/_internal/index/collector.py:83: error: Argument 2 to "_NotAPIContent" has incompatible type "str | None"; expected "str"  [arg-type]
Found 41 errors in 13 files (checked 1 source file)
nox > Command mypy --config-file=tests/typing/pyproject.toml ./src/pip/_internal/network/session.py failed with exit code 1
nox > Session typecheck failed.

Result when running against ./src/pip/_internal/network/session.py **without type hint information about pip._vendor.requests**

Running: nox -s typecheck -- ./src/pip/_internal/network/session.py

nox > Running session typecheck
nox > Re-using existing virtual environment at .nox/typecheck.
nox > python tools/protected_pip.py install mypy types-PyYAML keyring nox packaging pytest httpx rich
nox > python tools/protected_pip.py install --target=tests/typing/untracked-vendored-stubs types-docutils 'types-urllib3==1.*' types-setuptools types-six types-PyYAML
nox > stubgen src/pip/__init__.py tests/typing/untracked-vendored-stubs/pip_vendor.py --output tests/typing/untracked-vendored-stubs
Processed 2 modules
Generated files under tests/typing/untracked-vendored-stubs/
nox > mypy --config-file=tests/typing/pyproject.toml ./src/pip/_internal/network/session.py
src/pip/_internal/utils/compat.py:24: error: Module "pip._vendor.urllib3.util" has no attribute "IS_PYOPENSSL"  [attr-defined]
src/pip/_internal/utils/compat.py:26: error: Returning Any from function declared to return "bool"  [no-any-return]
src/pip/_internal/locations/_sysconfig.py:71: error: Returning Any from function declared to return "str"  [no-any-return]
src/pip/_internal/locations/_sysconfig.py:90: error: Returning Any from function declared to return "str"  [no-any-return]
src/pip/_internal/locations/_sysconfig.py:105: error: Returning Any from function declared to return "str"  [no-any-return]
src/pip/_internal/locations/__init__.py:139: error: Returning Any from function declared to return "bool"  [no-any-return]
src/pip/_internal/utils/misc.py:551: error: Returning Any from function declared to return "bool"  [no-any-return]
src/pip/_internal/network/utils.py:43: error: Incompatible types in assignment (expression has type "Any | None", variable has type "str")  [assignment]
src/pip/_internal/network/utils.py:45: error: Unsupported operand types for <= ("int" and "None")  [operator]
src/pip/_internal/network/utils.py:45: note: Right operand is of type "Any | None"
src/pip/_internal/network/utils.py:45: error: Unsupported operand types for > ("int" and "None")  [operator]
src/pip/_internal/network/utils.py:45: note: Left operand is of type "Any | None"
src/pip/_internal/network/utils.py:50: error: Unsupported operand types for <= ("int" and "None")  [operator]
src/pip/_internal/network/utils.py:50: note: Right operand is of type "Any | None"
src/pip/_internal/network/utils.py:50: error: Unsupported operand types for > ("int" and "None")  [operator]
src/pip/_internal/network/utils.py:50: note: Left operand is of type "Any | None"
src/pip/_internal/network/utils.py:65: error: Item "None" of "Any | None" has no attribute "stream"  [union-attr]
src/pip/_internal/network/utils.py:95: error: Item "None" of "Any | None" has no attribute "read"  [union-attr]
src/pip/_internal/metadata/pkg_resources.py:68: error: Returning Any from function declared to return "Iterable[str]"  [no-any-return]
src/pip/_internal/metadata/pkg_resources.py:128: error: Argument "metadata" to "DistInfoDistribution" has incompatible type "InMemoryMetadata"; expected "IResourceProvider | None"  [arg-type]
src/pip/_internal/metadata/pkg_resources.py:149: error: Argument "metadata" to "DistInfoDistribution" has incompatible type "InMemoryMetadata"; expected "IResourceProvider | None"  [arg-type]
src/pip/_internal/metadata/pkg_resources.py:171: error: Returning Any from function declared to return "str | None"  [no-any-return]
src/pip/_internal/metadata/pkg_resources.py:179: error: Item "IResourceProvider" of "IResourceProvider | EmptyProvider" has no attribute "path"  [union-attr]
src/pip/_internal/metadata/pkg_resources.py:179: error: Item "EmptyProvider" of "IResourceProvider | EmptyProvider" has no attribute "path"  [union-attr]
src/pip/_internal/metadata/pkg_resources.py:193: error: Returning Any from function declared to return "str"  [no-any-return]
src/pip/_internal/metadata/pkg_resources.py:196: error: Returning Any from function declared to return "bool"  [no-any-return]
src/pip/_internal/metadata/pkg_resources.py:208: error: Returning Any from function declared to return "str"  [no-any-return]
src/pip/_internal/metadata/pkg_resources.py:244: error: Returning Any from function declared to return "Iterable[Requirement]"  [no-any-return]
src/pip/_internal/req/req_file.py:142: error: Returning Any from function declared to return "str | None"  [no-any-return]
src/pip/_internal/network/auth.py:375: error: Returning Any from function declared to return "tuple[str | None, str | None]"  [no-any-return]
src/pip/_internal/network/auth.py:489: error: Argument 1 to "_get_new_credentials" of "MultiDomainBasicAuth" has incompatible type "Any | None"; expected "str"  [arg-type]
src/pip/_internal/network/auth.py:503: error: Argument 1 to "_prompt_for_password" of "MultiDomainBasicAuth" has incompatible type "bytes"; expected "str"  [arg-type]
src/pip/_internal/network/auth.py:508: error: Invalid index type "bytes" for "dict[str, tuple[str | None, str | None]]"; expected type "str"  [index]
src/pip/_internal/network/auth.py:513: error: Argument "url" to "Credentials" has incompatible type "bytes"; expected "str"  [arg-type]
src/pip/_internal/network/auth.py:523: error: Item "None" of "Any | None" has no attribute "release_conn"  [union-attr]
src/pip/_internal/network/auth.py:536: error: "Response" has no attribute "connection"  [attr-defined]
src/pip/_internal/network/auth.py:539: error: Returning Any from function declared to return "Response"  [no-any-return]
src/pip/_internal/network/auth.py:546: error: Item "None" of "Any | None" has no attribute "url"  [union-attr]
src/pip/_internal/network/auth.py:557: error: Unsupported operand types for > ("int" and "None")  [operator]
src/pip/_internal/network/auth.py:557: note: Left operand is of type "Any | None"
src/pip/_internal/commands/__init__.py:125: error: Returning Any from function declared to return "Command"  [no-any-return]
src/pip/_internal/commands/__init__.py:137: error: Returning Any from function declared to return "str | None"  [no-any-return]
src/pip/_internal/operations/install/wheel.py:424: error: Returning Any from function declared to return "list[str]"  [no-any-return]
src/pip/_internal/cli/autocompletion.py:152: error: Returning Any from function declared to return "str | None"  [no-any-return]
src/pip/_internal/network/session.py:54: error: Cannot find implementation or library stub for module named "pip._vendor.urllib3.proxymanager"  [import-not-found]
src/pip/_internal/network/session.py:54: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
src/pip/_internal/network/session.py:220: error: Argument 1 to "url_to_path" has incompatible type "Any | None"; expected "str"  [arg-type]
src/pip/_internal/network/session.py:246: error: Cannot assign to a method  [method-assign]
src/pip/_internal/network/session.py:280: error: Returning Any from function declared to return "PoolManager"  [no-any-return]
src/pip/_internal/network/session.py:280: note: Error code "no-any-return" not covered by "type: ignore" comment
src/pip/_internal/network/session.py:397: error: Incompatible types in assignment (expression has type "HTTPAdapter", variable has type "CacheControlAdapter")  [assignment]
src/pip/_internal/network/session.py:398: error: Incompatible types in assignment (expression has type "InsecureHTTPAdapter", variable has type "InsecureCacheControlAdapter")  [assignment]
src/pip/_internal/network/session.py:414: error: Item "None" of "Any | None" has no attribute "index_urls"  [union-attr]
src/pip/_internal/network/session.py:528: error: Returning Any from function declared to return "Response"  [no-any-return]
src/pip/_internal/network/download.py:135: error: Argument 1 to "splitext" has incompatible type "Any | None"; expected "PathLike[Any]"  [arg-type]
src/pip/_internal/network/download.py:342: error: Returning Any from function declared to return "Response"  [no-any-return]
src/pip/_internal/index/collector.py:83: error: Item "None" of "Any | None" has no attribute "method"  [union-attr]
src/pip/_internal/index/collector.py:168: error: Returning Any from function declared to return "Response"  [no-any-return]
src/pip/_internal/index/collector.py:314: error: Argument "url" to "IndexContent" has incompatible type "Any | None"; expected "str"  [arg-type]
Found 52 errors in 14 files (checked 1 source file)
nox > Command mypy --config-file=tests/typing/pyproject.toml ./src/pip/_internal/network/session.py failed with exit code 1
nox > Session typecheck failed.

[notatallshaw]

Thanks for the PR, I do agree it's a good idea to set this up before fixing type hints. I will add reviewing this approach to my review list.