Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

INTERNAL ERROR when using mypy with pre-commit (all versions) #18579

Open
TravisHardman opened this issue Jan 31, 2025 · 10 comments
Open

INTERNAL ERROR when using mypy with pre-commit (all versions) #18579

TravisHardman opened this issue Jan 31, 2025 · 10 comments
Labels

Comments

@TravisHardman
Copy link

Crash Report

Mypy was working fine on my django project until about a week ago, when I started getting this error. I'm not sure what caused it. My python code used to be in the root directory of my project and now I moved it to the backend/ directory, and the problem started around then, but I'm not sure whether that is related.

I am using pre-commit and have mypy set up in my .pre-commit-config.yaml. The error actually only happens when I run mypy as part of pre-commit. If I just run mypy . from the command line, it works fine.

Presently, whenever I run pre-commit, I get an INTERNAL ERROR (details below).

Traceback

/home/travis/Repos/bethpage/backend/bethpage/models.py:23: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 1.16.0+dev.23e2d0f8cbaa2f4874d8f6a6bee3756922c78407
Traceback (most recent call last):
  File "/home/travis/Repos/bethpage/.tox/py312/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/__main__.py", line 15, in console_entry
    main()
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/main.py", line 119, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/main.py", line 203, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/build.py", line 191, in build
    result = _build(
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/build.py", line 267, in _build
    graph = dispatch(sources, manager, stdout)
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/build.py", line 2937, in dispatch
    process_graph(graph, manager)
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/build.py", line 3335, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/build.py", line 3430, in process_stale_scc
    mypy.semanal_main.semantic_analysis_for_scc(graph, scc, manager.errors)
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/semanal_main.py", line 94, in semantic_analysis_for_scc
    process_functions(graph, scc, patches)
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/semanal_main.py", line 252, in process_functions
    process_top_level_function(
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/semanal_main.py", line 291, in process_top_level_function
    deferred, incomplete, progress = semantic_analyze_target(
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/semanal_main.py", line 351, in semantic_analyze_target
    analyzer.refresh_partial(
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/semanal.py", line 655, in refresh_partial
    self.accept(node)
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/semanal.py", line 7254, in accept
    node.accept(self)
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/nodes.py", line 811, in accept
    return visitor.visit_func_def(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/semanal.py", line 913, in visit_func_def
    self.analyze_func_def(defn)
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/semanal.py", line 958, in analyze_func_def
    self.defer(defn)
  File "/home/travis/Repos/bethpage/.tox/py312/lib/python3.12/site-packages/mypy/semanal.py", line 6900, in defer
    assert not self.final_iteration, "Must not defer during final iteration"
           ^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Must not defer during final iteration
/home/travis/Repos/bethpage/backend/bethpage/models.py:23: : note: use --pdb to drop into pdb

To Reproduce

This crash happens any time I change any file in my backend/ directory and then run git add -A; pre-commit.

It does not happen if I just run cd backend; mypy .

If I use the --no-incremental flag or delete the cache directory, the problem still persists.

If I open up backend/bethpage/models.py and add a line that just says "pass" somewhere above line 23 (the line cited in the error), then run git add -A; pre-commit, it actually works fine and completes without errors. The next time I need to commit something, I'll get the error again, but I can just remove that "pass" line and once again, the error goes away.

Here is the beginning of that /backend/bethpage/models.py file:

import uuid
from typing import (
    Any,
    ClassVar,
    Dict,
    Generic,
    Iterable,
    List,
    Mapping,
    Optional,
    Self,
    Tuple,
    TypeVar,
)

from django.db import models, transaction
from django.db.models.fields.reverse_related import ForeignObjectRel
from django.utils import timezone

T = TypeVar("T", bound="BaseModel")


class BaseQuerySet(models.QuerySet[T], Generic[T]):

    def bulk_create(self, objs: Iterable[T], *args: Any, **kwargs: Any) -> List[T]:
        for obj in objs:
            obj.full_clean()
        return super().bulk_create(objs, *args, **kwargs)
...etc...

Your Environment

  • Mypy version used: I have tried versions of mypy from 1.0.0 to 1.14.1 and have also tried using the master branch on github, always with the same error.
  • Python version used: 3.12
  • Operating system: Arch Linux

.pre-commit-config.yaml

repos:
  - repo: local
    hooks:
      - id: mypy
        name: mypy
        entry: mypy
        language: system
        types: [python]
        args:
          - --config-file=backend/mypy.ini
          - --show-traceback
...etc...

mypy.ini

[mypy]
python_version = 3.12
mypy_path = backend
plugins =
  mypy_django_plugin.main,
  mypy_drf_plugin.main
strict = True
show_error_codes = True
disallow_untyped_calls = True
disallow_untyped_defs = True
disallow_incomplete_defs = True
disallow_any_unimported = True
check_untyped_defs = True
no_implicit_optional = True
warn_unused_ignores = True
warn_return_any = True
warn_unreachable = True

[mypy.plugins.django-stubs]
django_settings_module = bethpage.settings
@A5rocks
Copy link
Collaborator

A5rocks commented Feb 1, 2025

What happens if you disable the plugins?

And does pre-commit run -a fail?

@TravisHardman
Copy link
Author

pre-commit run -a does not work. I get the same error I have in the description above.

Disabling the plugins does work, and mypy no longer returns an error. Specifically, disabling the mypy_django_plugin.main causes it to work. I can keep the mypy_drf_plugin.main plugin and there are no issues.

I updated django-stubs and django-stubs-ext to the latest version but the problem persists. Of course, I do kind of need the django-stubs plugin for my project to work with mypy.

@A5rocks
Copy link
Collaborator

A5rocks commented Feb 3, 2025

If you run mypy <list of files> --config-file=backend/mypy.ini --show-traceback that should also crash, I think. Could you check?

cc @sobolevn

@TravisHardman
Copy link
Author

Actually no. It still works when running mypy directly.

(py312) [travis@bananas bethpage]$ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        modified:   backend/bethpage/settings.py

(py312) [travis@bananas bethpage]$ git add -A
(py312) [travis@bananas bethpage]$ pre-commit
mypy.....................................................................Failed
- hook id: mypy
- exit code: 2

/home/travis/Repos/bethpage/backend/bethpage/models.py:24: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
If this issue continues with mypy master, please report a bug at https://github.com/python/mypy/issues
version: 1.14.1
/home/travis/Repos/bethpage/backend/bethpage/models.py:24: : note: please use --show-traceback to print a traceback when reporting a bug

black....................................................................Passed
isort....................................................................Passed
flake8...................................................................Passed
codespell................................................................Passed
ESLint...............................................(no files to check)Skipped
Prettier.............................................(no files to check)Skipped
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed

At this point I made another change in backend/bethpage/settings.py to make sure to bypass any cache created from the prior step, then continued:

(py312) [travis@bananas bethpage]$ git add -A
(py312) [travis@bananas bethpage]$ mypy backend/ --config-file=backend/mypy.ini --show-traceback
Success: no issues found in 80 source files

@A5rocks
Copy link
Collaborator

A5rocks commented Feb 3, 2025

Sorry just to be explicit, I mean passing mypy a list of .py files. IIRC the way pre-commit works is it does that, so I assume that's the cause of the crash. Such as, find -name "*.py" | xargs mypy --config-file=backend/mypy.ini --show-traceback. (maybe I'm wrong about how pre-commit works)

@sobolevn
Copy link
Member

sobolevn commented Feb 3, 2025

Does it happen without django-stubs enabled?

@TravisHardman
Copy link
Author

@A5rocks The issue still does not happen when passing specific files to mypy. The problem does happen with pre-commit, but then I run directly with the list of files and get:

(py312) [travis@bananas bethpage]$ find backend/ -name "*.py" | xargs mypy --config-file=backend/mypy.ini --show-traceback
Success: no issues found in 80 source files

So it is still limited only to pre-commit as far as I can tell.

@sobolevn When I remove django-stubs, the error does not happen, even with pre-commit. It does seem to be tied specifically to pre-commit and specifically to django-stubs. For clarity, earlier post where I checked that: #18579 (comment)

@sterliakov
Copy link
Collaborator

This is indeed likely specific to django-stubs, but has nothing to do with pre-commit.

Full script:

cd "$(mktemp -d)"
uv venv --python 3.12 && . ./.venv/bin/activate
uv pip install mypy django django-stubs

mkdir -p backend/bethpage
touch backend/bethpage/__init__.py

cat <<EOF > backend/bethpage/models.py
from typing import Generic, TypeVar

from django.db import models

T = TypeVar("T", bound="BaseModel")

class BaseQuerySet(models.QuerySet[T], Generic[T]): pass
EOF

cat <<EOF > backend/bethpage/settings.py
from pathlib import Path

BASE_DIR = Path(__file__).resolve().parent.parent
INSTALLED_APPS = []
EOF

cat <<EOF > backend/mypy.ini
[mypy]
python_version = 3.12
mypy_path = backend
plugins =
  mypy_django_plugin.main
strict = True

[mypy.plugins.django-stubs]
django_settings_module = bethpage.settings
EOF

mypy backend --config-file backend/mypy.ini --show-traceback

Out:

backend/bethpage/settings.py:4: error: Need type annotation for "INSTALLED_APPS" (hint: "INSTALLED_APPS: list[<type>] = ...")  [var-annotated]
backend/bethpage/models.py:5: error: Name "BaseModel" is not defined  [name-defined]
backend/bethpage/models.py:7: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 1.14.1
Traceback (most recent call last):
  File "mypy/semanal.py", line 7286, in accept
  File "mypy/nodes.py", line 827, in accept
  File "mypy/semanal.py", line 907, in visit_func_def
  File "mypy/semanal.py", line 952, in analyze_func_def
  File "mypy/semanal.py", line 6940, in defer
AssertionError: Must not defer during final iteration
backend/bethpage/models.py:7: : note: use --pdb to drop into pdb

@sterliakov
Copy link
Collaborator

The core of the problem is in unresolved quoted name in typevar bound: replacing BaseModel with something defined or removing the surrounding quotes prevents the crash.

@sobolevn
Copy link
Member

sobolevn commented Feb 4, 2025

PRs to django-stubs are welcome :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants