[BUG] Pydantic V2.11 Seems to cause up to x2.5 memory usage to ninja models build

[M3te0r]

Describe the bug
A clear and concise description of what the bug is.

Hi, during a dependencies upgrade work, upgrading both ninja and pydantic to last version (1.4, 2.11)

I've seen up to x2.5 in terms of memory allocation, in my case 1.2 GB, with pydantic 2.10 I was at 510 MB, at this point I don't know if it's ninja or pydantic istelf but i've seen that pydantic._internal._model_construction.complete_model_class / pydantic.plugin._schema_validator.create_schema_validator are memory intensive

A bit weird because 2.11 claim is to reduce memory allocation..

I've used memray to have some flamegraph representation:

Before (2.10):

After (2.11)

Versions (please complete the following information):

• Python version: 3.13
• Django version: 5.0.14
• Django-Ninja version: 1.4.1
• Pydantic version: 2.10/2.11

Note you can quickly get this by runninng in ./manage.py shell this line:

import django; import pydantic; import ninja; django.__version__; ninja.__version__; pydantic.__version__

[vitalik]

HI @M3te0r

hm.. well I do not remember any intrusion into models/schema in ninja 1.3->1.4

I guess you have no choices but run a matrix test to measure

Ninja	Pydantic	usage
1.3	2.10
1.3	2.11
1.4	2.10
1.4	2.11

I think it's pretty safe to switch ninja 1.3/1.4 back and force (same for pydantic as I did not detected any deprecations or errors)

[M3te0r]

Something definitely weird with pydantic 2.11, ran the matrix

Code is exactly the same between run, used python manage.py check for usage reference as url checks runs the whole api/schemas/models stuff

Ninja	Pydantic	usage
1.3	2.10	Peak memory usage: 497.3 MiB
1.3	2.11	Peak memory usage: 1.2 GiB
1.4	2.10	Peak memory usage: 497.3 MiB
1.4	2.11	Peak memory usage: 1.2 GiB

Details (same order):

Ninja 1.3/Pydantic 2.10.6

 
 
 (nova-py3.13)  m3te0r@m3te0r-ThinkPad-E15-Gen-2  ~/PycharmProjects/bepatient-server   NEXTG-8674 ±  python -c "import django; import pydantic; import ninja; print(django.__version__); print(ninja.__version__); print(pydantic.__version__)"
5.0.14
1.3.0
2.10.6
(nova-py3.13)  m3te0r@m3te0r-ThinkPad-E15-Gen-2  ~/PycharmProjects/bepatient-server   NEXTG-8674 ±  memray run manage.py check                                                                                                
Writing profile results into memray-manage.py.421105.bin
⚠ Memray support for Greenlet is experimental ⚠
Please report any issues at https://github.com/bloomberg/memray/issues


INFO 2025-04-11 23:35:18,694 [None] [None-None] apps 421105 129488432196480 AXES: BEGIN version 7.0.2, blocking by username
System check identified no issues (2 silenced).
[memray] Successfully generated profile results.

You can now generate reports from the stored allocation records.
Some example commands to generate reports:

/home/m3te0r/PycharmProjects/bepatient-server/.venv/bin/python -m memray flamegraph memray-manage.py.421105.bin
(nova-py3.13)  m3te0r@m3te0r-ThinkPad-E15-Gen-2  ~/PycharmProjects/bepatient-server   NEXTG-8674 ±  memray summary ./memray-manage.py.421105.bin                                                                                           
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Location                                                                                                              ┃         <Total Memory> ┃         Total Memory % ┃             Own Memory ┃          Own Memory % ┃      Allocation Count ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩
│ run_path at <frozen runpy>                                                                                            │              473.764MB │                 95.27% │                 0.000B │                 0.00% │               1243102 │
│ _run_tracker at                                                                                                       │              473.764MB │                 95.27% │                 0.000B │                 0.00% │               1243102 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/memray/commands/run.py               │                        │                        │                        │                       │                       │
│ <module> at manage.py                                                                                                 │              473.749MB │                 95.27% │                 0.000B │                 0.00% │               1243085 │
│ _run_code at <frozen runpy>                                                                                           │              473.749MB │                 95.27% │                 0.000B │                 0.00% │               1243085 │
│ _run_module_code at <frozen runpy>                                                                                    │              473.749MB │                 95.27% │                 0.000B │                 0.00% │               1243085 │
│ _find_and_load_unlocked at <frozen importlib._bootstrap>                                                              │              473.536MB │                 95.23% │               42.969KB │                 0.01% │               1242917 │
│ _find_and_load at <frozen importlib._bootstrap>                                                                       │              473.536MB │                 95.23% │                 0.000B │                 0.00% │               1242917 │
│ exec_module at <frozen importlib._bootstrap_external>                                                                 │              473.525MB │                 95.23% │                 0.000B │                 0.00% │               1242908 │
│ _load_unlocked at <frozen importlib._bootstrap>                                                                       │              473.525MB │                 95.23% │              101.359KB │                 0.02% │               1242908 │
│ _call_with_frames_removed at <frozen importlib._bootstrap>                                                            │              473.395MB │                 95.20% │                1.270MB │                 0.26% │               1242743 │
│ execute at                                                                                                            │              470.982MB │                 94.71% │                 0.000B │                 0.00% │               1242282 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/__init__.py   │                        │                        │                        │                       │                       │
│ execute_from_command_line at                                                                                          │              470.982MB │                 94.71% │                 0.000B │                 0.00% │               1242282 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/__init__.py   │                        │                        │                        │                       │                       │
│ _gcd_import at <frozen importlib._bootstrap>                                                                          │              459.544MB │                 92.41% │                 0.000B │                 0.00% │               1224273 │
│ import_module at /home/m3te0r/.pyenv/versions/3.13.0/lib/python3.13/importlib/__init__.py                             │              459.544MB │                 92.41% │                 0.000B │                 0.00% │               1224273 │
│ __new__ at                                                                                                            │              375.362MB │                 75.49% │              158.969KB │                 0.03% │               1170140 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/pydantic/_internal/_model_construct… │                        │                        │                        │                       │                       │
│ complete_model_class at                                                                                               │              368.669MB │                 74.14% │               87.308MB │                17.56% │               1166916 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/pydantic/_internal/_model_construct… │                        │                        │                        │                       │                       │
│ __new__ at /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/ninja/schema.py           │              361.194MB │                 72.64% │               30.000KB │                 0.01% │               1128628 │
│ cached_import at                                                                                                      │              339.727MB │                 68.32% │                 0.000B │                 0.00% │                988186 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/utils/module_loading.py       │                        │                        │                        │                       │                       │
│ import_string at                                                                                                      │              339.727MB │                 68.32% │                 0.000B │                 0.00% │                988186 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/utils/module_loading.py       │                        │                        │                        │                       │                       │
│ __get__ at                                                                                                            │              334.359MB │                 67.24% │              116.172KB │                 0.02% │                986625 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/utils/functional.py           │                        │                        │                        │                       │                       │
│ run_from_argv at                                                                                                      │              333.590MB │                 67.08% │                 0.000B │                 0.00% │                986800 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/base.py       │                        │                        │                        │                       │                       │
│ run_checks at                                                                                                         │              333.590MB │                 67.08% │                 0.000B │                 0.00% │                986799 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/checks/registry.py       │                        │                        │                        │                       │                       │
│ check at                                                                                                              │              333.590MB │                 67.08% │                 0.000B │                 0.00% │                986799 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/base.py       │                        │                        │                        │                       │                       │
│ handle at                                                                                                             │              333.590MB │                 67.08% │                 0.000B │                 0.00% │                986799 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/commands/che… │                        │                        │                        │                       │                       │
│ execute at                                                                                                            │              333.590MB │                 67.08% │                 0.000B │                 0.00% │                986799 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/base.py       │                        │                        │                        │                       │                       │
│ check_url_namespaces_unique at                                                                                        │              333.235MB │                 67.01% │                 0.000B │                 0.00% │                986460 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/checks/urls.py           │                        │                        │                        │                       │                       │
│ <module> at /home/m3te0r/PycharmProjects/bepatient-server/config/urls.py                                              │              333.235MB │                 67.01% │                 0.000B │                 0.00% │                986459 │
│ urlconf_module at                                                                                                     │              333.235MB │                 67.01% │                 0.000B │                 0.00% │                986459 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/urls/resolvers.py             │                        │                        │                        │                       │                       │
│ url_patterns at                                                                                                       │              333.235MB │                 67.01% │                 0.000B │                 0.00% │                986459 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/urls/resolvers.py             │                        │                        │                        │                       │                       │
│ _load_all_namespaces at                                                                                               │              333.235MB │                 67.01% │                 0.000B │                 0.00% │                986459 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/checks/urls.py           │                        │                        │                        │                       │                       │
│ include at /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/urls/conf.py       │              333.224MB │                 67.01% │                 0.000B │                 0.00% │                986444 │
│ <module> at /home/m3te0r/PycharmProjects/bepatient-server/nova/api/urls.py                                            │              333.038MB │                 66.97% │                 0.000B │                 0.00% │                986221 │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────┴────────────────────────┴────────────────────────┴───────────────────────┴───────────────────────┘

Peak memory usage: 497.3 MiB

Ninja 1.3/Pydantic 2.11.3

 (nova-py3.13)  m3te0r@m3te0r-ThinkPad-E15-Gen-2  ~/PycharmProjects/bepatient-server   NEXTG-8674 ±  python -c "import django; import pydantic; import ninja; print(django.__version__); print(ninja.__version__); print(pydantic.__version__)"
5.0.14
1.3.0
2.11.3
(nova-py3.13)  m3te0r@m3te0r-ThinkPad-E15-Gen-2  ~/PycharmProjects/bepatient-server   NEXTG-8674 ±  memray run manage.py check                                                                                                
Writing profile results into memray-manage.py.421796.bin
⚠ Memray support for Greenlet is experimental ⚠
Please report any issues at https://github.com/bloomberg/memray/issues


INFO 2025-04-11 23:37:37,393 [None] [None-None] apps 421796 127102232263552 AXES: BEGIN version 7.0.2, blocking by username
System check identified no issues (2 silenced).
[memray] Successfully generated profile results.

You can now generate reports from the stored allocation records.
Some example commands to generate reports:

/home/m3te0r/PycharmProjects/bepatient-server/.venv/bin/python -m memray flamegraph memray-manage.py.421796.bin
(nova-py3.13)  m3te0r@m3te0r-ThinkPad-E15-Gen-2  ~/PycharmProjects/bepatient-server   NEXTG-8674 ±  memray summary ./memray-manage.py.421796.bin                                                                                           
⚠  This capture file is large and may take a long time to process  ⚠

Next time, consider using the `--aggregate` option to `memray run` to reduce the size of the file.
Check https://bloomberg.github.io/memray/run.html#aggregated-capture-files for more information.

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Location                                                                                                              ┃         <Total Memory> ┃         Total Memory % ┃             Own Memory ┃          Own Memory % ┃      Allocation Count ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩
│ run_path at <frozen runpy>                                                                                            │                1.115GB │                 95.67% │                 0.000B │                 0.00% │               3545883 │
│ _run_tracker at                                                                                                       │                1.115GB │                 95.67% │                 0.000B │                 0.00% │               3545883 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/memray/commands/run.py               │                        │                        │                        │                       │                       │
│ <module> at manage.py                                                                                                 │                1.115GB │                 95.67% │                 0.000B │                 0.00% │               3545868 │
│ _run_code at <frozen runpy>                                                                                           │                1.115GB │                 95.67% │                 0.000B │                 0.00% │               3545868 │
│ _run_module_code at <frozen runpy>                                                                                    │                1.115GB │                 95.67% │                 0.000B │                 0.00% │               3545868 │
│ _find_and_load_unlocked at <frozen importlib._bootstrap>                                                              │                1.114GB │                 95.57% │               36.797KB │                 0.00% │               3545704 │
│ _find_and_load at <frozen importlib._bootstrap>                                                                       │                1.114GB │                 95.57% │                 0.000B │                 0.00% │               3545704 │
│ _load_unlocked at <frozen importlib._bootstrap>                                                                       │                1.114GB │                 95.57% │              101.359KB │                 0.01% │               3545697 │
│ exec_module at <frozen importlib._bootstrap_external>                                                                 │                1.114GB │                 95.56% │                 0.000B │                 0.00% │               3545696 │
│ _call_with_frames_removed at <frozen importlib._bootstrap>                                                            │                1.113GB │                 95.47% │                2.273MB │                 0.19% │               3545524 │
│ execute at                                                                                                            │                1.113GB │                 95.44% │                 0.000B │                 0.00% │               3545094 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/__init__.py   │                        │                        │                        │                       │                       │
│ execute_from_command_line at                                                                                          │                1.113GB │                 95.44% │                 0.000B │                 0.00% │               3545094 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/__init__.py   │                        │                        │                        │                       │                       │
│ _gcd_import at <frozen importlib._bootstrap>                                                                          │                1.101GB │                 94.48% │                 0.000B │                 0.00% │               3532171 │
│ import_module at /home/m3te0r/.pyenv/versions/3.13.0/lib/python3.13/importlib/__init__.py                             │                1.101GB │                 94.48% │                 0.000B │                 0.00% │               3532171 │
│ __new__ at                                                                                                            │             1023.147MB │                 85.71% │                1.236MB │                 0.10% │               3416010 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/pydantic/_internal/_model_construct… │                        │                        │                        │                       │                       │
│ create_schema_validator at                                                                                            │             1016.751MB │                 85.17% │             1009.750MB │                84.59% │               3476493 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/pydantic/plugin/_schema_validator.py │                        │                        │                        │                       │                       │
│ complete_model_class at                                                                                               │             1011.389MB │                 84.72% │                6.335MB │                 0.53% │               3412685 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/pydantic/_internal/_model_construct… │                        │                        │                        │                       │                       │
│ __new__ at /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/ninja/schema.py           │             1008.622MB │                 84.49% │              330.750KB │                 0.03% │               3378729 │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────┴────────────────────────┴────────────────────────┴───────────────────────┴───────────────────────┘
(nova-py3.13)  m3te0r@m3te0r-ThinkPad-E15-Gen-2  ~/PycharmProjects/bepatient-server   NEXTG-8674 ±  

Peak memory usage: 1.2 GiB

Ninja 1.4.1/Pydantic 2.10.6

(nova-py3.13)  m3te0r@m3te0r-ThinkPad-E15-Gen-2  ~/PycharmProjects/bepatient-server   NEXTG-8674 ±  memray summary ./memray-manage.py.420438.bin                                                                                              
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Location                                                                                                              ┃         <Total Memory> ┃         Total Memory % ┃             Own Memory ┃          Own Memory % ┃      Allocation Count ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩
│ run_path at <frozen runpy>                                                                                            │              472.452MB │                 95.00% │                 0.000B │                 0.00% │               1227124 │
│ _run_tracker at                                                                                                       │              472.452MB │                 95.00% │                 0.000B │                 0.00% │               1227124 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/memray/commands/run.py               │                        │                        │                        │                       │                       │
│ <module> at manage.py                                                                                                 │              472.437MB │                 95.00% │                 0.000B │                 0.00% │               1227107 │
│ _run_code at <frozen runpy>                                                                                           │              472.437MB │                 95.00% │                 0.000B │                 0.00% │               1227107 │
│ _run_module_code at <frozen runpy>                                                                                    │              472.437MB │                 95.00% │                 0.000B │                 0.00% │               1227107 │
│ _find_and_load_unlocked at <frozen importlib._bootstrap>                                                              │              472.279MB │                 94.97% │               37.547KB │                 0.01% │               1227022 │
│ _find_and_load at <frozen importlib._bootstrap>                                                                       │              472.279MB │                 94.97% │                 0.000B │                 0.00% │               1227022 │
│ _load_unlocked at <frozen importlib._bootstrap>                                                                       │              472.268MB │                 94.96% │              101.359KB │                 0.02% │               1227013 │
│ exec_module at <frozen importlib._bootstrap_external>                                                                 │              472.169MB │                 94.94% │                 0.000B │                 0.00% │               1227012 │
│ _call_with_frames_removed at <frozen importlib._bootstrap>                                                            │              472.032MB │                 94.92% │                1.283MB │                 0.26% │               1226837 │
│ execute at                                                                                                            │              470.663MB │                 94.64% │                 0.000B │                 0.00% │               1226291 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/__init__.py   │                        │                        │                        │                       │                       │
│ execute_from_command_line at                                                                                          │              470.663MB │                 94.64% │                 0.000B │                 0.00% │               1226291 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/__init__.py   │                        │                        │                        │                       │                       │
│ _gcd_import at <frozen importlib._bootstrap>                                                                          │              459.366MB │                 92.37% │                 0.000B │                 0.00% │               1208393 │
│ import_module at /home/m3te0r/.pyenv/versions/3.13.0/lib/python3.13/importlib/__init__.py                             │              459.366MB │                 92.37% │                 0.000B │                 0.00% │               1208393 │
│ __new__ at                                                                                                            │              374.170MB │                 75.24% │                1.168MB │                 0.23% │               1154079 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/pydantic/_internal/_model_construct… │                        │                        │                        │                       │                       │
│ complete_model_class at                                                                                               │              364.375MB │                 73.27% │               87.815MB │                17.66% │               1150727 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/pydantic/_internal/_model_construct… │                        │                        │                        │                       │                       │
│ __new__ at /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/ninja/schema.py           │              352.829MB │                 70.95% │               29.250KB │                 0.01% │               1111826 │
│ cached_import at                                                                                                      │              335.880MB │                 67.54% │                 0.000B │                 0.00% │                978984 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/utils/module_loading.py       │                        │                        │                        │                       │                       │
│ import_string at                                                                                                      │              335.880MB │                 67.54% │                 0.000B │                 0.00% │                978984 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/utils/module_loading.py       │                        │                        │                        │                       │                       │
│ run_from_argv at                                                                                                      │              330.672MB │                 66.49% │                 0.000B │                 0.00% │                977503 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/base.py       │                        │                        │                        │                       │                       │
│ run_checks at                                                                                                         │              330.672MB │                 66.49% │                 0.000B │                 0.00% │                977502 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/checks/registry.py       │                        │                        │                        │                       │                       │
│ check at                                                                                                              │              330.672MB │                 66.49% │                 0.000B │                 0.00% │                977502 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/base.py       │                        │                        │                        │                       │                       │
│ handle at                                                                                                             │              330.672MB │                 66.49% │                 0.000B │                 0.00% │                977502 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/commands/che… │                        │                        │                        │                       │                       │
│ execute at                                                                                                            │              330.672MB │                 66.49% │                 0.000B │                 0.00% │                977502 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/base.py       │                        │                        │                        │                       │                       │
│ __get__ at                                                                                                            │              330.442MB │                 66.44% │               53.156KB │                 0.01% │                977328 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/utils/functional.py           │                        │                        │                        │                       │                       │
│ check_url_namespaces_unique at                                                                                        │              330.379MB │                 66.43% │                 0.000B │                 0.00% │                977249 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/checks/urls.py           │                        │                        │                        │                       │                       │
│ <module> at /home/m3te0r/PycharmProjects/bepatient-server/config/urls.py                                              │              330.379MB │                 66.43% │                 0.000B │                 0.00% │                977248 │
│ urlconf_module at                                                                                                     │              330.379MB │                 66.43% │                 0.000B │                 0.00% │                977248 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/urls/resolvers.py             │                        │                        │                        │                       │                       │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────┴────────────────────────┴────────────────────────┴───────────────────────┴───────────────────────┘

Peak memory usage: 497.3 MiB

Ninja 1.4.1/Pydantic 2.11.3

(nova-py3.13)  m3te0r@m3te0r-ThinkPad-E15-Gen-2  ~/PycharmProjects/bepatient-server   NEXTG-8674 ±  python -c "import django; import pydantic; import ninja; print(django.__version__); print(ninja.__version__); print(pydantic.__version__)"
5.0.14
1.4.1
2.11.3
(nova-py3.13)  m3te0r@m3te0r-ThinkPad-E15-Gen-2  ~/PycharmProjects/bepatient-server   NEXTG-8674 ±  memray run manage.py check                                                                                                         
Writing profile results into memray-manage.py.417618.bin
⚠ Memray support for Greenlet is experimental ⚠
Please report any issues at https://github.com/bloomberg/memray/issues


INFO 2025-04-11 23:15:58,599 [None] [None-None] apps 417618 129802299046784 AXES: BEGIN version 7.0.2, blocking by username
System check identified no issues (2 silenced).
[memray] Successfully generated profile results.

You can now generate reports from the stored allocation records.
Some example commands to generate reports:

(nova-py3.13)  m3te0r@m3te0r-ThinkPad-E15-Gen-2  ~/PycharmProjects/bepatient-server   NEXTG-8674 ±  memray summary ./memray-manage.py.417618.bin 
⚠  This capture file is large and may take a long time to process  ⚠

Next time, consider using the `--aggregate` option to `memray run` to reduce the size of the file.
Check https://bloomberg.github.io/memray/run.html#aggregated-capture-files for more information.

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Location                                                                                                              ┃         <Total Memory> ┃         Total Memory % ┃             Own Memory ┃          Own Memory % ┃      Allocation Count ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩
│ run_path at <frozen runpy>                                                                                            │                1.121GB │                 96.12% │                 0.000B │                 0.00% │               3544119 │
│ _run_tracker at                                                                                                       │                1.121GB │                 96.12% │                 0.000B │                 0.00% │               3544119 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/memray/commands/run.py               │                        │                        │                        │                       │                       │
│ <module> at manage.py                                                                                                 │                1.121GB │                 96.12% │                 0.000B │                 0.00% │               3544102 │
│ _run_code at <frozen runpy>                                                                                           │                1.121GB │                 96.12% │                 0.000B │                 0.00% │               3544102 │
│ _run_module_code at <frozen runpy>                                                                                    │                1.121GB │                 96.12% │                 0.000B │                 0.00% │               3544102 │
│ _find_and_load at <frozen importlib._bootstrap>                                                                       │                1.120GB │                 96.02% │                 0.000B │                 0.00% │               3543935 │
│ _find_and_load_unlocked at <frozen importlib._bootstrap>                                                              │                1.120GB │                 96.02% │                1.042MB │                 0.09% │               3543935 │
│ exec_module at <frozen importlib._bootstrap_external>                                                                 │                1.120GB │                 96.02% │                 0.000B │                 0.00% │               3543926 │
│ _load_unlocked at <frozen importlib._bootstrap>                                                                       │                1.120GB │                 96.02% │              101.359KB │                 0.01% │               3543926 │
│ _call_with_frames_removed at <frozen importlib._bootstrap>                                                            │                1.119GB │                 95.92% │              286.065KB │                 0.02% │               3543766 │
│ execute at                                                                                                            │                1.119GB │                 95.88% │                 0.000B │                 0.00% │               3543281 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/__init__.py   │                        │                        │                        │                       │                       │
│ execute_from_command_line at                                                                                          │                1.119GB │                 95.88% │                 0.000B │                 0.00% │               3543281 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/django/core/management/__init__.py   │                        │                        │                        │                       │                       │
│ _gcd_import at <frozen importlib._bootstrap>                                                                          │                1.107GB │                 94.90% │                 0.000B │                 0.00% │               3529802 │
│ import_module at /home/m3te0r/.pyenv/versions/3.13.0/lib/python3.13/importlib/__init__.py                             │                1.107GB │                 94.90% │                 0.000B │                 0.00% │               3529802 │
│ __new__ at                                                                                                            │                1.000GB │                 85.73% │              254.984KB │                 0.02% │               3412841 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/pydantic/_internal/_model_construct… │                        │                        │                        │                       │                       │
│ create_schema_validator at                                                                                            │             1019.391MB │                 85.32% │             1016.391MB │                85.07% │               3472141 │
│ /home/m3te0r/PycharmProjects/bepatient-server/.venv/lib/python3.13/site-packages/pydantic/plugin/_schema_validator.py │                        │                        │                        │                       │                       │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────┴────────────────────────┴────────────────────────┴───────────────────────┴───────────────────────┘

Peak memory usage: 1.2 GiB

[vitalik]

Ok, at least ninja is not 100% to blame :D

maybe you can extract some minimal code (few schemas, operations) where the difference is visible - I will be able then to talk something concrete with pydantic folks

[M3te0r]

@vitalik ATM I'm not able to make a minimal reproducible exemple (it may be noticeable on quite large and complicated schemas like my work project (discrimate annotated unions, inheritance etc), but :

• Tried to use 2.11 alpha/betas version
• https://github.com/pydantic/pydantic/releases/tag/v2.11.0a2 is OK
• https://github.com/pydantic/pydantic/releases/tag/v2.11.0b1 is NOK
• v2.11.0a2 uses pydantic-core 2.29.0
• v2.11.0b1 uses pydantic-core v2.31.1
• There was an attempt to reduce memory usage of validators (wrapped, after) in v2.11.0a2, but was removed in v2.11.0b1
  I think it's linked to these issues:

Original optimization : pydantic/pydantic-core#1616
Fix : pydantic/pydantic-core#1660

As ninja.Schema use a wrapped validator on _run_root_validator validators may me evaluated on each schemas, building up memory

Just for testing perpuposes, I commented out _run_root_validator and was at 147MB

Original code:

Commented out _run_root_validator:

[M3te0r]

@vitalik I managed to extract and reproduce a part of our code that increases memory usage with Pydantic v2.11.3

from decimal import Decimal
from typing import Annotated
from typing import Any
from typing import Literal
from typing import Union
from uuid import UUID

from django.db.models import TextChoices
from django.utils.translation import gettext as _
from ninja import Field
from ninja import NinjaAPI
from ninja import Router
from ninja import Schema

############## Basic ninja API declaration

api = NinjaAPI(title="Pydantic mem test API", version="1.0.0")
router = Router()
api.add_router("/test", router)


################### DJANGO ENUMS
class DocumentTypes(TextChoices):
    PLAN = "PLAN", _("PLAN")
    REPORT = "REPORT", _("REPORT")


class DocumentVisibility(TextChoices):
    HCP = "HCP", _("HCP")
    PATIENT = "PATIENT", _("PATIENT")
    BOTH = "BOTH", _("BOTH")


class DateScale(TextChoices):
    DAYS = "d", _("Day(s)")
    WEEKS = "w", _("Week(s)")
    MONTHS = "m", _("Month(s)")
    YEARS = "y", _("Year(s)")


class RecordsActivityStatus211(TextChoices):
    DRAFT = "DRAFT", _("Draft")
    PUBLISHED = "PUBLISHED", _("Published")


class DateFormat(TextChoices):
    YYYY_MM_DD = "YYYY-MM-DD", _("Year/Month/Day")
    YYYY_MM = "YYYY-MM", _("Year/Month")
    YYYY = "YYYY", _("Year")


class ContainerType(TextChoices):
    FORM = "Form", _("Form")
    CONTENT = "Content", _("Content")
    RECORDS = "Records", _("Records")
    DOCUMENT = "Document", _("Document")
    PATIENT_PROFILE = "PatientProfile", _("Patient profile")


class ElementType211(TextChoices):
    TEXT = "text", _("Text")
    TEXTAREA = "textarea", _("Textarea")
    INTEGER = "integer", _("Integer")
    NUMBER = "number", _("Number")
    BOOLEAN = "boolean", _("Boolean")
    CHOICES = "choices", _("Choices")
    DATE = "date", _("Date")
    DATETIME = "datetime", _("Datetime")
    TIME = "time", _("Time")
    RANGE = "range", _("Range")
    NRS_SLIDER = "nrs_slider", _("NRS Slider")
    EQ5DL = "eq5dl", _("EQ5DL")
    RICH = "rich", _("Rich text")
    LINK = "link", _("Link")
    PHONE = "phone", _("Phone")
    EMAIL = "email", _("Email")
    COUNTRY = "country", _("Country")
    FRENCH_SOCIAL_SECURITY_NUMBER = "french_social_security_number", _(
        "French Social Security Number",
    )
    FILE = "file", _("File")
    TABLE = "table", _("Table")
    REUSABLE = "reusable", _("Reusable")
    FIRST_NAME = "first_name", _("First name")
    LAST_NAME = "last_name", _("Last name")
    SITE = "site", _("Site")
    PATIENT_GROUP = "patient_group", _("Patient group")
    LANGUAGE = "language", _("Language")


################ SCHEMAS
class AllTranslationsOut(Schema):
    translations: dict[str, dict[str, Any | None]] | None = None


class ImageOut211(Schema):
    id: int
    image_url: str | None = None
    thumbnail_url: str | None = None
    filename: str | None = None
    title: str | None = None
    alt: str | None = None


class LightChoiceOut(Schema):
    id: int
    title: str | None = None
    index: int


class ChoiceOut(LightChoiceOut):

    href: str | None = None
    weighting: Decimal | None = None
    response_code: str | None = None


class LightContainerOut(Schema):
    id: int
    uuid: UUID
    title: str | None = None
    color: str
    href: str | None = None
    content_type: str
    pages_count: int
    index: int


# Build a list of concrete subclasses of BaseContainerOut to use as list spread
# when building discriminated annotated union NovaContainerElementOut
BASE_CONTAINER_OUT_SUBCLASSES = []


class BaseContainerOut(AllTranslationsOut, Schema):
    id: int
    uuid: UUID
    title: str | None = None
    label: str | None = None
    description: str | None = None
    index: int = Field(
        ...,
        description="This value indicate the sorting order of the element inside the page or the table, depending on the direct parent",
    )
    editable: bool
    movable: bool
    deletable: bool
    href: str | None = None

    # def __init_subclass__(cls, **kwargs):
    #     super().__init_subclass__(**kwargs)
    #     if not cls.__name__.startswith("Base"):
    #         BASE_CONTAINER_OUT_SUBCLASSES.append(cls)

    # Not the root cause (had to test in regard of __init_subclass__ pydantic comment)
    @classmethod
    def __pydantic_init_subclass__(cls, **kwargs: Any) -> None:
        super().__init_subclass__(**kwargs)
        if not cls.__name__.startswith("Base"):
            BASE_CONTAINER_OUT_SUBCLASSES.append(cls)

# TESTED with each subclass uncommented one-by-one
# Peak memory usage : 70.29MB
class RichTextContainerElementOut(BaseContainerOut):
    element_type: Literal[ElementType211.RICH]
    content: str

# Peak memory usage : 70.358MB
class BaseContainerElementOut(BaseContainerOut):
    tips: str | None = None
    placeholder: str = ""
    required: bool
    image: ImageOut211 | None = None
    validators: list[dict[str, Any]] = Field(
        ...,
        alias="validators_as_schemas",
        title="List of field validators (required, min/max/value/length)",
    )
    primary: bool | None = False
    question_code: str | None = None

# Peak memory usage : 79.337MB
class PatientGroupElementOut(BaseContainerElementOut):
    used_by_hcp: bool
    element_type: Literal[ElementType211.PATIENT_GROUP]

# Peak memory usage : 87.853MB
class IntegerContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.INTEGER]
    min_value: int | None = None
    max_value: int | None = None

# Peak memory usage : 97.706MB
class NumericContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.NUMBER]
    min_value: float | None = None
    max_value: float | None = None
    max_digits: int | None = None

# Peak memory usage : 106.228MB
class CharContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.TEXT]
    max_length: int | None = None

# Peak memory usage : 114.393MB
class FirstNameContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.FIRST_NAME]
    max_length: int | None = None

# Peak memory usage : 122.557MB
class LastNameContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.LAST_NAME]
    max_length: int | None = None

# Peak memory usage : 131.720MB
class TextContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.TEXTAREA]
    max_length: int | None = None

# Peak memory usage : 140.269MB
class EmailContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.EMAIL]

# Peak memory usage : 148.106MB
class FrenchSocialSecurityNumberContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.FRENCH_SOCIAL_SECURITY_NUMBER]

# Peak memory usage : 157.180MB
class FileContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.FILE]
    allow_multiple_files: bool

# Peak memory usage : 165.004MB
class PhoneContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.PHONE]

# Peak memory usage : 172.829MB
class CountryContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.COUNTRY]

# Peak memory usage : 181.647MB
class LanguageContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.LANGUAGE]

# Peak memory usage : 191.200MB
class DateContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.DATE]
    date_format: DateFormat
    min_bound_nb: int | None = None
    min_bound_scale: DateScale | None = None
    max_bound_nb: int | None = None
    max_bound_scale: DateScale | None = None

# Peak memory usage : 201.617MB
class DateTimeContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.DATETIME]
    min_bound_nb: int | None = None
    min_bound_scale: DateScale | None = None
    max_bound_nb: int | None = None
    max_bound_scale: DateScale | None = None

# Peak memory usage : 210.828MB
class BooleanContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.BOOLEAN]

# Peak memory usage : 220.020MB
class RangeContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.RANGE]
    min_value: int
    max_value: int
    step: int | None
    min_label: str = ""
    max_label: str = ""
    discrete: bool
    show_drag_input_value: bool

# Peak memory usage : 229.810MB
class NRSSliderContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.NRS_SLIDER]
    min_value: int
    max_value: int
    step: int
    min_label: str = ""
    max_label: str = ""

# Peak memory usage : 239.102MB
class EQ5DLContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.EQ5DL]
    min_value: int
    max_value: int
    min_label: str | None = None
    max_label: str | None = None
    display_value: bool
    use_validated_slide: bool
    step: int

# Peak memory usage : 250.360MB
class ChoicesContainerElementOut(BaseContainerElementOut):
    element_type: Literal[ElementType211.CHOICES]
    multiple: bool
    sorted_by_alpha: bool
    options: list[ChoiceOut]

# Peak memory usage : 256.160MB
class LinkContainerElementOut(BaseContainerOut):
    element_type: Literal[ElementType211.LINK]
    container: LightContainerOut | None = None


NovaContainerElementOut = Annotated[
    Union[*BASE_CONTAINER_OUT_SUBCLASSES],
    Field(discriminator="element_type"),
]


class PageOut211(Schema):
    id: int
    uuid: UUID
    index: int
    href: str | None = None
    title: str | None = None


class NovaPageOut211(PageOut211):
    elements: list[NovaContainerElementOut]


class ContainerOut211(Schema):
    id: int
    uuid: UUID
    # pages: list[NovaPageOut211]
    title: str | None = None
    color: str
    description: str | None = None
    href: str | None = None
    content_type: str
    index: int
    scoring_activated: bool = False
    summary_title: str | None = None
    summary_content: str | None = None


class NovaContainerOut211(AllTranslationsOut, ContainerOut211):
    container_type: (
        Literal[ContainerType.FORM]
        | Literal[ContainerType.CONTENT]
        | Literal[ContainerType.PATIENT_PROFILE]
    )
    image: ImageOut211 | None = None
    used_in_active_workflow: bool | None = False
    pages: list[NovaPageOut211]


class DocumentContainerOut211(AllTranslationsOut, ContainerOut211):
    container_type: Literal[ContainerType.DOCUMENT]
    type: DocumentTypes | None
    visibility: DocumentVisibility | None
    pages: list[NovaPageOut211]


class RecordsContainerOut211(AllTranslationsOut, Schema):
    id: int
    uuid: UUID
    title: str | None = None
    color: str
    description: str | None = None
    href: str | None = None
    content_type: str
    container_type: Literal[ContainerType.RECORDS]
    index: int
    scoring_activated: bool = False
    summary_title: str | None = None
    summary_content: str | None = None
    image: ImageOut211 | None = None
    pages: list[NovaPageOut211]


ContainerOutUnion211 = Annotated[
    Union[NovaContainerOut211, DocumentContainerOut211, RecordsContainerOut211],
    Field(discriminator="container_type"),
]

############ FAKE API OPERATIONS DEFINITION

def fake_view_get(request):
    pass


def fake_view_post(request):
    pass


def fake_view_patch(request):
    pass


fake_operations = (
    ("GET", fake_view_get, 200),
    ("POST", fake_view_post, 201),
    ("PATCH", fake_view_post, 200),
)

for i in range(20):
    for method, view_func, response_code in fake_operations:
        router.add_api_operation(
            f"/fake_{i}/activities/",
            [method],
            view_func,
            response={
                response_code: ContainerOutUnion211,
            },
            summary=f"Test summary {i}",
        )

With Pydantic v2.10.6

Peak memory usage: 174.4 MiB

With Pydantic v2.11.3

Peak memory usage: 257.3 MiB

Maybe this have someting to do with nested tagged unions and schemas inheritance ?

EDIT 1 : Updated snippet with comment about memory usage uncommenting one by one each BaseContainerOut sublass
As we can see, each subclass adds up to ~10MB even smallest ones with only the discriminator element_type

[M3te0r]

Hi @vitalik did you get the time to check this ?