segfaults with swig4.3 and PYTHONFAULTHANDLER=1 #2565

dweindl · 2024-10-24T09:25:48Z

scripts/run-python-tests.sh: line 22: 7262 Segmentation fault: 11 pytest --ignore-glob=*petab* --ignore-glob=*test_splines.py --durations=10 $@

Happens randomly. So far, twice this week and only on macos. Unclear.

Being updated:

The text was updated successfully, but these errors were encountered:

Set PYTHONFAULTHANDLER=1 for more output on crashes (AMICI-dev#2565). So far, this was only set for Ubuntu tests.

dweindl · 2024-10-24T09:36:38Z

Another crash, now for python/examples/example_splines_swameye//ExampleSplinesSwameye2003.ipynb on Ubuntu: https://github.com/AMICI-dev/AMICI/actions/runs/11495925466/job/31996499560?pr=2563 :-/

May or may not be related.

dweindl · 2024-10-24T13:54:40Z

This happens already during import of the amici-extension. I would currently guess that this is related to some external changes (runner image, brew packages, ...)

https://github.com/dweindl/AMICI/actions/runs/11500013663/job/32009264065:

Run scripts/run-python-tests.sh \
Fatal Python error: Segmentation fault

Current thread 0x00000001ef9a0f40 (most recent call first):
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1233 in create_module
  File "<frozen importlib._bootstrap>", line 573 in module_from_spec
  File "<frozen importlib._bootstrap>", line 676 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1232 in _handle_fromlist
  File "/Users/runner/work/AMICI/AMICI/python/sdist/amici/amici.py", line 28 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1232 in _handle_fromlist
  File "/Users/runner/work/AMICI/AMICI/python/sdist/amici/__init__.py", line 104 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/Users/runner/work/AMICI/AMICI/python/tests/conftest.py", line 8 in <module>
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/assertion/rewrite.py", line 184 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1204 in _gcd_import
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126 in import_module
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/pathlib.py", line 582 in import_path
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 710 in _importconftest
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 659 in _loadconftestmodules
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 619 in _try_load_conftest
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 581 in _set_initial_conftests
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1222 in pytest_load_initial_conftests
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1398 in _preparse
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1494 in parse
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1140 in pytest_cmdline_parse
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 341 in _prepareconfig
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 156 in main
  File "/Users/runner/work/AMICI/AMICI/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/Users/runner/work/AMICI/AMICI/venv/bin/pytest", line 8 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg (total: 2)
scripts/run-python-tests.sh: line 22: 25340 Segmentation fault: 11  pytest -s --ignore-glob=*petab* --ignore-glob=*test_splines.py --durations=10 $@
Error: Process completed with exit code 139.

dweindl · 2024-10-24T13:59:18Z

I would currently guess that this is related to some external changes (runner image, brew packages, ...)

Different jobs from the same workflow run:

https://github.com/dweindl/AMICI/actions/runs/11500013663/job/32009264899
2024-10-24T13:11:11.1000130Z Image: macos-14-arm64
2024-10-24T13:11:11.1000630Z Version: 20241014.301
PASS
https://github.com/dweindl/AMICI/actions/runs/11500013663/job/32009264065
2024-10-24T13:11:08.5866110Z Image: macos-14-arm64
2024-10-24T13:11:08.5866450Z Version: 20241022.361
FAIL

💩

FFroehlich · 2024-10-24T14:39:33Z

💩

Set PYTHONFAULTHANDLER=1 for more output on crashes (#2565). So far, this was only set for Ubuntu tests.

FFroehlich · 2024-10-27T19:07:39Z

I now get pretty consistent segfaults on the mac workflow in #1861 (comment)

dweindl · 2024-11-05T17:00:43Z

Not quite sure which of those changes could explain those issues -- xcode?

https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20241022.361

actions/runner-images@macos-14-arm64/20241014.301...macos-14-arm64/20241022.361

dweindl · 2024-11-05T17:41:42Z

The changes in the runner-images could just be a coincidence. There also was a new swig release around the same time (https://github.com/swig/swig/releases/tag/v4.3.0).

Recent changes in dependencies:
https://ghcr.io/v2/homebrew/core/swig/manifests/4.2.1 -> https://ghcr.io/v2/homebrew/core/swig/manifests/4.3.0
https://ghcr.io/v2/homebrew/core/libomp/manifests/19.1.1 -> https://ghcr.io/v2/homebrew/core/libomp/manifests/19.1.2

further changes: numpy, cmake

dweindl · 2024-11-05T18:27:44Z

It seems to be about swig 4.3.0. I can reproduce that on Ubuntu.

Python crashes specifically when importing amici under pytest. Otherwise it works fine. Interesting...

Update:

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
Downloading source file /usr/src/python3.12-3.12.7-1/build-static/../Objects/object.c
0x0000000000584cbe in _PyObject_New (tp=0x0) at ../Objects/object.c:319                                                                                                                                            
warning: 319    ../Objects/object.c: No such file or directory
(gdb) bt
#0  0x0000000000584cbe in _PyObject_New (tp=0x0) at ../Objects/object.c:319
#1  0x00007fffd09cc276 in SWIG_Python_newvarlink () at /tmp/tmpi3x042za.build-temp_amici/swig/CMakeFiles/_amici.dir/amiciPYTHON_wrap.cxx:1576
#2  0x00007fffd09cc36c in SWIG_globals () at /tmp/tmpi3x042za.build-temp_amici/swig/CMakeFiles/_amici.dir/amiciPYTHON_wrap.cxx:1606
#3  0x00007fffd0af0dc9 in PyInit__amici () at /tmp/tmpi3x042za.build-temp_amici/swig/CMakeFiles/_amici.dir/amiciPYTHON_wrap.cxx:85896
#4  0x00000000006a7711 in _PyImport_LoadDynamicModuleWithSpec (spec=0x7ffff414aae0, fp=<optimized out>) at ../Python/importdl.c:169

dweindl · 2024-11-05T19:50:28Z

Python crashes specifically when importing amici under pytest. Otherwise it works fine. Interesting...

It's swig/swig#3061

dweindl · 2024-11-06T14:23:15Z

Leaving this open as a reminder to revert #2572 once the issue has been fixed in swig.

dweindl · 2024-11-10T10:48:18Z

A better fix than #2572 (comment) might be just ignoring the problematic swig warnings before importing the extension. Untested.

dweindl added the ci label Oct 24, 2024

dweindl added a commit to dweindl/AMICI that referenced this issue Oct 24, 2024

GHA: PYTHONFAULTHANDLER=1

e4c4606

Set PYTHONFAULTHANDLER=1 for more output on crashes (AMICI-dev#2565). So far, this was only set for Ubuntu tests.

dweindl mentioned this issue Oct 24, 2024

GHA: PYTHONFAULTHANDLER=1 #2566

Merged

dweindl added a commit that referenced this issue Oct 24, 2024

GHA: PYTHONFAULTHANDLER=1 (#2566)

3b7c6d1

Set PYTHONFAULTHANDLER=1 for more output on crashes (#2565). So far, this was only set for Ubuntu tests.

dweindl mentioned this issue Nov 5, 2024

GHA: Workaround for macos segfaults #2572

Merged

dweindl changed the title ~~scripts/run-python-tests.sh: line 22: 7262 Segmentation fault~~ segfaults with swig4.3 and PYTHONFAULTHANDLER=1 Nov 26, 2024

dweindl added the swig label Nov 27, 2024

flferretti mentioned this issue Dec 4, 2024

Tests on Python bindings fail with SEGFAULT when PYTHONWARNINGS=error is set robotology/idyntree#1217

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

segfaults with swig4.3 and PYTHONFAULTHANDLER=1 #2565

segfaults with swig4.3 and PYTHONFAULTHANDLER=1 #2565

dweindl commented Oct 24, 2024 •

edited

Loading

dweindl commented Oct 24, 2024

dweindl commented Oct 24, 2024

dweindl commented Oct 24, 2024 •

edited

Loading

FFroehlich commented Oct 24, 2024

FFroehlich commented Oct 27, 2024

dweindl commented Nov 5, 2024

dweindl commented Nov 5, 2024 •

edited

Loading

dweindl commented Nov 5, 2024 •

edited

Loading

dweindl commented Nov 5, 2024

dweindl commented Nov 6, 2024

dweindl commented Nov 10, 2024

segfaults with swig4.3 and PYTHONFAULTHANDLER=1 #2565

segfaults with swig4.3 and PYTHONFAULTHANDLER=1 #2565

Comments

dweindl commented Oct 24, 2024 • edited Loading

dweindl commented Oct 24, 2024

dweindl commented Oct 24, 2024

dweindl commented Oct 24, 2024 • edited Loading

FFroehlich commented Oct 24, 2024

FFroehlich commented Oct 27, 2024

dweindl commented Nov 5, 2024

dweindl commented Nov 5, 2024 • edited Loading

dweindl commented Nov 5, 2024 • edited Loading

dweindl commented Nov 5, 2024

dweindl commented Nov 6, 2024

dweindl commented Nov 10, 2024

dweindl commented Oct 24, 2024 •

edited

Loading

dweindl commented Oct 24, 2024 •

edited

Loading

dweindl commented Nov 5, 2024 •

edited

Loading

dweindl commented Nov 5, 2024 •

edited

Loading