Seamless debugging between python and Rust?

[topherbuckley]

Hello, I just started looking into a project that appears to use PyO3 and I've never used Rust before so wondering how people are going about debugging such projects.

I read through the debugging section of the docs, but it seems the gdb usecase there is running a rust binary as the entry point rather than python. And the python example is just a single function test and I can't access the python source from within rust-gdb it seems.

I'm no expert, but I am somewhat familiar with the debugging tools available in gdb, PyCharm and VSCode, but not sure if there is a way to inline debug (or even print things) seamlessly between the python and rust code.

Taking the README example of sum_as_string, I can start the Pycharm debugger and step into the generated stub of string_sum.py, but its just a stub. I'm looking for some kind of debugging tools that would allow me to either inline debug and go from python to rust and back again, or some way to print things (variable values, types, etc.) from rust while running from python.

The string_sum example is fairly straightforward to understand, but I'm trying to get a grasp on a MUCH larger project. I don't necessarily need to know all the innards of the project, as I just want to create a PR to fix one bug affecting me as an end user. Trying to find a good way to track down the source of such a bug by creating breakpoints near the identified methods controlling the user-input and following them thereafter.

Without being intimately familiar with a given project, what is a good way to step through the code of a PyO3 project like this to get a feel for how it works?

[adamreichold]

This will not improve the debugging experience w.r.t. the Python code, but you should be able to attach GDB to the CPython process executing your top-level Python code and it still place break points within any native extensions the interpreter loaded. (This should also work with rust-gdb which is just a wrapper loading some printing helpers.)

But you can certainly also apply "printf-style" debugging to the Rust code itself which should be visible on standard output like calling Python's print if nothing fancy is done to that. There is also a macro called dbg! to make this painless in the standard library.

[adamreichold]

Where the ones obviously did not find a matching source.

This is to be expected as you need to actually dynamically load the extension before those breakpoints can be resolved. When you start the Python interpreter, before the import string_sum statement is executed, the extension is not loaded yet.

Maybe I have to follow through a few more calls with finish but I'm not really sure what I am supposed to be looking for.

The point here is at the end of _PyImport_LoadDynamicModuleWithSpec, the native extension is loaded and you should be able to inspect it sources from within GDB. This would give you a first starting point if you do not know anything about the source structure of the extension.

[topherbuckley]

Maybe I've misunderstood, but I though that running through run main.py within gdb would have loaded the extension. Otherwise I don't really understand how the breakpoint 5 @ string_sum::sum_as_string would have been resolved. I should have clarified that I ran info breakpoints after running run main.py and it finished execution. After which, the lib.rs could be resolved within gdb.

you should be able to inspect it sources from within GDB.

You mean find it directly, or find it within the list of info sources? Based on what you wrote last time, I was under the impression that finish would result in me entering the lib.rs file, but I think I've misunderstood that.

[adamreichold]

Maybe I've misunderstood, but I though that running through run main.py within gdb would have loaded the extension. Otherwise I don't really understand how the breakpoint 5 @ string_sum::sum_as_string would have been resolved. I should have clarified that I ran info breakpoints after running run main.py and it finished execution. After which, the lib.rs could be resolved within gdb.

Yes, if you ran that after running the script, the breakpoint to the init function should have been resolved. Not sure why GDB does find the function but does find sum_as_string, it works here:

string_sum> rust-gdb python3
GNU gdb (GDB; openSUSE Tumbleweed) 12.1
[..]
Reading symbols from python3...
(No debugging symbols found in python3)
Missing separate debuginfos, use: zypper install python310-base-debuginfo-3.10.9-3.1.x86_64
(gdb) break string_sum::string_sum::init
Function "string_sum::string_sum::init" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (string_sum::string_sum::init) pending.
(gdb) r
Starting program: /usr/bin/python3 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Python 3.10.9 (main, Dec 08 2022, 14:49:06) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import string_sum

Breakpoint 1, string_sum::string_sum::init () at src/lib.rs:12
12      #[pymodule]
(gdb) bt
#0  string_sum::string_sum::init () at src/lib.rs:12
#1  0x00007ffff7dcb8e6 in ?? () from /lib64/libpython3.10.so.1.0
#2  0x00007ffff7d2cb1b in ?? () from /lib64/libpython3.10.so.1.0
#3  0x00007ffff7d24269 in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0
#4  0x00007ffff7d208a3 in ?? () from /lib64/libpython3.10.so.1.0
#5  0x00007ffff7d25ea0 in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0
#6  0x00007ffff7d208a3 in ?? () from /lib64/libpython3.10.so.1.0
#7  0x00007ffff7d21d47 in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0
#8  0x00007ffff7d208a3 in ?? () from /lib64/libpython3.10.so.1.0
#9  0x00007ffff7d219ca in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0
#10 0x00007ffff7d208a3 in ?? () from /lib64/libpython3.10.so.1.0
#11 0x00007ffff7d219ca in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0
#12 0x00007ffff7d208a3 in ?? () from /lib64/libpython3.10.so.1.0
#13 0x00007ffff7d219ca in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0
#14 0x00007ffff7d208a3 in ?? () from /lib64/libpython3.10.so.1.0
#15 0x00007ffff7d2c14a in ?? () from /lib64/libpython3.10.so.1.0
#16 0x00007ffff7d3786e in _PyObject_CallMethodIdObjArgs () from /lib64/libpython3.10.so.1.0
#17 0x00007ffff7d370ed in PyImport_ImportModuleLevelObject () from /lib64/libpython3.10.so.1.0
#18 0x00007ffff7d24ee7 in _PyEval_EvalFrameDefault () from /lib64/libpython3.10.so.1.0
#19 0x00007ffff7d208a3 in ?? () from /lib64/libpython3.10.so.1.0
#20 0x00007ffff7d974c0 in PyEval_EvalCode () from /lib64/libpython3.10.so.1.0
#21 0x00007ffff7da89a3 in ?? () from /lib64/libpython3.10.so.1.0
#22 0x00007ffff7da423a in ?? () from /lib64/libpython3.10.so.1.0
#23 0x00007ffff7d0491b in ?? () from /lib64/libpython3.10.so.1.0
#24 0x00007ffff7d04a90 in _PyRun_InteractiveLoopObject () from /lib64/libpython3.10.so.1.0
#25 0x00007ffff7c95a75 in _PyRun_AnyFileObject () from /lib64/libpython3.10.so.1.0
#26 0x00007ffff7d04c01 in PyRun_AnyFileExFlags () from /lib64/libpython3.10.so.1.0
#27 0x00007ffff7c8bf57 in ?? () from /lib64/libpython3.10.so.1.0
#28 0x00007ffff7d893b7 in Py_BytesMain () from /lib64/libpython3.10.so.1.0
#29 0x00007ffff7a2caf0 in __libc_start_call_main () from /lib64/libc.so.6
#30 0x00007ffff7a2cbb9 in __libc_start_main_impl () from /lib64/libc.so.6
#31 0x0000555555555075 in _start ()
(gdb) 

(Using a debug build of the crate.)

You mean find it directly, or find it within the list of info sources? Based on what you wrote last time, I was under the impression that finish would result in me entering the lib.rs file, but I think I've misunderstood that.

Yes, I just meant that you can intercept the Python interpreter loading a PyO3 extension on which you basically have no prior knowledge. But indeed, you could also single step your way into the extensions initialization function this way.

[topherbuckley]

Huh, I also am no longer having an issue with it. I can't reproduce what I was seeing before ¯_(ツ)_/¯. Thanks for confirming though. Can you generalize the syntax you used for that though? i.e. where are the three terms in string_sum::string_sum::init defined? Would this be <Cargo.toml-package>::<Cargo.tom-lib>::init? Why is it init vs __init__ or something else? Happy to read relevant documentation, so if this is something specific to rust/cargo that I need to read up on, please just let me know that.

[kngwyu]

init is generated by our macro (https://github.com/PyO3/pyo3/blob/v0.18.1/pyo3-macros-backend/src/module.rs#L86), so it's just an ad-hoc naming by us. <crate-name>::<crate-name>::init should work. Here crate-name means lib.name in Cargo.toml.
(The second one can be different for non-top-level modules, but you might not need to care)

[kngwyu]

Do you mean that you want to set the breakpoint in Python code and jump into Rust code from that break point? I don't know whether there's any Python debugger that works like that.

[topherbuckley]

Ideally yes, but the workaround to use rust-gdb to run the python code (but limited to inspecting rust code) is also helpful.