cython tasks are KI protected

[graingert]

if you write a python function:

# tests/cython/pyfunc.py

import threading

def sleep() -> None:
    threading.Event().wait()

and call it from a cython task:

# cython: language_level=3
import trio

from .pyfunc import sleep

# the output of the prints are not currently checked, we only check
# if the program can be compiled and doesn't crash when run.

# The content of the program can easily be extended if there's other behaviour
# that might be likely to be problematic for cython.
async def foo() -> None:
    print('.')

async def bad():
    sleep()

async def trio_main() -> None:
    print('hello...')
    await trio.sleep(1)
    print(' world !')

    async with trio.open_nursery() as nursery:
        nursery.start_soon(foo)
        nursery.start_soon(foo)
        nursery.start_soon(foo)

    await bad()

trio.run(trio_main)

it hangs forever and can't be interrupted with ctrl+c

currently_ki_protected sees the following stack:

frame=<frame at 0x71d9198c7920, file '/usr/lib/python3.12/threading.py', line 355, code wait>
frame=<frame at 0x71d9198edd80, file '/usr/lib/python3.12/threading.py', line 655, code wait>
frame=<frame at 0x71d91992ed40, file '/home/graingert/projects/trio/tests/cython/pyfunc.py', line 4, code sleep>
frame=<frame at 0x71d91a083040, file '/home/graingert/projects/trio/src/trio/_core/_run.py', line 2720, code unrolled_run> (protected=True)

[graingert]

this is because Cython coroutines appear to have a frame, so it doesn't get a wrapper python coroutine:

trio/src/trio/_core/_run.py

Lines 1859 to 1867 in 57452ad

	# very old Cython versions (<0.29.24) has the attribute, but with a value of None
	if getattr(coro, "cr_frame", None) is None:
	# This async function is implemented in C or Cython
	async def python_wrapper(orig_coro: Awaitable[RetT]) -> RetT:
	return await orig_coro
	coro = python_wrapper(coro)
	assert coro.cr_frame is not None, "Coroutine frame should exist"

but don't actually push that frame when the coroutine runs

I think we need to check if not cr_frame.f_code.co_code:

[A5rocks]

but don't actually push that frame when the coroutine runs

Should this be raised as a Cython bug? (I know that doesn't help us in the meanwhile but having an attribute in some cases but not all is not useful)

I think we need to check if not cr_frame.f_code.co_code:

I assume this would undo any sort of KI protection enabled by outer scopes, because it's impossible to know whether it's enabling or disabling KI protection? I think that makes sense, though maybe it would be better to detect that the frame will not be there and then create a wrapper, unless bad gets a frame if Cython detects there's a decorator. This should presumably still block forever in Cython:

@enable_ki_protection
async def bad():
    sleep()

[graingert]

We could have two async wrappers for cython, one protected and one unprotected, and use the KI protected-ness of the coro.cr_frame.co_code to pick which one you use

This would not work for cases like this:

@enable_ki_protection
async def bad():
    sleep()

async def outer():
    return await bad()

[graingert]

The other option is for enable_ki_protection to detect cython functions and wrap them with an extra (async, generator,asyncgenerater)function

[A5rocks]

I think detecting Cython functions and wrapping then would ultimately be simpler and that's better.

We could have two async wrappers for cython, one protected and one unprotected, and use the KI protected-ness of the coro.cr_frame.co_code to pick which one you use

I don't get what you mean by this, though I'm also tired so it could be that too.

[A5rocks]

OK so looking at this more, in general sys._getframe doesn't get the frame for any given cython function, sync or async:

hello.pyx:

import sys

def synctest():
  print(sys._getframe(0))

def synctestwrapper():
  synctest()

in a REPL:

>>> import hello
>>> hello.synctest()
<frame at 0x00000166F3D6A200, file '<stdin>', line 1, code <module>>
>>> hello.synctestwrapper()
<frame at 0x00000166F3D6A200, file '<stdin>', line 1, code <module>>

Looking at cython's limitations page, this is intentional:

Currently we generate fake tracebacks as part of exception propagation, but don’t fill in locals and can’t fill in co_code. To be fully compatible, we would have to generate these stack frame objects at function call time (with a potential performance penalty). We may have an option to enable this for debugging.

Given that, I think we should create a wrapper around things if inspect.isfunction returns False? I can't imagine builtin methods work with KI protection either...

That somewhat negates the benefits of using the __code__ for ki protection (because we'll still need the lines for wrapping generators vs normal functions vs whatever) but we should still have the speed benefits in the common case.