Question about pythran support + LowLevelCallables

[peekxc]

Pythran is great, and it covers many of my own use-cases: python/numpy + pythran = efficient and maintainable code.

Occasionally, though, I want to pythranize a function that has an input parameter whose type is either not directly supported by Pythran (but maybe can be, via something like #2089) or is simply not known ahead-of-time. Use-cases include sparse matrix multiplication (the former) or a user-supplied function (the latter). Supporting one of the two above options would extend that coverage.

So, I had to ask, is it feasible, planned, or even in Pythrans philosophy to eventually extend support to:
a) something like LowLevelCallables
b) arbitrary Callables

I realize (b) is probably much harder to support than (a), and maybe both are out of scope. For example, I think (b) would involve expanding Pythran to support running arbitrary Python code via the interpreter. Nonetheless, there are situations where its useful to have the ability to invoke the interpreter in the middle or otherwise natively compiled code.

Other libraries used in similar contexts to pythran tend to support these features to some extent. Cython supports (b) of course (and I think (a)), and Numba supports (b) via object mode. Like Cython, pybind11 supports both (b) via invoking the interpreter and also supports something close to (a) when the C function is stateless and its signature is known (see the 'warning' section here).

[serge-sans-paille]

Is a) be close to https://pythran.readthedocs.io/en/latest/MANUAL.html#capsule-corp ?
Pytrhan-generated modules release the GIL once they have parsed the exported function arguments, which is incompatible with b) as far as I understand.

[peekxc]

Going off this great blog post, I was hoping to be able to write something like:

#pythran export capsule f(int32, float64*, float64* )
def f(n, x, cp):
    c = cp[0]
    return c + x[0] - x[1] * x[2]

Then, somewhere else (possibly in a different library!), have something like:

#pythran export my_fancy_nquad(double (int, double *, void *), float64* )
def my_fancy_nquad(func, dat) -> float: 
   ...

Usage would then be like:

from testlib import f
from otherlib import my_fancy_nquad

c = ctypes.c_double(1.0)
user_data = ctypes.cast(ctypes.pointer(c), ctypes.c_void_p)
func = LowLevelCallable(f, user_data, signature="double (int, double *, void *)")

my_fancy_nquad(func, user_data)

[serge-sans-paille]

Yes! If that doesn't work, then you should open a bug :-)

[peekxc]

I dont believe Pythran supports passing pointers-to-functions, i.e. this doesn't compile:

#pythran export my_fancy_nquad(double (int, double *, void *), float64* )
def my_fancy_nquad(func, dat) -> float: 
   return 0

[serge-sans-paille]

It does, and indeed the message you get when trying to compile the above are explicit:

 error: Unexpected identifier `double` at that point

then

error: Unexpected identifier `void` at that point

but the following compiles ok:

#pythran export my_fancy_nquad(float64 (int, float64 *, int *), float64* )
def my_fancy_nquad(func, dat) -> float:
   return 0

[serge-sans-paille]

(note: #2255 should avoid the confusion between double and float64, etc)

Should I consider this issue closed then?

[peekxc]

yes, sorry I didn't realize I was making a simple mistake