Equivalent to PyCall.pyembed

[pabloferz]

Is there a way to keep a julia object around, e.g. within PYJLVALUES and remove it once a python object gets garbage collected via a weak reference (something similar to PyCall.pyembed)?

[cjdoris]

Nothing built in for this but you can put the Julia object as an attribute on the Python object whose lifetime you want to tie it to. Or use a WeakKeyDict with the Python object as a key and Julia object as a value.

[pabloferz]

Thanks. Yes, this should also work.

[pabloferz]

I might have spoken too soon, for my particular case using a WeakKeyDict won't work, the python objects I would like to have as keys seem to return None for hash so using using them I get an ERROR: Python: TypeError: unhashable type.

On the other hand. trying to set the object as attribute throws ERROR: Python: AttributeError: 'DeviceArray' object has no attribute '__jl_val__'

[cjdoris]

What's the context of this, what are you trying to do?

[pabloferz]

I'm writing a package that allows to share tensors (including GPU lived ones) without copying the data, both from julia to python and the other way around (https://github.com/pabloferz/DLPack.jl). There is a C API for this that many libraries such as jax, pytorch, cupy, etc. support. The conventions to perform data sharing through this API are described here https://data-apis.org/array-api/latest/design_topics/data_interchange.html.

I can wrap python tensors already and access them from julia with both PyCall and PythonCall. For sharing from julia to python I need to generate the python objects each library provide (I can already do this), but I would like to keep the julia arrays around at least for as long as the corresponding python tensors live. In PyCall I can call pyembed, but I was looking for the simplest way of doing the same in PythonCall. Unfortunately the most straightforward ways of doing it are not really working, but in the end I can always try to build my own pool to held the julia arrays and have python weak references with callbacks to release the arrays.

[cjdoris]

Ah cool! I hadn't heard of dlpack before, but looks useful.

I guess there are two ways to do the Julia-to-Python sharing, it's not clear which you're doing?

• Directly create a Python array from a pointer to the Julia data, etc.
• Create a Python object wrapping the Julia object, which supports the __dlpack__ interface, then create an array from that.

It sounds like you're doing the first option? If so, yeah, you'll have to make your own pool. I guess PythonCall could provide a function for that.

If you do the second option (which I think is preferred), you can instead use the DLManagedTensor.deleter to delete the Julia object from the pool. This way you don't need weak references.

Note that PythonCall already does something very similar to the second option in that Py([1,2,3]) exposes the numpy array interface, namely it has a __array_interface__ attribute which has a data pointer, shape, strides, etc, very similar to __dlpack__.

It would be cool if a package (such as this) introduced some traits like is_dlpack_array(::Type) and to_dlpack(x) which other packages such as CUDA could implement to signal that CuArray can be dlpackified. Then we could add a __dlpack__ attribute generically in PythonCall so that Py(::CuArray) can be consumed by torch or jax or.....

[pabloferz]

Yes, I'm doing the first. The second option is what the API establishes as semantics but is not yet implemented by all python libraries. In addition DLManagedTensor is supposed to be a transient sharing format, that is, the __dlpack__ method is supposed to generate PyCapsule around a DLManagedTensor all libraries that support the interface are supposed to have a from_dlpack that can generate an array from these PyCapsules (ideally from the __dlpack__ attribute). The DLManagedTensor passes information such as the pointer, dimensions, type, etc. and although the dlpack API requires the ownership of the DLManagedTensor to be transferred, again, not all dlpack-compatible python libraries support the proposed semantics yet. So for the time being I wanted to to have a stopgap solution based on the first option.

I'll think about having traits for this. For now I'm providing conditional support for CUDA (through Requires) in the the package itself.

[cjdoris]

Ok fair enough. Once you've figured it out it would be cool to add __dlpack__ to the PythonCall Julia wrapper types.

On the traits front, I think ArrayInterface.jl gives you everything you need already, which is neat, though the mapping from their device to dlpack device isn't totally obvious.

I noticed you use some capsule functions not provided by PythonCall. Feel free to open a PR to add them if you like.