How to use a struct(pyclass) as argument in both Rust and Python side without copy?

[sun-rs]

Because Python for-loop is slow, I use Pyo3 to run a loop and manipulate some struct(contains Python class with methods).

To do that I have three struct-Player(Pure Rust) Account(Pure Rust) Context(empty in Rust and user define its real code in Python), Player has Account and Context in it, it will loop the method of Context .

While in each loop, it would send its Account instance as &mut self.acc argument into its Context instance. Because Account would contains a lot of data, and it would be also use in Rust side and Python side. I couldn't copy it everytime.

So here is the question: what should I do to self.acc in Player struct to make it could be used as argument in Rust and Python at sametime without copy? Or say how to send a pyclass as an argument to a PyAny object and call ites method?

use pyo3::exceptions::PyValueError;
use pyo3::prelude::*;
use pyo3::types::PyAny;

#[pymodule]
fn play(_py: Python, m: &PyModule) -> PyResult<()> {
    
    m.add_class::<Context>()?;
    m.add_class::<Player>()?;
    m.add_class::<Account>()?;
    Ok(())
}

#[pyclass]
#[derive(Debug, Clone)]
pub struct Account {
    cash: i64
}

#[pymethods]
impl Account {
    #[new]
    pub fn new(cash: i64) -> Self {
        Account { cash:100 }
    }
    pub fn add_money(&mut self, money: i64) {
        self.cash += money
    }
    pub fn clear_cash(&mut self) {
        self.cash = 0
    }
}


#[pyclass]
#[derive(Debug, Clone)]
pub struct Context {
    pycls: Py<PyAny>,
}

#[pymethods]
impl Context {
    #[new]
    pub fn new(pycls: Py<PyAny>) -> Self {
        Context { pycls }
    }

    pub fn on_calculation(&mut self, acc: &mut Account) {
       
        Python::with_gil(|py| {
            //-----------------------------------------------------------------------------------------------
            // Account need to borrowed or as_ref and send into python method "on_calculation"
            // this can't work ??????
            let acc: PyObject = acc.into_py(py); 
            //-----------------------------------------------------------------------------------------------
            let py_model = self.pycls.as_ref(py);
            py_model
                .call_method("on_calculation", (acc,), None)
                .unwrap();
        });
        
    }
}

#[pyclass]
pub struct Player {
    acc: Acc,
    context: Context,
}

#[pymethods]
impl Player {
    #[new]
    pub fn new(acc:Account, ctx: Context) -> Self {
        Player { acc, context:ctx }
    }

    pub fn run(&mut self, num:i64) {
        for _i in 0..num {
            self.context.on_calculation(&mut self.acc);
        }
        self.acc.clear_cash()  // Account need to be used in Rust !!!!
    }
}

Python code:

import play
class TestContext:
    
    def on_calculation(self, acc): 
        print("here is user define code and would run in rust")
        acc.add_money(99)   #use method of acc here!!!!

pycls = TestContext()
rust_struct = play.Context(pycls)
acc = play.Account(999)
x = x.Player(acc, rust_struct)
x.run(3)

[davidhewitt]

You should change your Player struct to:

#[pyclass]
pub struct Player {
    acc: Py<Acc>,
    context: Py<Context>,
}

This way the acc and context fields will be stored as their Python representations.

[sun-rs]

You should change your Player struct to:

#[pyclass]

pub struct Player {

    acc: Py<Acc>,

    context: Py<Context>,

}

This way the acc and context fields will be stored as their Python representations.

Thanks for reply. If method of 'Py‹Account›' need to be called in rust， I must aquire gil first then call the method？ If so， the performance would be worse than pure rust method of origin struct？

[adamreichold]

I must aquire gil first then call the method？ If so， the performance would be worse than pure rust method of origin struct？

Yes, but in your example you already have the GIL when entering run, you just need to add an argument to access it, e.g.

#[pymethods]
impl Player {
    pub fn run(&mut self, num:i64, py: Python) {
        for _i in 0..num {
            self.context.on_calculation(&self.acc, py);
        }
        self.acc.as_ref(py).clear_cash()  // Account need to be used in Rust !!!!
    }
}

[sun-rs]

I must aquire gil first then call the method？ If so， the performance would be worse than pure rust method of origin struct？

Yes, but in your example you already have the GIL when entering run, you just need to add an argument to access it, e.g.

#[pymethods]

impl Player {

    pub fn run(&mut self, num:i64, py: Python) {

        for _i in 0..num {

            self.context.on_calculation(&self.acc, py);

        }

        self.acc.as_ref(py).clear_cash()  // Account need to be used in Rust !!!!

    }

}

Could I make this conclusion that if I want to fully achieve pure rust performance I must make sure no 'acquire gil' in my rust code of the extension module？

[adamreichold]

Could I make this conclusion that if I want to fully achieve pure rust performance I must make sure no 'acquire gil' in my rust code of the extension module？

This is probably not the definite answer you want to hear, but when assessing the performance of software, then you just need to measure. Whether taking the GIL is problem or not is hard to say in a general manner and without additional context. There can even be places where taking the GIL can advantageous, e.g. if separate synchronization mechanisms would be required otherwise.

In your particular case, Player::run will definitely be invoked with the GIL held as it is invoked from Python code. So in this particular case, you should just make sure that you do not do Python::with_gil within a loop because that is superfluous work (it will just check that the GIL is already held and move on). The mitigation as shown above is to pass the Python token via function arguments.

[sun-rs]

Thanks! I understand that.

What I want to achieve is to let the flexible part be written in Python while the fixed part executed in Rust.

Some people use Python+ctypes+cpp to do the same stuff like what I did.But they have a trick that they can use cython to re-write their Python codes easily with its 'nongil' mode, so they could get a better performance than a for-loop with gil.

Could we do some thing like that？

[davidhewitt]

We've been talking about making a #[pyclass(immutable)] variation. I think that the interesting thing there is that you wouldn't need to have to hold the GIL to access the Rust data stored in a Py<Acc> if Acc was an immutable pyclass, because there would be no need to mediate access to the Rust data by the GIL if it was immutable.

The draft PR is #1979

Perhaps you'd be interested in helping test and finish off the functionality? Hopefully we can get it complete for the 0.17 release.

cc @mejrs

[sun-rs]

We've been talking about making a #[pyclass(immutable)] variation. I think that the interesting thing there is that you wouldn't need to have to hold the GIL to access the Rust data stored in a Py<Acc> if Acc was an immutable pyclass, because there would be no need to mediate access to the Rust data by the GIL if it was immutable.

The draft PR is #1979

Perhaps you'd be interested in helping test and finish off the functionality? Hopefully we can get it complete for the 0.17 release.

cc @mejrs

It seems that immutable pyclass is very suitable for read data which is read-only？

In fact I understand that if we call a method in Python we must get gil first， but what I do confused is that if 'Acc' is write in pure Rust，why couldn't we call its methods and change value directly while skip gil action？

At the beginning， the reason I wtire the example is to expose less to the Python side and let 'Player' do some manipulate work on 'Acc' so I wouldn't need to write much in Python side. But if I must use gil to call acc in Player ，it seems make no sense to put it in Rust side？

[davidhewitt]

The point is that in Py<Acc>, Python threads can modify the Acc data, so we cannot access it without borrow checking. The GIL is used to make the borrow-checking thread safe.

If Acc was our theoretical immutable pyclass, then Py<Acc> could be accessed by as many threads as we like without needing any synchronization, because the data would be guaranteed to not change.

[sun-rs]

Thank you very much! I'm fully understand this mechanism.

When access some vector or array type data, people may use unsafe fn get_unchecked to skip the bound check while he could make sure not exceed the boundary. Sadly it seems we can not skip borrow check even in unsafe. https://steveklabnik.com/writing/you-can-t-turn-off-the-borrow-checker-in-rust

[mejrs]

To put it another way, what Rust really cares about preventing is aliased mutability, which has been a big problem in language design.

But if there is no aliasing (by, for example using a Mutex or RefCell, or in this case PyCell), or if there is no mutation (because the object is immutable) there are no problems.