Update Pyo3 bounds

[Michael-J-Ward]

pyo3 0.21 is in arrow-rs master (see apache/arrow-rs#5566), but not yet released.

• Fixes: Use new Bound API of pyO3  #437

Notes

After updating the deps, this was completely a compiler / clippy driven refactor where clippy highlights deprecation warnings, and I would update using the migration guide.

I did not go looking for any other opportunities to use the new Bounds api.

[phil-opp]

Looks good, thanks a lot! Let's hope that the arrow project publishes a new release soon.

[Michael-J-Ward]

@haixuanTao the last deprecation warning was here.

dora/binaries/runtime/src/operator/python.rs

Lines 187 to 197 in bbabdc3

	// We need to create a new scoped `GILPool` because the dora-runtime
	// is currently started through a `start_runtime` wrapper function,
	// which is annotated with `#[pyfunction]`. This attribute creates an
	// initial `GILPool` that lasts for the entire lifetime of the `dora-runtime`.
	// However, we want the `PyBytes` created below to be freed earlier.
	// creating a new scoped `GILPool` tied to this closure, will free `PyBytes`
	// at the end of the closure.
	// See https://github.com/PyO3/pyo3/pull/2864 and
	// https://github.com/PyO3/pyo3/issues/2853 for more details.
	let pool = unsafe { py.new_pool() };
	let py = pool.python();

I think that the new bounds API makes this unnecessary, and the deprecation warning states "code not using the GIL Refs API can safely remove the use of Py::new_pool".

I included that change as the last commit, so if you find that it is still necessary it'll be an easy rollback.

[haixuanTao]

@haixuanTao the last deprecation warning was here.

dora/binaries/runtime/src/operator/python.rs

Lines 187 to 197 in bbabdc3

	// We need to create a new scoped `GILPool` because the dora-runtime
	// is currently started through a `start_runtime` wrapper function,
	// which is annotated with `#[pyfunction]`. This attribute creates an
	// initial `GILPool` that lasts for the entire lifetime of the `dora-runtime`.
	// However, we want the `PyBytes` created below to be freed earlier.
	// creating a new scoped `GILPool` tied to this closure, will free `PyBytes`
	// at the end of the closure.
	// See https://github.com/PyO3/pyo3/pull/2864 and
	// https://github.com/PyO3/pyo3/issues/2853 for more details.
	let pool = unsafe { py.new_pool() };
	let py = pool.python();

I think that the new bounds API makes this unnecessary, and the deprecation warning states "code not using the GIL Refs API can safely remove the use of Py::new_pool".

I included that change as the last commit, so if you find that it is still necessary it'll be an easy rollback.

That makes sense, thanks!

[haixuanTao]

This fix an issue that I had about stopping a dataflow that has been present for a bit!

Thanks a lot!

Can't wait to merge it.

[Michael-J-Ward]

@haixuanTao - It could be a few months before arrow-rs gets released.

If you don't mind pointing to arrow-rs's master, then I'll rebase this so you can merge now, and I'll update the deps once it does get released.

[haixuanTao]

If we merge it, we will not be able to publish on cargo, which will makes our release stuck. So I wouldn´t do it.

What we could do, is to only merge arrow version within dora-node-api-python that is distributed using pip which does not check for git packages and distribute dora-node-api-python with the latest arrow.

But not sure if we can handle multi arrow versions.

[Michael-J-Ward]

@haixuanTao This is rebased and ready for review.

[haixuanTao]

Looks awesome!

Thanks a million!