feat: Check Python version when deserializing UDFs

[stinodego]

cloudpickle does not support serde across Python versions:

Cloudpickle can only be used to send objects between the exact same version of Python.

Currently, failing to adhere to this restriction can cause code execution to hang.

So we should only use cloudpickle when necessary, and when we do, we should encode the Python version during serialization and check it during deserialization. I chose to only encode the minor version, as we only support Python 3 anyway and the patch version doesn't seem to matter.

[stinodego]

cloudpickle simply re-exports pickle.loads, so trying to import cloudpickle during deserialization is a waste.

[stinodego]

Seems the test_fork_safety test is failing for this branch - though it seems unrelated to my changes.

@itamarst Could you have a look what could be going on? I think it was introduced in your recent PR (#19149)

[codecov]

Codecov Report

Attention: Patch coverage is 98.36066% with 1 line in your changes missing coverage. Please review.

Project coverage is 79.81%. Comparing base (48bc09b) to head (6dd7865).
Report is 32 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/polars-plan/src/dsl/python_udf.rs	98.36%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #19175   +/-   ##
=======================================
  Coverage   79.80%   79.81%           
=======================================
  Files        1532     1532           
  Lines      208500   208539   +39     
  Branches     2418     2418           
=======================================
+ Hits       166399   166436   +37     
- Misses      41554    41556    +2     
  Partials      547      547           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[itamarst]

Looking...

[itamarst]

Ah, so, jax has the same limitations as polars in that it doesn't like fork(), so they installed a fork() handler that does a warning, and a test that uses fork() then triggers that warning if jax happened to be used or maybe just imported first. So solution might be... suppress warnings inside that test? E.g. adding a @pytest.mark.filterwarnings("ignore") on that particular test would probably work.

Shall I do that as a separate PR?

[stinodego]

Shall I do that as a separate PR?

@itamarst sounds great, thanks!

[ritchie46]

Can we cache this behind a lazylock?

[stinodego]

Yes, done!

Unfortunately, now I can no longer patch the sys.version_info from the Python side, so we can no longer really test this. If you have an idea how to test it, let me know. Otherwise, we'll just have to trust the code 😬