Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build(python): Update pyo3 and numpy crates to version 0.23 #20111

Draft
wants to merge 16 commits into
base: main
Choose a base branch
from
Draft

build(python): Update pyo3 and numpy crates to version 0.23 #20111

wants to merge 16 commits into from
+614 −581

Conversation

bschoenmaeckers
Copy link

Updated to latest pyo3 while ignoring most deprecations for now.

@codecov Codecov
Copy link

codecov bot commented Dec 2, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 74.78632% with 59 lines in your changes missing coverage. Please review.

Project coverage is 79.56%. Comparing base (bcfa7ec) to head (1286e79).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
.../polars-python/src/lazyframe/visitor/expr_nodes.rs 0.00% 20 Missing ⚠️
crates/polars-python/src/map/series.rs 58.69% 19 Missing ⚠️
crates/polars-python/src/cloud.rs 33.33% 4 Missing ⚠️
crates/polars-python/src/map/lazy.rs 69.23% 4 Missing ⚠️
crates/polars-python/src/conversion/mod.rs 62.50% 3 Missing ⚠️
crates/polars-error/src/lib.rs 0.00% 2 Missing ⚠️
crates/polars-python/src/dataframe/export.rs 93.93% 2 Missing ⚠️
crates/polars-python/src/dataframe/serde.rs 0.00% 2 Missing ⚠️
crates/polars-python/src/file.rs 81.81% 2 Missing ⚠️
crates/polars-python/src/dataframe/general.rs 75.00% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main   #20111      +/-   ##
==========================================
- Coverage   79.57%   79.56%   -0.01%     
==========================================
  Files        1563     1563              
  Lines      217395   217418      +23     
  Branches     2472     2472              
==========================================
- Hits       172986   172985       -1     
- Misses      43841    43865      +24     
  Partials      568      568

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@stinodego stinodego changed the title chore: update to pyo3 v0.23 build: Update pyo3 and numpy crates to version 0.23 Dec 2, 2024
@github-actions github-actions bot added build Changes that affect the build system or external dependencies python Related to Python Polars rust Related to Rust Polars and removed title needs formatting labels Dec 2, 2024
@stinodego stinodego changed the title build: Update pyo3 and numpy crates to version 0.23 build(python): Update pyo3 and numpy crates to version 0.23 Dec 2, 2024
@stinodego stinodego removed the rust Related to Rust Polars label Dec 2, 2024
@stinodego
Copy link
Member

stinodego commented Dec 3, 2024 •
edited
Loading

@bschoenmaeckers Thanks for picking this up! Looks like the new PyO3 version causes deadlock somewhere - can you reproduce this locally? There's probably a clue in the migration guide somewhere on what could be causing this.

@jqnatividad jqnatividad mentioned this pull request Dec 3, 2024
Closed
@bschoenmaeckers
Copy link
Author

@bschoenmaeckers Thanks for picking this up! Looks like the new PyO3 version causes deadlock somewhere - can you reproduce this locally? There's probably a clue in the migration guide somewhere on what could be causing this.

Haven't had time to dig deeper today, but I could quite consistently reproduce this error locally with the following command.

pytest tests/unit/operations/map/test_map_elements.py -n 4 --verbose

@jqnatividad jqnatividad mentioned this pull request Dec 4, 2024
Open
@bschoenmaeckers
Copy link
Author

Deadlock happens in the Display implementation of PyErr when 2 threads call warning_function at the same time. Will investigate further to reproduce this within Pyo3.

polars/crates/polars-python/src/on_startup.rs

Lines 58 to 66 in bcfa7ec

fn warning_function(msg: &str, warning: PolarsWarning) {
Python::with_gil(|py| {
let warn_fn = UTILS.bind(py).getattr(intern!(py, "_polars_warn")).unwrap();
if let Err(e) = warn_fn.call1((msg, Wrap(warning))) {
eprintln!("{e}")
}
});
}

@bschoenmaeckers bschoenmaeckers mentioned this pull request Dec 4, 2024
Open
@ritchie46
Copy link
Member

ritchie46 commented Dec 4, 2024 •
edited
Loading

@bschoenmaeckers

Ah (spent too many hours on this0, I also came to the conclusion that it is related to the warning. What I observe is that the GIL doesn't matter, but if I call warning.warn in _polars_warn we deadlock. if we do anything else there it seems to be ok.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ritchie46 ritchie46 Awaiting requested review from ritchie46 ritchie46 will be requested when the pull request is marked ready for review ritchie46 is a code owner

@c-peters c-peters Awaiting requested review from c-peters c-peters will be requested when the pull request is marked ready for review c-peters is a code owner

@alexander-beedie alexander-beedie Awaiting requested review from alexander-beedie alexander-beedie will be requested when the pull request is marked ready for review alexander-beedie is a code owner

@MarcoGorelli MarcoGorelli Awaiting requested review from MarcoGorelli MarcoGorelli will be requested when the pull request is marked ready for review MarcoGorelli is a code owner

@reswqa reswqa Awaiting requested review from reswqa reswqa will be requested when the pull request is marked ready for review reswqa is a code owner

@wence- wence- Awaiting requested review from wence- wence- will be requested when the pull request is marked ready for review wence- is a code owner

@orlp orlp Awaiting requested review from orlp orlp will be requested when the pull request is marked ready for review orlp is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
build Changes that affect the build system or external dependencies python Related to Python Polars
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@bschoenmaeckers @stinodego @ritchie46