Closed
Description
When installing tokenizers from main
with the following command invoked:
python3.13t -m pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python
there is a build failure originating from the numpy crate:
Compiling numpy v0.23.0
error[E0432]: unresolved import `pyo3::sync::GILProtected`
--> /home/vinayak/.cargo/registry/src/index.crates.io-6f17d22bba15001f/numpy-0.23.0/src/datetime.rs:63:12
|
63 | use pyo3::{sync::GILProtected, Bound, Py, Python};
| ^^^^^^^^^^^^^^^^^^ no `GILProtected` in `sync`
|
note: found an item that was configured out
--> /home/vinayak/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.23.5/src/sync.rs:47:12
|
47 | pub struct GILProtected<T> {
| ^^^^^^^^^^^^
error[E0432]: unresolved import `pyo3::sync::GILProtected`
--> /home/vinayak/.cargo/registry/src/index.crates.io-6f17d22bba15001f/numpy-0.23.0/src/strings.rs:15:5
|
15 | sync::GILProtected,
| ^^^^^^^^^^^^^^^^^^ no `GILProtected` in `sync`
|
= note: unresolved item `crate::datetime::units::GILProtected` exists but is inaccessible
note: found an item that was configured out
--> /home/vinayak/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.23.5/src/sync.rs:47:12
|
47 | pub struct GILProtected<T> {
| ^^^^^^^^^^^^
For more information about this error, try `rustc --explain E0432`.
error: could not compile `numpy` (lib) due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
💥 maturin failed
Caused by: Failed to build a native library through cargo
Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.13-64bit" PYO3_PYTHON="/home/vinayak/.venv/python3.13_ft/bin/python3.13t" PYTHON_SYS_EXECUTABLE="/home/vinayak/.venv/python3.13_ft/bin/python3.13t" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/pip-req-build-7xcc2xt3/bindings/python/Cargo.toml" "--release" "--lib"`
Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/home/vinayak/.venv/python3.13_ft/bin/python3.13t', '--compatibility', 'off'] returned non-zero exit status 1
[end of output]
tokenizers
appears to be using v0.23.0 of the crate, which has since released a v0.24.0.
Here is more info about my Python version:
$ python3.13t -VV
Python 3.13.2+ experimental free-threading build (heads/3.13:4d7d249, Mar 16 2025, 23:30:31) [GCC 11.4.0]
Metadata
Metadata
Assignees
Labels
No labels