Skip to content

Tokenizers fails to build in Python3.13t (NoGIL build) #1749

Closed
@vinayakdsci

Description

@vinayakdsci

When installing tokenizers from main with the following command invoked:

python3.13t -m pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python

there is a build failure originating from the numpy crate:

        Compiling numpy v0.23.0
      error[E0432]: unresolved import `pyo3::sync::GILProtected`
        --> /home/vinayak/.cargo/registry/src/index.crates.io-6f17d22bba15001f/numpy-0.23.0/src/datetime.rs:63:12
         |
      63 | use pyo3::{sync::GILProtected, Bound, Py, Python};
         |            ^^^^^^^^^^^^^^^^^^ no `GILProtected` in `sync`
         |
      note: found an item that was configured out
        --> /home/vinayak/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.23.5/src/sync.rs:47:12
         |
      47 | pub struct GILProtected<T> {
         |            ^^^^^^^^^^^^

      error[E0432]: unresolved import `pyo3::sync::GILProtected`
        --> /home/vinayak/.cargo/registry/src/index.crates.io-6f17d22bba15001f/numpy-0.23.0/src/strings.rs:15:5
         |
      15 |     sync::GILProtected,
         |     ^^^^^^^^^^^^^^^^^^ no `GILProtected` in `sync`
         |
         = note: unresolved item `crate::datetime::units::GILProtected` exists but is inaccessible
      note: found an item that was configured out
        --> /home/vinayak/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.23.5/src/sync.rs:47:12
         |
      47 | pub struct GILProtected<T> {
         |            ^^^^^^^^^^^^

      For more information about this error, try `rustc --explain E0432`.
      error: could not compile `numpy` (lib) due to 2 previous errors
      warning: build failed, waiting for other jobs to finish...
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.13-64bit" PYO3_PYTHON="/home/vinayak/.venv/python3.13_ft/bin/python3.13t" PYTHON_SYS_EXECUTABLE="/home/vinayak/.venv/python3.13_ft/bin/python3.13t" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/pip-req-build-7xcc2xt3/bindings/python/Cargo.toml" "--release" "--lib"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/home/vinayak/.venv/python3.13_ft/bin/python3.13t', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]

tokenizers appears to be using v0.23.0 of the crate, which has since released a v0.24.0.

Here is more info about my Python version:

$ python3.13t -VV

Python 3.13.2+ experimental free-threading build (heads/3.13:4d7d249, Mar 16 2025, 23:30:31) [GCC 11.4.0]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions