Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload wheels for free-threaded ABI #530

Open
ngoldbaum opened this issue Nov 11, 2024 · 3 comments
Open

Upload wheels for free-threaded ABI #530

ngoldbaum opened this issue Nov 11, 2024 · 3 comments

Comments

@ngoldbaum
Copy link

Currently trying to install orjson on the free-threaded build will fail:

goldbaum at Nathans-MBP in ~
○  python -VV
Python 3.13.0 experimental free-threading build (main, Nov  5 2024, 16:45:19) [Clang 16.0.0 (clang-1600.0.26.3)]

goldbaum at Nathans-MBP in ~
○  pip install orjson
Collecting orjson
  Using cached orjson-3.10.11.tar.gz (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: orjson
  Building wheel for orjson (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for orjson (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [79 lines of output]
      Running `maturin pep517 build-wheel -i /Users/goldbaum/.pyenv/versions/3.13.0t/bin/python3.13 --compatibility off`
      📦 Including license file "/private/var/folders/nk/yds4mlh97kg9qdq745g715rw0000gn/T/pip-install-mn3fc3k3/orjson_e67d5d1fb96f4eb2a05e585070cd98f4/LICENSE-APACHE"
      📦 Including license file "/private/var/folders/nk/yds4mlh97kg9qdq745g715rw0000gn/T/pip-install-mn3fc3k3/orjson_e67d5d1fb96f4eb2a05e585070cd98f4/LICENSE-MIT"
      🍹 Building a mixed python/rust project
      🔗 Found pyo3-ffi bindings
      🐍 Found CPython 3.13t at /Users/goldbaum/.pyenv/versions/3.13.0t/bin/python3.13
      💻 Using `MACOSX_DEPLOYMENT_TARGET=11.0` for aarch64-apple-darwin by default
         Compiling target-lexicon v0.12.16
         Compiling proc-macro2 v1.0.89
         Compiling unicode-ident v1.0.13
         Compiling once_cell v1.20.2
         Compiling rustversion v1.0.18
         Compiling serde v1.0.214
         Compiling libc v0.2.161
         Compiling cfg-if v1.0.0
         Compiling shlex v1.3.0
         Compiling itoa v1.0.11
         Compiling version_check v0.9.5
         Compiling ryu v1.0.18
         Compiling cc v1.1.31
         Compiling serde_json v1.0.132
         Compiling static_assertions v1.1.0
         Compiling memchr v2.7.4
         Compiling encoding_rs v0.8.35
         Compiling half v2.4.1
         Compiling smallvec v1.13.2
         Compiling uuid v1.11.0
         Compiling xxhash-rust v0.8.12
         Compiling bytecount v0.6.8
         Compiling itoap v1.0.1
         Compiling simdutf8 v0.1.5
         Compiling jiff v0.1.14
         Compiling associative-cache v2.0.0
         Compiling quote v1.0.37
         Compiling syn v2.0.86
         Compiling pyo3-build-config v0.23.0-dev (/private/var/folders/nk/yds4mlh97kg9qdq745g715rw0000gn/T/pip-install-mn3fc3k3/orjson_e67d5d1fb96f4eb2a05e585070cd98f4/include/pyo3/pyo3-build-config)
         Compiling castaway v0.2.3
         Compiling serde_derive v1.0.214
         Compiling pyo3-ffi v0.23.0-dev (/private/var/folders/nk/yds4mlh97kg9qdq745g715rw0000gn/T/pip-install-mn3fc3k3/orjson_e67d5d1fb96f4eb2a05e585070cd98f4/include/pyo3/pyo3-ffi)
         Compiling orjson v3.10.11 (/private/var/folders/nk/yds4mlh97kg9qdq745g715rw0000gn/T/pip-install-mn3fc3k3/orjson_e67d5d1fb96f4eb2a05e585070cd98f4)
      error: failed to run custom build command for `pyo3-ffi v0.23.0-dev (/private/var/folders/nk/yds4mlh97kg9qdq745g715rw0000gn/T/pip-install-mn3fc3k3/orjson_e67d5d1fb96f4eb2a05e585070cd98f4/include/pyo3/pyo3-ffi)`

      Caused by:
        process didn't exit successfully: `/private/var/folders/nk/yds4mlh97kg9qdq745g715rw0000gn/T/pip-install-mn3fc3k3/orjson_e67d5d1fb96f4eb2a05e585070cd98f4/target/release/build/pyo3-ffi-6f9989b7b5e4166f/build-script-build` (exit status: 1)
        --- stdout
        cargo:rustc-check-cfg=cfg(Py_LIMITED_API)
        cargo:rustc-check-cfg=cfg(Py_GIL_DISABLED)
        cargo:rustc-check-cfg=cfg(PyPy)
        cargo:rustc-check-cfg=cfg(GraalPy)
        cargo:rustc-check-cfg=cfg(py_sys_config, values("Py_DEBUG", "Py_REF_DEBUG", "Py_TRACE_REFS", "COUNT_ALLOCS"))
        cargo:rustc-check-cfg=cfg(invalid_from_utf8_lint)
        cargo:rustc-check-cfg=cfg(pyo3_disable_reference_pool)
        cargo:rustc-check-cfg=cfg(pyo3_leak_on_drop_without_reference_pool)
        cargo:rustc-check-cfg=cfg(diagnostic_namespace)
        cargo:rustc-check-cfg=cfg(c_str_lit)
        cargo:rustc-check-cfg=cfg(Py_3_7)
        cargo:rustc-check-cfg=cfg(Py_3_8)
        cargo:rustc-check-cfg=cfg(Py_3_9)
        cargo:rustc-check-cfg=cfg(Py_3_10)
        cargo:rustc-check-cfg=cfg(Py_3_11)
        cargo:rustc-check-cfg=cfg(Py_3_12)
        cargo:rustc-check-cfg=cfg(Py_3_13)
        cargo::rustc-check-cfg=cfg(Py_3_14)
        cargo:rerun-if-env-changed=PYO3_CROSS
        cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
        cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
        cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION
        cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG

        --- stderr
        error: the Python interpreter was built with the GIL disabled, which is not yet supported by PyO3
        = help: see https://github.com/PyO3/pyo3/issues/4265 for more information
        = help: please check if an updated version of PyO3 is available. Current version: 0.23.0-dev
        = help: set UNSAFE_PYO3_BUILD_FREE_THREADED=1 to suppress this check and build anyway for free-threaded Python
      warning: build failed, waiting for other jobs to finish...
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO MACOSX_DEPLOYMENT_TARGET="11.0" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.13-64bit" PYO3_PYTHON="/Users/goldbaum/.pyenv/versions/3.13.0t/bin/python3.13" PYTHON_SYS_EXECUTABLE="/Users/goldbaum/.pyenv/versions/3.13.0t/bin/python3.13" "cargo" "rustc" "--message-format" "json-render-diagnostics" "--manifest-path" "/private/var/folders/nk/yds4mlh97kg9qdq745g715rw0000gn/T/pip-install-mn3fc3k3/orjson_e67d5d1fb96f4eb2a05e585070cd98f4/Cargo.toml" "--release" "--lib" "--" "-C" "link-arg=-undefined" "-C" "link-arg=dynamic_lookup" "-C" "link-args=-Wl,-install_name,@rpath/orjson.cpython-313t-darwin.so"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/Users/goldbaum/.pyenv/versions/3.13.0t/bin/python3.13', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for orjson
Failed to build orjson

[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: pip install --upgrade pip
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (orjson)

PyO3 0.23 is coming out soon, but I see you vendor the bits of PyO3 you need. If I update your vendored pyo3-ffi and pyo3-build-config to what's in the PyO3 main branch right now, I get the following build errors:

   Compiling orjson v3.10.11 (/Users/goldbaum/Documents/orjson)
warning: [email protected]: In file included from include/yyjson/yyjson.c:23:
warning: [email protected]: include/yyjson/yyjson.h:938:60: warning: parameter 'flg' set but not used [-Wunused-but-set-parameter]
warning: [email protected]:   938 |                                           yyjson_read_flag flg) {
warning: [email protected]:       |                                                            ^
warning: [email protected]: include/yyjson/yyjson.c:5268:29: warning: variable 'pos' set but not used [-Wunused-but-set-variable]
warning: [email protected]:  5268 |     u8 *src = ++cur, *dst, *pos;
warning: [email protected]:       |                             ^
warning: [email protected]: include/yyjson/yyjson.c:6586:45: warning: parameter 'flg' set but not used [-Wunused-but-set-parameter]
warning: [email protected]:  6586 |                            yyjson_read_flag flg,
warning: [email protected]:       |                                             ^
warning: [email protected]: 3 warnings generated.
error[E0425]: cannot find function `_Py_IsImmortal` in crate `pyo3_ffi`
   --> src/util.rs:105:26
    |
105 |             if pyo3_ffi::_Py_IsImmortal($op) == 0 {
    |                          ^^^^^^^^^^^^^^ not found in `pyo3_ffi`
    |
   ::: src/deserialize/backend/yyjson.rs:261:21
    |
261 |                     reverse_pydict_incref!(pykey);
    |                     ----------------------------- in this macro invocation
    |
    = note: this error originates in the macro `reverse_pydict_incref` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0425]: cannot find function `_Py_IsImmortal` in crate `pyo3_ffi`
   --> src/util.rs:105:26
    |
105 |             if pyo3_ffi::_Py_IsImmortal($op) == 0 {
    |                          ^^^^^^^^^^^^^^ not found in `pyo3_ffi`
    |
   ::: src/deserialize/backend/yyjson.rs:262:21
    |
262 |                     reverse_pydict_incref!(pyval);
    |                     ----------------------------- in this macro invocation
    |
    = note: this error originates in the macro `reverse_pydict_incref` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0425]: cannot find function `_Py_IsImmortal` in crate `pyo3_ffi`
   --> src/util.rs:105:26
    |
105 |             if pyo3_ffi::_Py_IsImmortal($op) == 0 {
    |                          ^^^^^^^^^^^^^^ not found in `pyo3_ffi`
    |
   ::: src/deserialize/backend/yyjson.rs:269:21
    |
269 |                     reverse_pydict_incref!(pykey);
    |                     ----------------------------- in this macro invocation
    |
    = note: this error originates in the macro `reverse_pydict_incref` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0425]: cannot find function `_Py_IsImmortal` in crate `pyo3_ffi`
   --> src/util.rs:105:26
    |
105 |             if pyo3_ffi::_Py_IsImmortal($op) == 0 {
    |                          ^^^^^^^^^^^^^^ not found in `pyo3_ffi`
    |
   ::: src/deserialize/backend/yyjson.rs:270:21
    |
270 |                     reverse_pydict_incref!(pyval);
    |                     ----------------------------- in this macro invocation
    |
    = note: this error originates in the macro `reverse_pydict_incref` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0425]: cannot find function `_Py_IsImmortal` in crate `pyo3_ffi`
   --> src/util.rs:105:26
    |
105 |             if pyo3_ffi::_Py_IsImmortal($op) == 0 {
    |                          ^^^^^^^^^^^^^^ not found in `pyo3_ffi`
    |
   ::: src/deserialize/backend/yyjson.rs:290:17
    |
290 |                 reverse_pydict_incref!(pykey);
    |                 ----------------------------- in this macro invocation
    |
    = note: this error originates in the macro `reverse_pydict_incref` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0425]: cannot find function `_Py_IsImmortal` in crate `pyo3_ffi`
   --> src/util.rs:105:26
    |
105 |             if pyo3_ffi::_Py_IsImmortal($op) == 0 {
    |                          ^^^^^^^^^^^^^^ not found in `pyo3_ffi`
    |
   ::: src/deserialize/backend/yyjson.rs:291:17
    |
291 |                 reverse_pydict_incref!(pyval.as_ptr());
    |                 -------------------------------------- in this macro invocation
    |
    = note: this error originates in the macro `reverse_pydict_incref` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0422]: cannot find struct, variant or union type `PyObjectObRefcnt` in crate `pyo3_ffi`
  --> src/ffi/fragment.rs:75:38
   |
75 |                 ob_refcnt: pyo3_ffi::PyObjectObRefcnt { ob_refcnt: 0 },
   |                                      ^^^^^^^^^^^^^^^^ not found in `pyo3_ffi`

error[E0609]: no field `ob_refcnt` on type `pyo3_ffi::PyObject`
   --> src/util.rs:107:24
    |
107 |                 (*$op).ob_refcnt.ob_refcnt -= 1;
    |                        ^^^^^^^^^ unknown field
    |
   ::: src/deserialize/backend/yyjson.rs:261:21
    |
261 |                     reverse_pydict_incref!(pykey);
    |                     ----------------------------- in this macro invocation
    |
    = note: this error originates in the macro `reverse_pydict_incref` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0609]: no field `ob_refcnt` on type `pyo3_ffi::PyObject`
   --> src/util.rs:107:24
    |
107 |                 (*$op).ob_refcnt.ob_refcnt -= 1;
    |                        ^^^^^^^^^ unknown field
    |
   ::: src/deserialize/backend/yyjson.rs:262:21
    |
262 |                     reverse_pydict_incref!(pyval);
    |                     ----------------------------- in this macro invocation
    |
    = note: this error originates in the macro `reverse_pydict_incref` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0609]: no field `ob_refcnt` on type `pyo3_ffi::PyObject`
   --> src/util.rs:107:24
    |
107 |                 (*$op).ob_refcnt.ob_refcnt -= 1;
    |                        ^^^^^^^^^ unknown field
    |
   ::: src/deserialize/backend/yyjson.rs:269:21
    |
269 |                     reverse_pydict_incref!(pykey);
    |                     ----------------------------- in this macro invocation
    |
    = note: this error originates in the macro `reverse_pydict_incref` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0609]: no field `ob_refcnt` on type `pyo3_ffi::PyObject`
   --> src/util.rs:107:24
    |
107 |                 (*$op).ob_refcnt.ob_refcnt -= 1;
    |                        ^^^^^^^^^ unknown field
    |
   ::: src/deserialize/backend/yyjson.rs:270:21
    |
270 |                     reverse_pydict_incref!(pyval);
    |                     ----------------------------- in this macro invocation
    |
    = note: this error originates in the macro `reverse_pydict_incref` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0609]: no field `ob_refcnt` on type `pyo3_ffi::PyObject`
   --> src/util.rs:107:24
    |
107 |                 (*$op).ob_refcnt.ob_refcnt -= 1;
    |                        ^^^^^^^^^ unknown field
    |
   ::: src/deserialize/backend/yyjson.rs:290:17
    |
290 |                 reverse_pydict_incref!(pykey);
    |                 ----------------------------- in this macro invocation
    |
    = note: this error originates in the macro `reverse_pydict_incref` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0609]: no field `ob_refcnt` on type `pyo3_ffi::PyObject`
   --> src/util.rs:107:24
    |
107 |                 (*$op).ob_refcnt.ob_refcnt -= 1;
    |                        ^^^^^^^^^ unknown field
    |
   ::: src/deserialize/backend/yyjson.rs:291:17
    |
291 |                 reverse_pydict_incref!(pyval.as_ptr());
    |                 -------------------------------------- in this macro invocation
    |
    = note: this error originates in the macro `reverse_pydict_incref` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0560]: struct `pyo3_ffi::PyObject` has no field named `ob_refcnt`
  --> src/ffi/fragment.rs:75:17
   |
75 |                 ob_refcnt: pyo3_ffi::PyObjectObRefcnt { ob_refcnt: 0 },
   |                 ^^^^^^^^^ `pyo3_ffi::PyObject` does not have this field
   |
   = note: available fields are: `ob_tid`, `_padding`, `ob_mutex`, `ob_gc_bits`, `ob_ref_local`, `ob_ref_shared`

error[E0308]: mismatched types
  --> src/ffi/fragment.rs:88:19
   |
88 |         tp_flags: FRAGMENT_TP_FLAGS,
   |                   ^^^^^^^^^^^^^^^^^ expected `AtomicU64`, found `u64`
   |
help: call `Into::into` on this expression to convert `u64` into `AtomicU64`
   |
88 |         tp_flags: FRAGMENT_TP_FLAGS.into(),
   |                                    +++++++

error[E0369]: no implementation for `AtomicU64 & u64`
    --> src/util.rs:25:41
     |
25   |         unsafe { (((*$ob_type).tp_flags & pyo3_ffi::$flag) != 0) }
     |                    -------------------- ^ --------------- u64
     |                    |
     |                    AtomicU64
     |
    ::: src/serialize/obtype.rs:79:12
     |
79   |         if is_subclass_by_flag!(ob_type, Py_TPFLAGS_UNICODE_SUBCLASS) {
     |            ---------------------------------------------------------- in this macro invocation
     |
note: the foreign item type `AtomicU64` doesn't implement `BitAnd<u64>`
    --> /Users/goldbaum/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:3154:1
     |
3154 | / atomic_int! {
3155 | |     cfg(target_has_atomic = "64"),
3156 | |     cfg(target_has_atomic_equal_alignment = "64"),
3157 | |     stable(feature = "integer_atomics_stable", since = "1.34.0"),
...    |
3169 | |     u64 AtomicU64
3170 | | }
     | |_^ not implement `BitAnd<u64>`
     = note: this error originates in the macro `is_subclass_by_flag` which comes from the expansion of the macro `atomic_int` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0369]: no implementation for `AtomicU64 & u64`
    --> src/util.rs:25:41
     |
25   |         unsafe { (((*$ob_type).tp_flags & pyo3_ffi::$flag) != 0) }
     |                    -------------------- ^ --------------- u64
     |                    |
     |                    AtomicU64
     |
    ::: src/serialize/obtype.rs:81:19
     |
81   |         } else if is_subclass_by_flag!(ob_type, Py_TPFLAGS_LONG_SUBCLASS) {
     |                   ------------------------------------------------------- in this macro invocation
     |
note: the foreign item type `AtomicU64` doesn't implement `BitAnd<u64>`
    --> /Users/goldbaum/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:3154:1
     |
3154 | / atomic_int! {
3155 | |     cfg(target_has_atomic = "64"),
3156 | |     cfg(target_has_atomic_equal_alignment = "64"),
3157 | |     stable(feature = "integer_atomics_stable", since = "1.34.0"),
...    |
3169 | |     u64 AtomicU64
3170 | | }
     | |_^ not implement `BitAnd<u64>`
     = note: this error originates in the macro `is_subclass_by_flag` which comes from the expansion of the macro `atomic_int` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0369]: no implementation for `AtomicU64 & u64`
    --> src/util.rs:25:41
     |
25   |         unsafe { (((*$ob_type).tp_flags & pyo3_ffi::$flag) != 0) }
     |                    -------------------- ^ --------------- u64
     |                    |
     |                    AtomicU64
     |
    ::: src/serialize/obtype.rs:83:19
     |
83   |         } else if is_subclass_by_flag!(ob_type, Py_TPFLAGS_LIST_SUBCLASS) {
     |                   ------------------------------------------------------- in this macro invocation
     |
note: the foreign item type `AtomicU64` doesn't implement `BitAnd<u64>`
    --> /Users/goldbaum/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:3154:1
     |
3154 | / atomic_int! {
3155 | |     cfg(target_has_atomic = "64"),
3156 | |     cfg(target_has_atomic_equal_alignment = "64"),
3157 | |     stable(feature = "integer_atomics_stable", since = "1.34.0"),
...    |
3169 | |     u64 AtomicU64
3170 | | }
     | |_^ not implement `BitAnd<u64>`
     = note: this error originates in the macro `is_subclass_by_flag` which comes from the expansion of the macro `atomic_int` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0369]: no implementation for `AtomicU64 & u64`
    --> src/util.rs:25:41
     |
25   |         unsafe { (((*$ob_type).tp_flags & pyo3_ffi::$flag) != 0) }
     |                    -------------------- ^ --------------- u64
     |                    |
     |                    AtomicU64
     |
    ::: src/serialize/obtype.rs:85:19
     |
85   |         } else if is_subclass_by_flag!(ob_type, Py_TPFLAGS_DICT_SUBCLASS) {
     |                   ------------------------------------------------------- in this macro invocation
     |
note: the foreign item type `AtomicU64` doesn't implement `BitAnd<u64>`
    --> /Users/goldbaum/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:3154:1
     |
3154 | / atomic_int! {
3155 | |     cfg(target_has_atomic = "64"),
3156 | |     cfg(target_has_atomic_equal_alignment = "64"),
3157 | |     stable(feature = "integer_atomics_stable", since = "1.34.0"),
...    |
3169 | |     u64 AtomicU64
3170 | | }
     | |_^ not implement `BitAnd<u64>`
     = note: this error originates in the macro `is_subclass_by_flag` which comes from the expansion of the macro `atomic_int` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0369]: no implementation for `AtomicU64 & u64`
    --> src/util.rs:25:41
     |
25   |         unsafe { (((*$ob_type).tp_flags & pyo3_ffi::$flag) != 0) }
     |                    -------------------- ^ --------------- u64
     |                    |
     |                    AtomicU64
     |
    ::: src/serialize/per_type/list.rs:50:20
     |
50   |                 || is_subclass_by_flag!(ob_type!(ptr), Py_TPFLAGS_LIST_SUBCLASS)
     |                    ------------------------------------------------------------- in this macro invocation
     |
note: the foreign item type `AtomicU64` doesn't implement `BitAnd<u64>`
    --> /Users/goldbaum/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:3154:1
     |
3154 | / atomic_int! {
3155 | |     cfg(target_has_atomic = "64"),
3156 | |     cfg(target_has_atomic_equal_alignment = "64"),
3157 | |     stable(feature = "integer_atomics_stable", since = "1.34.0"),
...    |
3169 | |     u64 AtomicU64
3170 | | }
     | |_^ not implement `BitAnd<u64>`
     = note: this error originates in the macro `is_subclass_by_flag` which comes from the expansion of the macro `atomic_int` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0369]: no implementation for `AtomicU64 & u64`
    --> src/util.rs:25:41
     |
25   |         unsafe { (((*$ob_type).tp_flags & pyo3_ffi::$flag) != 0) }
     |                    -------------------- ^ --------------- u64
     |                    |
     |                    AtomicU64
     |
    ::: src/serialize/per_type/list.rs:69:20
     |
69   |                 || is_subclass_by_flag!(ob_type!(ptr), Py_TPFLAGS_TUPLE_SUBCLASS)
     |                    -------------------------------------------------------------- in this macro invocation
     |
note: the foreign item type `AtomicU64` doesn't implement `BitAnd<u64>`
    --> /Users/goldbaum/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:3154:1
     |
3154 | / atomic_int! {
3155 | |     cfg(target_has_atomic = "64"),
3156 | |     cfg(target_has_atomic_equal_alignment = "64"),
3157 | |     stable(feature = "integer_atomics_stable", since = "1.34.0"),
...    |
3169 | |     u64 AtomicU64
3170 | | }
     | |_^ not implement `BitAnd<u64>`
     = note: this error originates in the macro `is_subclass_by_flag` which comes from the expansion of the macro `atomic_int` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0425]: cannot find function, tuple struct or tuple variant `PyModule_Add` in this scope
  --> src/lib.rs:40:9
   |
40 |         PyModule_Add($mptr, $name.as_ptr() as *const c_char, $obj);
   |         ^^^^^^^^^^^^ not found in this scope
...
79 |         add!(mptr, "__version__\0", pyversion);
   |         -------------------------------------- in this macro invocation
   |
   = note: this error originates in the macro `add` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0425]: cannot find function, tuple struct or tuple variant `PyModule_Add` in this scope
   --> src/lib.rs:40:9
    |
40  |         PyModule_Add($mptr, $name.as_ptr() as *const c_char, $obj);
    |         ^^^^^^^^^^^^ not found in this scope
...
102 |         add!(mptr, "dumps\0", func);
    |         --------------------------- in this macro invocation
    |
    = note: this error originates in the macro `add` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0425]: cannot find function, tuple struct or tuple variant `PyModule_Add` in this scope
   --> src/lib.rs:40:9
    |
40  |         PyModule_Add($mptr, $name.as_ptr() as *const c_char, $obj);
    |         ^^^^^^^^^^^^ not found in this scope
...
119 |         add!(mptr, "loads\0", func);
    |         --------------------------- in this macro invocation
    |
    = note: this error originates in the macro `add` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0425]: cannot find function, tuple struct or tuple variant `PyModule_Add` in this scope
   --> src/lib.rs:40:9
    |
40  |         PyModule_Add($mptr, $name.as_ptr() as *const c_char, $obj);
    |         ^^^^^^^^^^^^ not found in this scope
...
122 |     add!(mptr, "Fragment\0", typeref::FRAGMENT_TYPE as *mut PyObject);
    |     ----------------------------------------------------------------- in this macro invocation
    |
    = note: this error originates in the macro `add` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0425]: cannot find function, tuple struct or tuple variant `PyModule_Add` in this scope
   --> src/lib.rs:40:9
    |
40  |         PyModule_Add($mptr, $name.as_ptr() as *const c_char, $obj);
    |         ^^^^^^^^^^^^ not found in this scope
...
151 |     add!(mptr, "JSONDecodeError\0", typeref::JsonDecodeError);
    |     --------------------------------------------------------- in this macro invocation
    |
    = note: this error originates in the macro `add` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0425]: cannot find function, tuple struct or tuple variant `PyModule_Add` in this scope
   --> src/lib.rs:40:9
    |
40  |         PyModule_Add($mptr, $name.as_ptr() as *const c_char, $obj);
    |         ^^^^^^^^^^^^ not found in this scope
...
152 |     add!(mptr, "JSONEncodeError\0", typeref::JsonEncodeError);
    |     --------------------------------------------------------- in this macro invocation
    |
    = note: this error originates in the macro `add` (in Nightly builds, run with -Z macro-backtrace for more info)

Some errors have detailed explanations: E0308, E0369, E0422, E0425, E0560, E0609.
For more information about an error, try `rustc --explain E0308`.
The following warnings were emitted during compilation:

warning: [email protected]: In file included from include/yyjson/yyjson.c:23:
warning: [email protected]: include/yyjson/yyjson.h:938:60: warning: parameter 'flg' set but not used [-Wunused-but-set-parameter]
warning: [email protected]:   938 |                                           yyjson_read_flag flg) {
warning: [email protected]:       |                                                            ^
warning: [email protected]: include/yyjson/yyjson.c:5268:29: warning: variable 'pos' set but not used [-Wunused-but-set-variable]
warning: [email protected]:  5268 |     u8 *src = ++cur, *dst, *pos;
warning: [email protected]:       |                             ^
warning: [email protected]: include/yyjson/yyjson.c:6586:45: warning: parameter 'flg' set but not used [-Wunused-but-set-parameter]
warning: [email protected]:  6586 |                            yyjson_read_flag flg,
warning: [email protected]:       |                                             ^
warning: [email protected]: 3 warnings generated.

error: could not compile `orjson` (lib test) due to 27 previous errors

It looks like you're using _Py_IsImmortal, which PyO3 removed because it's private but you could add it back if you want. There's also a few places where ob_refcnt is getting touched directly, those will need to be replaced by the macros. If the refcounts are used in a thread-unsafe way, you'll need to rethink that. You'll also have to deal with the fact that PyTypeObject.tp_flags is an atomic in the free-threaded build.

Of course other thread safety issues that the authors of this project are aware of will need to be fixed as well.

I'm very happy to help out with updating orjson to support the free-threaded build and am actively working on updating python modules that depend on PyO3 as part of the work tracked at https://py-free-threading.github.io/tracking/. Please let me know what help you'd like.

@ijl
Copy link
Owner

ijl commented Nov 12, 2024

For 3.13.0, the normal test suite passes, but any kind of interesting concurrent workload segfaults due to issues I can't seem to avoid. The requirement for wheels would be no known issues. That might be 3.13.1 in December or 3.14.

@ngoldbaum
Copy link
Author

due to issues I can't seem to avoid

If you can share a little more detail here both myself and @davidhewitt are interested in working on this.

orjson shows up in a lot of dependency trees so it would be good to get it working. It also uses the C API directly and touches some CPython internals that might have different semantics under the free-threaded ABI, so I could understand why there might be issues on multithreaded workflows.

@itamaro
Copy link

itamaro commented Nov 25, 2024

fwiw, trying to use orjson with 3.13, I spotted usage of _PyDict_Contains_KnownHash in orjson. that function is no longer exported in 3.13 (as of python/cpython#107145)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants