Various CI fixes

[qstokkink]

This PR targets master because that is where the issue of missing wheel types occurs (the original target of this PR).

There are a few layers of issues that have surfaced. This PR addresses the following:

• continuous-integration/appveyor/pr cannot find boost (c:\Libraries\boost_1_77_0 does not exist, only boost_1_83_0, boost_1_84_0, boost_1_85_0 exist on the image). Resolution: switched to boost_1_85_0 boost_1_83_0 (see below).
• Cirrus CI / Build macOS arm64 wheels cannot find types. Resolution: fixed in mypy-tox.ini.
• types are not in the appropriate location in the built wheel. Resolution: fixed in bindings/python/setup.py.
• Test macOS arm64 "flaky test" test_privacy fails with Cputime limit exceeded: 24. Does not work with boost_1_85_0, boost_1_84_0. Mitigation 1: downgraded to boost_1_83_0. Mitigation 2: allow errors other than connection_refused when waiting for Python to start during testing. Even so, this test is still unstable.
• continuous-integration/appveyor/pr fails with EXIT STATUS: -1073741515 for variant=release. Resolution: included the OpenSSL bin directory in the path.
• Tests (webtorrent=on deprecated-functions=off) failing with set but not used errors. Resolution: wont fix. Bumping the version of libdatachannel to 0.19.0 should fix this error but it seems to cause other issues.
• build dist (ubuntu-22.04) failing due to missing install_data/libtorrent in tarball. Resolution: added the typing files to the Makefile.

[qstokkink]

@arvidn I managed to find a configuration that pleases the first 6 CI checks. Could you O.K. me for the following 8?

[arvidn]

py run-test: commands[0] | 'D:\a\libtorrent\libtorrent\.tox\py\Scripts\python' setup.py build_ext --b2-args 'asserts=on invariant-checks=full' bdist_wheel
Traceback (most recent call last):
  File "setup.py", line 510, in <module>
    shutil.copytree("install_data/libtorrent", "libtorrent", dirs_exist_ok=True)
TypeError: copytree() got an unexpected keyword argument 'dirs_exist_ok'

[qstokkink]

@arvidn sorry about that, that argument is Python 3.8+. I now switched to the old way of catching a FileExistsError. Could you give me another full run?

As a side note: I'm not sure why __init__.pyi and py.typed are not just in the bindings/python/libtorrent directory instead of bindings/python/install_data/libtorrent. That would avoid the copy altogether. For now, I'll focus on making the existing stuff work instead of changing its intention though.

[arvidn]

I suppose it might be time to drop support for python 3.7 as well. I'm surprised any CI image is still using it.

[qstokkink]

That seems sensible. Upgrading to different Python 3.x series is not that bad for users (at least not as bad as the whole Python 2➡️3 switch 🙂) and Python 3.7 is no longer receiving security updates.

On that topic, on your master branch, Python 3.6 is still being used:

libtorrent/.github/workflows/python.yml

Line 55 in 6f4b7b0

python-version: 3.6

That aside, the two failures from the previous "full" run should be fixed now. Could you approve another run?

[arvidn]

it looks like the datachannel submodule was accidentally changed in your latest push

[arvidn]

oh, was that on purpose?

[qstokkink]

@arvidn Yes, that was on purpose, in an attempt to solve this: https://github.com/arvidn/libtorrent/actions/runs/9661198749/job/26661959014

click me for excerpt

[ 37%] Building C object usrsctplib/CMakeFiles/usrsctp.dir/netinet/sctp_output.c.o /Users/runner/work/libtorrent/libtorrent/deps/libdatachannel/deps/usrsctp/usrsctplib/netinet/sctp_indata.c:3323:6: error: variable 'tot_retrans' set but not used [-Werror,-Wunused-but-set-variable] int tot_retrans = 0; ^ 1 error generated. make[3]: *** [usrsctplib/CMakeFiles/usrsctp.dir/netinet/sctp_indata.c.o] Error 1 make[3]: *** Waiting for unfinished jobs.... /Users/runner/work/libtorrent/libtorrent/deps/libdatachannel/deps/usrsctp/usrsctplib/netinet/sctp_output.c:10018:33: error: variable 'cnt_thru' set but not used [-Werror,-Wunused-but-set-variable] int no_fragmentflg, bundle_at, cnt_thru; ^ 1 error generated. make[3]: *** [usrsctplib/CMakeFiles/usrsctp.dir/netinet/sctp_output.c.o] Error 1 make[2]: *** [usrsctplib/CMakeFiles/usrsctp.dir/all] Error 2 make[1]: *** [usrsctplib/CMakeFiles/usrsctp.dir/rule] Error 2 make: *** [usrsctp] Error 2 cp: /Users/runner/work/libtorrent/libtorrent/deps/libdatachannel/deps/usrsctp/build-debug-mode/usrsctplib/libusrsctp.a: No such file or directory (cd /Users/runner/work/libtorrent/libtorrent/deps/libdatachannel/deps/usrsctp && mkdir -p build-debug-mode && cd build-debug-mode && cmake -DCMAKE_BUILD_TYPE=debug-mode -DCMAKE_C_FLAGS="-fPIC -Wno-unknown-warning-option -Wno-format-truncation" -Dsctp_build_shared_lib=0 -Dsctp_build_programs=0 -Dsctp_inet=0 -Dsctp_inet6=0 .. && make -j2 usrsctp) cp /Users/runner/work/libtorrent/libtorrent/deps/libdatachannel/deps/usrsctp/build-debug-mode/usrsctplib/libusrsctp.a ../deps/libdatachannel/bin/libusrsctp.a ...failed Jamfile</Users/runner/work/libtorrent/libtorrent/deps/libdatachannel>.make_libusrsctp ../deps/libdatachannel/bin/libusrsctp.a...

The changelog of libdatachannel suggests that bumping up from 0.18.1 to 0.18.5 (EDIT: 0.19.0) will resolve this. If this is a no-go for you, I'll drop the commit. Of course, I'll also drop the commit if this does not end up fixing the issue at hand.

EDIT: Hold on, I made a mistake. The issue was fixed in libdatachannel 0.19.0.

[qstokkink]

I dropped the libdatachannel bump commit because it seems to have caused the Mac tests to fail (it may also be a random failure though).

[qstokkink]

Apart from the MacOS / Tests (webtorrent=on deprecated-functions=off), which consistently fails, the remaining checks only seem to fail intermittently.

I think this is a good place to stop adding to this PR and I'll mark it as ready for review when the checks are done running.

[arvidn]

thanks!

[qstokkink]

You're welcome. 👍 I would've liked to fix all of your problems in one fell swoop but I'm going on vacation next week, so I can't commit to that right now.

Short term, I think it's a good idea to see if you can bump up libdatachannel to 0.19.0 or beyond. Then, all your CI check at least have a chance at passing.