Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python3Packages.pynacl: fix cross compilation #135447

Merged
merged 1 commit into from
Aug 23, 2021
Merged

python3Packages.pynacl: fix cross compilation #135447

merged 1 commit into from
Aug 23, 2021

Conversation

legendofmiracles
Copy link
Contributor

Motivation for this change

Closes #135149

Similar PR's should probably be created for every other derivation that also lists cffi in such a cross-compiling unfriendly way.

image

image

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • 21.11 Release Notes (or backporting 21.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@legendofmiracles legendofmiracles changed the title pynacl: fix cross compilation python3Packages.pynacl: fix cross compilation Aug 23, 2021
@legendofmiracles
Copy link
Contributor Author

Should this go in staging? It seems to be that many packages rely on this library.

@SuperSandro2000
Copy link
Member

Should this go in staging? It seems to be that many packages rely on this library.

Under 500 is fine for master.

@SuperSandro2000
Copy link
Member

Similar PR's should probably be created for every other derivation that also lists cffi in such a cross-compiling unfriendly way.

So cffi should always be in nativePropa..?

@legendofmiracles
Copy link
Contributor Author

legendofmiracles commented Aug 23, 2021 •
edited
Loading

For python packages, it seems like yes. I'm not sure why this package then relies on six when doing that but I just tried another python package that relies on cffi (brotlicffi) and it was fixed by moving it to propagatedNative... without introducing any new deps:
image

@SuperSandro2000
Copy link
Member

For python packages, it seems like yes. I'm not sure why this package then relies on six but I just tried another python package that relies on cffi (brotlicffi) and it was fixed by moving it to propagatedNative... without introducing any new deps:
image

I think that might be worth a treewide PR after this one.

@legendofmiracles
Copy link
Contributor Author

I won't have time this week for doing a larger PR, but what are my options for doing a large scale PR? Find every mention and fix it manually? Find every wrong declaration and open a tracking issue, pinging the maintainers? Or can I fully automate it, including testing?

@SuperSandro2000
Copy link
Member

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).

Result of nixpkgs-review pr 135447 run on x86_64-linux 1

19 packages marked as broken and skipped:
  • mlflow-server
  • python38Packages.clifford
  • python38Packages.dask-jobqueue
  • python38Packages.dask-xgboost
  • python38Packages.mlflow
  • python38Packages.pathy
  • python38Packages.rl-coach
  • python38Packages.spacy
  • python38Packages.spacy-transformers
  • python38Packages.textacy
  • python39Packages.clifford
  • python39Packages.dask-jobqueue
  • python39Packages.dask-xgboost
  • python39Packages.mlflow
  • python39Packages.pathy
  • python39Packages.rl-coach
  • python39Packages.spacy
  • python39Packages.spacy-transformers
  • python39Packages.textacy
11 packages failed to build and already failed to build on hydra master:
  • home-assistant: log was empty
  • mailman: log was empty
  • paperless-ng: log was empty
  • python38Packages.mailman: log was empty
  • python38Packages.mailman-hyperkitty: log was empty
  • python38Packages.tokenizers: log was empty
  • python38Packages.transformers: log was empty
  • python39Packages.mailman: log was empty
  • python39Packages.mailman-hyperkitty: log was empty
  • python39Packages.tokenizers: log was empty
  • python39Packages.transformers: log was empty
1 package failed to build and are new build failure:
  • ib-controller: log was empty
229 packages built:
  • Fabric (python39Packages.Fabric)
  • afew
  • ansible (ansible_2_11 ,python39Packages.ansible-core)
  • ansible-lint (python39Packages.ansible-lint)
  • ansible_2_10 (python39Packages.ansible-base)
  • ansible_2_9 (python39Packages.ansible)
  • arion
  • aws-sam-cli
  • azure-cli
  • b4
  • buildbot (python39Packages.buildbot)
  • buildbot-full (python39Packages.buildbot-full)
  • buildbot-ui (python39Packages.buildbot-ui)
  • cinnamon.warpinator
  • cligh
  • crowbar
  • deja-dup
  • dkimpy (python39Packages.dkimpy)
  • docker-compose
  • duplicity
  • duply
  • dvc-with-remotes
  • etebase-server (python38Packages.etebase-server ,python39Packages.etebase-server)
  • fdroidserver
  • gnome-keysign
  • grab-site
  • magic-wormhole (python39Packages.magic-wormhole)
  • mathlibtools (python39Packages.mathlibtools)
  • matrix-synapse
  • matrix-synapse-plugins.matrix-synapse-mjolnir-antispam
  • mycli
  • onlykey-agent
  • onlykey-cli
  • pwntools (python39Packages.pwntools)
  • python38Packages.Fabric
  • python38Packages.PyGithub
  • python38Packages.ansible
  • python38Packages.ansible-base
  • python38Packages.ansible-core
  • python38Packages.ansible-kernel
  • python38Packages.ansible-lint
  • python38Packages.ansible-runner
  • python38Packages.aplpy
  • python38Packages.asdf
  • python38Packages.authheaders
  • python38Packages.autobahn
  • python38Packages.batchgenerators
  • python38Packages.buildbot
  • python38Packages.buildbot-full
  • python38Packages.buildbot-ui
  • python38Packages.caffe
  • python38Packages.celery
  • python38Packages.channels
  • python38Packages.channels-redis
  • python38Packages.criticality-score
  • python38Packages.daphne
  • python38Packages.dask
  • python38Packages.dask-gateway
  • python38Packages.dask-glm
  • python38Packages.dask-image
  • python38Packages.dask-ml
  • python38Packages.dask-mpi
  • python38Packages.datasets
  • python38Packages.datashader
  • python38Packages.discordpy
  • python38Packages.distributed
  • python38Packages.django-raster
  • python38Packages.djmail
  • python38Packages.dkimpy
  • python38Packages.docker
  • python38Packages.dockerspawner
  • python38Packages.entrance-with-router-features
  • python38Packages.fastparquet
  • python38Packages.flower
  • python38Packages.fortiosapi
  • python38Packages.fsspec
  • python38Packages.gcsfs
  • python38Packages.gensim
  • python38Packages.glymur
  • python38Packages.howdoi
  • python38Packages.image-match
  • python38Packages.imagecorruptions
  • python38Packages.imgaug
  • python38Packages.intake
  • python38Packages.junos-eznc
  • python38Packages.jupyter-repo2docker
  • python38Packages.keep
  • python38Packages.labgrid
  • python38Packages.magic-wormhole
  • python38Packages.magic-wormhole-mailbox-server
  • python38Packages.mask-rcnn
  • python38Packages.mathlibtools
  • python38Packages.moto
  • python38Packages.ncclient
  • python38Packages.paramiko
  • python38Packages.patator
  • python38Packages.pims
  • python38Packages.proxmoxer
  • python38Packages.pwntools
  • python38Packages.pyfftw
  • python38Packages.pymacaroons
  • python38Packages.pynacl
  • python38Packages.pyregion
  • python38Packages.pysftp
  • python38Packages.pytest-ansible
  • python38Packages.pytest-astropy
  • python38Packages.pytest-astropy-header
  • python38Packages.reproject
  • python38Packages.robotframework-sshlibrary
  • python38Packages.s3fs
  • python38Packages.scikitimage
  • python38Packages.scp
  • python38Packages.signedjson
  • python38Packages.slack-sdk
  • python38Packages.slicedimage
  • python38Packages.smart-open
  • python38Packages.sparse
  • python38Packages.spectral-cube
  • python38Packages.sshtunnel
  • python38Packages.streamz
  • python38Packages.stumpy
  • python38Packages.stytra
  • python38Packages.sunpy
  • python38Packages.tensorboardx
  • python38Packages.tensorly
  • python38Packages.tern
  • python38Packages.webssh
  • python39Packages.PyGithub
  • python39Packages.ansible-kernel
  • python39Packages.ansible-runner
  • python39Packages.aplpy
  • python39Packages.asdf
  • python39Packages.authheaders
  • python39Packages.autobahn
  • python39Packages.batchgenerators
  • python39Packages.caffe
  • python39Packages.celery
  • python39Packages.channels
  • python39Packages.channels-redis
  • python39Packages.criticality-score
  • python39Packages.daphne
  • python39Packages.dask
  • python39Packages.dask-gateway
  • python39Packages.dask-glm
  • python39Packages.dask-image
  • python39Packages.dask-ml
  • python39Packages.dask-mpi
  • python39Packages.datasets
  • python39Packages.datashader
  • python39Packages.discordpy
  • python39Packages.distributed
  • python39Packages.django-raster
  • python39Packages.djmail
  • python39Packages.docker
  • python39Packages.dockerspawner
  • python39Packages.entrance-with-router-features
  • python39Packages.fastparquet
  • python39Packages.flower
  • python39Packages.fortiosapi
  • python39Packages.fsspec
  • python39Packages.gcsfs
  • python39Packages.gensim
  • python39Packages.glymur
  • python39Packages.howdoi
  • python39Packages.image-match
  • python39Packages.imagecorruptions
  • python39Packages.imgaug
  • python39Packages.intake
  • python39Packages.junos-eznc
  • python39Packages.jupyter-repo2docker
  • python39Packages.keep
  • python39Packages.labgrid
  • python39Packages.magic-wormhole-mailbox-server
  • python39Packages.mask-rcnn
  • python39Packages.moto
  • python39Packages.ncclient
  • python39Packages.paramiko
  • python39Packages.pims
  • python39Packages.proxmoxer
  • python39Packages.pyfftw
  • python39Packages.pymacaroons
  • python39Packages.pynacl
  • python39Packages.pyregion
  • python39Packages.pysftp
  • python39Packages.pytest-ansible
  • python39Packages.pytest-astropy
  • python39Packages.pytest-astropy-header
  • python39Packages.reproject
  • python39Packages.robotframework-sshlibrary
  • python39Packages.s3fs
  • python39Packages.scikitimage
  • python39Packages.scp
  • python39Packages.signedjson
  • python39Packages.slack-sdk
  • python39Packages.slicedimage
  • python39Packages.smart-open
  • python39Packages.sparse
  • python39Packages.spectral-cube
  • python39Packages.sshtunnel
  • python39Packages.streamz
  • python39Packages.stumpy
  • python39Packages.stytra
  • python39Packages.sunpy
  • python39Packages.tensorboardx
  • python39Packages.tensorly
  • python39Packages.tern
  • webssh (python39Packages.webssh)
  • qutebrowser
  • remarkable-mouse
  • rmview
  • run-scaled
  • snakemake
  • sourcehut.buildsrht
  • sourcehut.coresrht
  • sourcehut.dispatchsrht
  • sourcehut.gitsrht
  • sourcehut.hgsrht
  • sourcehut.hubsrht
  • sourcehut.listssrht
  • sourcehut.mansrht
  • sourcehut.metasrht
  • sourcehut.pastesrht
  • sourcehut.todosrht
  • tahoe-lafs
  • toil
  • tts
  • vorta
  • xpra
  • xpraWithNvenc

@SuperSandro2000 SuperSandro2000 added the 6.topic: cross-compilation Building packages on a different platform than they will be used on label Aug 23, 2021
@SuperSandro2000
Copy link
Member

I won't have time this week for doing a larger PR

Thats okay. It is only cross compile which is not super important at this point in time.

but what are my options for doing a large scale PR? Find every mention and fix it manually?

Or most cases where it is wrong and is easily fixed. We can either leave comments on more complex cases or leave them untouched for now.

Find every wrong declaration and open a tracking issue, pinging the maintainers? Or can I fully automate it, including testing?

I think we can do one treewide commit or one commit per package in one PR. I don't think we need to wait for maintainer feedback. If there are other issues which prevent the package from cross compiling it might be good to do a separate PR.

@SuperSandro2000 SuperSandro2000 merged commit 233d204 into NixOS:master Aug 23, 2021
@legendofmiracles legendofmiracles mentioned this pull request Oct 20, 2021
Merged
12 tasks
@legendofmiracles
Copy link
Contributor Author

legendofmiracles commented Oct 20, 2021 •
edited
Loading

@SuperSandro2000 I've opened a tree-wide PR (#142301), and also discovered an issue with how I did this PR
This derivation should list cffi in both native and the normal propagated to not have import errors after the build on the target platform
(EDIT: i just tested it and it doesn't seem to happen with pynacl 🤔)

akonradi added a commit to akonradi/nixpkgs that referenced this pull request Sep 12, 2023
@akonradi
Move cffi to propagatedNativeBuildInputs
8b9b463 
Follow the approach of NixOS#135447 and move the cffi package from
propagatedBuildInputs to propagatedNativeBuildInputs. Also disable
passing the -msse2 flag to the compiler when cross-compiling. This is a
conservative approach that mirrors that in the argon2-cffi package.
@akonradi akonradi mentioned this pull request Sep 12, 2023
Open
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FRidh FRidh Awaiting requested review from FRidh

@jonringer jonringer Awaiting requested review from jonringer

Assignees
No one assigned
Labels
6.topic: cross-compilation Building packages on a different platform than they will be used on 6.topic: python 10.rebuild-darwin: 101-500 10.rebuild-linux: 101-500
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

python3Packages.pynacl: doesn't cross-compile
2 participants
@legendofmiracles @SuperSandro2000