Feature/nix support #120

jake-arkinstall · 2024-01-29T17:06:47Z

Description

Added nix support, The nix flake fetches the pytket flake to fetch pytket, and uses a custom derivation to generate pyqir. mypy and pytest are run as part of nix flake check, which is performed in CI (currently set to every sunday at 2am).

At the moment, tket and pytket will be built. The speed of performing the flake check will be improved significantly if we use cachix.

To try out this flake:

nix develop github:CQCL/pytket-qir/feature/nix-support

Then invoke python. pytket.qir is importable and working, e.g.:

$ nix develop github:CQCL/pytket-qir/feature/nix-support

$ python3
Python 3.11.7 (main, Dec  4 2023, 18:10:11) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pytket.circuit import Circuit
>>> from pytket.qir.conversion.api import QIRFormat, pytket_to_qir
>>> c = Circuit(2)
>>> c.H(0)
[H q[0]; ]
>>> c.H(1)
[H q[0]; H q[1]; ]
>>> c.CX(0, 1)
[H q[0]; H q[1]; CX q[0], q[1]; ]
>>> c.H(0)
[H q[0]; H q[1]; CX q[0], q[1]; H q[0]; ]
>>> c.H(1)
[H q[0]; H q[1]; CX q[0], q[1]; H q[0]; H q[1]; ]
>>> print(pytket_to_qir(c, name="example", qir_format=QIRFormat.STRING))
; ModuleID = 'example'
source_filename = "example"

%Qubit = type opaque
%Result = type opaque
; ... and so on

After the merge of this PR, the development shell will be accessible with nix develop github:CQCL/pytket-qir

Checklist

I have performed a self-review of my code.
I have commented hard-to-understand parts of my code.
I have made corresponding changes to the public API documentation. (N/A)
I have added tests that prove my fix is effective or that my feature works.
I have updated the changelog with any user-facing changes. (N/A)

cqc-melf

Thank you for the suggestion!
I am not familiar with nix, so I have added a few questions / comments

cqc-melf · 2024-01-29T18:06:03Z

.github/workflows/build_with_nix.yml

@@ -0,0 +1,20 @@
+name: build with nix
+on:
+  schedule:


Can you give some details why we would not want to run this on PRs? In case of changes we would like to know that this fails?

Ah, the moment we don't have a Cachix subscription, so we sint have a central store to place things that we have built before. This includes tket - so when this CI runs, it will need to build tket first.

Once we have a caching subscription, the latest tket build will already be available, so building and testing pytket-qir will just be a case of bundling the python package and running checks on it. Then, if we store the result in cachix, anyone can load up a pre-built environment with tket, pytket and pytket-qir very quickly.

Thank you for all the details!

cqc-melf · 2024-01-29T18:06:24Z

flake.lock

Is this a file we want to have in the git?

Yes. The flake.lock pins the nix flake dependencies to well-defined versions, which helps keep the repository and users of the repository in sync.

It's feasible that a future commit to nixpkgs e.g. breaks mypy, or updates scipy with a breaking change. By keeping the flake.lock file in the repository, maintainers of this repository get to decide when nixpkgs, tket etc can get updated, by pushing changes created by invoking nix flake update.

cqc-melf · 2024-01-29T18:07:21Z

nix-support/pyqir.nix

+  llvm = super.llvm_14;
+  llvm-v-major = lib.versions.major llvm.version;
+  llvm-v-minor = builtins.substring 0 1 (lib.versions.minor llvm.version);
+  pyqir-version = "0.10.0";


Is it possible to import the version from the python packages here, or do we now need to update this in two places?

In this case it is customary to provide the exact git revision to pull, then specify the build process, as pyqir isn't available on nixpkgs.

I can update it to scan setup.py for a specific version, though if it changes to using a loose version match then the process will fail.

Though I will add that I am committed to supporting the nix support for this repository and others, so if a change breaks the nix support, or if a version update is required, I'm absolutely happy to action it.

I case I am doing the update, is there anything else I need to do besides updating the version in this file as well?

cqc-melf

Do you have plans to add nix for the other extensions as well?

cqc-melf · 2024-01-30T11:44:46Z

.github/workflows/build_with_nix.yml

@@ -0,0 +1,20 @@
+name: build with nix
+on:
+  schedule:


Thank you for all the details!

cqc-melf · 2024-01-30T11:46:22Z

nix-support/pyqir.nix

+  llvm = super.llvm_14;
+  llvm-v-major = lib.versions.major llvm.version;
+  llvm-v-minor = builtins.substring 0 1 (lib.versions.minor llvm.version);
+  pyqir-version = "0.10.0";


I case I am doing the update, is there anything else I need to do besides updating the version in this file as well?

cqc-melf · 2024-01-30T11:59:38Z

And it would be nice if you could add something to the changelog?

jake-arkinstall · 2024-01-30T14:32:14Z

For some reason my github app isnt letting my reply in thread, so:

I case I am doing the update, is there anything else I need to do besides updating the version in this file as well?

If the new version requires to change to the setup process, then this is my usual update process:

Reset the hash to an empty string: ""
Update the version number to an available tag of the repository
Run "nix build"

The nix build log will then emit a warning along the lines of:
"Empty hash provided, assuming sha256 AAAAAAAA...."

Once it has built the package it will calculate the actual hash. It will emit an error along the lines of:

Incorrect hash.
    Specified: AAAAAAAA.....
    Got: "ABCDEF1234....."

What I do then is add the provided hash back into the nix file and run nix build again.

It should immediately notice that it has already seen a derivation with this source, build instructions, and output hash, and resolve to the package it previously built.

The hash thing is annoying but extremely useful. If Nix can verify the hash then it knows if it has seen it before and lets it skip steps.

Likewise, subsequent runs on that machine will serve that result immediately. If we get a cachix subscription, we can upload the resulting derivation there, and any user that needs it (as specified by the matching source, build instructions and output hash) will be offered a ready-to-go downloadable package instead of going through the build and test process.

Will you do the same with other packages?

Indeed! I was asked to look at pytket-quantinuum and pytket-qir is a dependency of it (in the testing pipeline) so I started here.

This, with the above about the caching possibilities, is quite exciting.

Cachix is when the real benefit of the nix integration will kick in: users will be able to run nix develop github:CQCL/pytket-foo on any Linux or Mac Silicon machine and everything from symengine to tket, pyket to pyqir, and everything they need, will be served over a download with no additional building or configuration required. One command and after a few seconds they'll be ready to go.

It doesn't stop at environments, but can also launch tools. So we could possibly have a one-liner to launch a Jupiter notebook pre-filled with demo code, with the environment all loaded up, ready to showcase. But this is all future ifs and maybes, and I don't want to push it in a direction you don't want to take it in.

cqc-melf · 2024-01-30T14:58:21Z

Thank you for all the information! I think it would be helpful if you could just copy this into a readme file and add this to the repo as well. Just to make sure this is documented somewhere.

Happy to review your PR on pytket-quantinuum as well.

cqc-melf · 2024-01-30T14:58:55Z

Maybe it would be good if we could add this to pytket-qiskit as well? As they are both the most used extensions?

jake-arkinstall · 2024-01-30T15:29:43Z

That is my next challenge after I've got pytket-quantinuum in. There are qiskit libraries already available on the nix store, however they are marked as broken due to conflicts with symengine. So the qiskit integration PR might be a bit more substantial of a PR than this one, depending on how it goes.

…

On Tue, 30 Jan 2024, 14:59 cqc-melf, ***@***.***> wrote: Maybe it would be good if we could add this to pytket-qiskit as well? As they are both the most used extensions? — Reply to this email directly, view it on GitHub <#120 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APSUR26F3C7Q6DHU2OZTNMDYREDDTAVCNFSM6AAAAABCPZQNWKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJXGA3TQNRRGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

cqc-melf · 2024-03-05T14:09:15Z

@jake-arkinstall is this ready for re-review?

jake-arkinstall · 2024-03-06T11:43:19Z

@jake-arkinstall is this ready for re-review?

Yes, it is! Although as some movement looks to have been made on the cachix side it might be worth waiting for that to land in tket itself, as it'll speed things up considerably (this repo can then rely on the tket's cached outputs instead of forcing a build).

cqc-alec · 2024-06-07T08:24:59Z

Please rebase and retarget to main which is now the default branch.

Added a nix flake and derivations for pyqir and pytket-qir, using the tket flake from github for pytket module inclusion. The tket flake's nixpkgs is overridden with the current package's flake to avoid unnecessary download, and to help spot compatibility problems more easily. Currently mypy tests are disabled because there are several type errors that appear. I do not know if these are expected, or if it is a problem with this derivation. I have also tested with my `feature/nix-support-add-mypy` branch of tket and, although the errors are reduced, there are some outstanding mypy issues coming from within this repository (at least, on the nix build). Advice would be appreciated on this aspect. Added result directory to .gitignore, as this can be created by nix during development.

Previous mypy failures were due to scipy and IPython stubs missing. By adding them to mypy.ini and setting them to ignore, mypy checks now work.

…builds

jake-arkinstall · 2024-06-07T11:00:22Z

I have rebased to main, and I have also updated the flake and workflow to utilise the TKET cachix (binary cache).

For the workflow to succeed, CACHIX_AUTH_TOKEN will need to be available in the workflow environment, as in the tket repository.

I have also run nix flake check locally (x86_64-linux) and everything passes.

jake-arkinstall · 2024-06-07T11:03:02Z

(Additionally, I set nixpkgs to follow TKET's - this will help with possible cache misses if the two disagree on nixpkgs. As more weight is on TKET's build, doing it this way around seems to make the most sense)

cqc-alec · 2024-06-07T11:06:14Z

I added CACHIX_AUTH_TOKEN to the repo.

cqc-alec · 2024-06-07T14:30:37Z

Closing and reopening to trigger CI...

.github/workflows/build_with_nix.yml

jake-arkinstall requested a review from cqc-melf as a code owner January 29, 2024 17:06

cqc-melf reviewed Jan 29, 2024

View reviewed changes

cqc-melf reviewed Jan 30, 2024

View reviewed changes

jake-arkinstall added 7 commits June 7, 2024 11:52

Updated flake.lock and pyqir version. Enabled mypy

1964d31

Previous mypy failures were due to scipy and IPython stubs missing. By adding them to mypy.ini and setting them to ignore, mypy checks now work.

Revert change on mypy.ini and move it to nix derivation for nix-only …

2195b7d

…builds

Added nix build workflow

f0bdfa5

Added Nix README

30e7a85

Updated nix flake to reflect TKET's latest commit

e38e68e

Updated workflow - NOTE: CACHIX_AUTH_TOKEN env is now required

ab0a4c2

jake-arkinstall force-pushed the feature/nix-support branch from c34ceb2 to ab0a4c2 Compare June 7, 2024 10:56

jake-arkinstall changed the base branch from develop to main June 7, 2024 10:57

cqc-alec closed this Jun 7, 2024

cqc-alec reopened this Jun 7, 2024

cqc-alec reviewed Jun 7, 2024

View reviewed changes

.github/workflows/build_with_nix.yml Outdated Show resolved Hide resolved

Update .github/workflows/build_with_nix.yml

19d8ec2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/nix support #120

Feature/nix support #120

jake-arkinstall commented Jan 29, 2024

cqc-melf left a comment

cqc-melf Jan 29, 2024

jake-arkinstall Jan 29, 2024

cqc-melf Jan 30, 2024

cqc-melf Jan 29, 2024

jake-arkinstall Jan 29, 2024

cqc-melf Jan 29, 2024

jake-arkinstall Jan 29, 2024

cqc-melf Jan 30, 2024

cqc-melf left a comment

cqc-melf Jan 30, 2024

cqc-melf Jan 30, 2024

cqc-melf commented Jan 30, 2024

jake-arkinstall commented Jan 30, 2024

cqc-melf commented Jan 30, 2024

cqc-melf commented Jan 30, 2024

jake-arkinstall commented Jan 30, 2024 via email

cqc-melf commented Mar 5, 2024

jake-arkinstall commented Mar 6, 2024

cqc-alec commented Jun 7, 2024

jake-arkinstall commented Jun 7, 2024

jake-arkinstall commented Jun 7, 2024

cqc-alec commented Jun 7, 2024

cqc-alec commented Jun 7, 2024

Feature/nix support #120

Are you sure you want to change the base?

Feature/nix support #120

Conversation

jake-arkinstall commented Jan 29, 2024

Description

Checklist

cqc-melf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cqc-melf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cqc-melf commented Jan 30, 2024

jake-arkinstall commented Jan 30, 2024

cqc-melf commented Jan 30, 2024

cqc-melf commented Jan 30, 2024

jake-arkinstall commented Jan 30, 2024 via email

cqc-melf commented Mar 5, 2024

jake-arkinstall commented Mar 6, 2024

cqc-alec commented Jun 7, 2024

jake-arkinstall commented Jun 7, 2024

jake-arkinstall commented Jun 7, 2024

cqc-alec commented Jun 7, 2024

cqc-alec commented Jun 7, 2024