big Nix refactor #56

jkachmar · 2025-02-11T18:15:51Z

description

this refactors a bunch of the Nix code along the following lines:

refactors from flake-utils to flake-parts, so that flake outputs can be defined in terms of Nix modules
introduces outputs.overlays & outputs.haskellOverlays
- outputs.overlays.temporal-bridge provides the temporal_bridge Rust package
- outputs.overlays.development provides the GHC package sets that we use for our own development environment & CI
- outputs.haskellOverlays.hs-temporal-sdk provides just the packages we manage herein
- outputs.haskellOverlays.dependencies.default provides package overrides for all GHC packages in our matrix
- outputs.haskellOverlays.dependencies.ghc910 provides package overrides for just the GHC 9.10 package set

following this change, downstream consumers should be able to do something like:

{
  inputs = {
    hs-temporal-sdk = {
      url = "github:MercuryTechnologies/hs-temporal-sdk";
      inputs.nixpkgs.follows = "";
      inputs.devenv.follows = "";
      inputs.fenix.follows = "";
    };
  };
}

...and then apply outputs.overlays.temporal-bridge to their Nix package set & outputs.haskellOverlays.hs-temporal-sdk to the applicable haskell.packages.${ghcVersion} package set.

todo

I should probably add a README to the Nix subdirectory explaining what's going on & providing a little context for how the overlays are meant to be consumed.

notes

there's some stuff here that isn't as clean as I would like for it to be, but I think it's probably better to have this reviewed so we can support GHC 9.10 (& newer nixpkgs package sets) more easily.

core/rust/default.nix

jkachmar · 2025-02-11T19:03:11Z

nix/overlays/haskell/ghc910-deps.nix

+  uuid = doJailbreak hprev.uuid;
+  proto-lens = doJailbreak hprev.proto-lens;
+  proto-lens-setup = doJailbreak hprev.proto-lens-setup;
+  proto-lens-protobuf-types = lib.pipe hprev.proto-lens-protobuf-types [
+    doJailbreak
+    (addSetupDepends [ protobuf ])
+  ];
+  proto-lens-runtime = doJailbreak hprev.proto-lens-runtime;
+  proto-lens-protoc = lib.pipe hprev.proto-lens-protoc [
+    doJailbreak
+    (appendPatch (fetchpatch {
+      url = "https://github.com/google/proto-lens/pull/494/commits/7694e28e3ebcfcbcf00ff3eae73949d069c4f21a.diff";
+      relative = "proto-lens-protoc";
+      # There's a patch conflict in the autogenerated hpack header.
+      excludes = [
+        "package.yaml"
+        "proto-lens-protoc.cabal"
+      ];
+      hash = "sha256-qpBh8XGwb30VjIb0VGrLhu/k5az6QrL8XioE9pxDtw4=";
+    }))
+  ];


this whole section is kind of frustrating but it looks like all of this is just straight-up broken upstream so there's no way to work around it.

we could maybe avoid the doJailbreak & appendPatch for proto-lens stuff by just fetching the latest source and building those package directly (since they've updated to support GHC 9.10 but not yet cut a new release).

.envrc

jkachmar · 2025-02-11T20:10:14Z

nix/overlays/haskell/deps.nix

+{ haskell, ... }:
+hfinal: hprev:
+let
+  inherit (haskell.lib.compose) markUnbroken;
+in
+{
+  hs-opentelemetry-propagator-datadog = markUnbroken hprev.hs-opentelemetry-propagator-datadog;
+  if-instance = markUnbroken hprev.if-instance;
+}


as of the latest nixpkgs unstable snapshot it looks like these need to be explicitly marked unbroken

jkachmar · 2025-02-11T20:11:14Z

sdk/src/Temporal/Worker.hs

@@ -507,7 +508,7 @@ startReplayWorker rt conf = provideCallStack $ runWorkerContext conf $ do
      workerActivityWorker = ()
      workerActivityLoop = ()
      workerType = Core.SReplay
-      workerTracer = makeTracer conf.tracerProvider (InstrumentationLibrary "hs-temporal-sdk" "0.0.1.0") tracerOptions
+      workerTracer = makeTracer conf.tracerProvider (InstrumentationLibrary "hs-temporal-sdk" "0.0.1.0" "" emptyAttributes) tracerOptions


this is basically what the detectInstrumentationLibrary splice does

* 'aarch64' + GHC 9.8 seems to fail w/ a lot of cache misses * 'exclude' doesn't appear to union across all branches * adding 'devenv-up' to all branch 'exclude' doesn't stop it on 'main'

a few dependencies in the GHC 9.10 Haskell overlay are broken, even on the lastest unstable nixpkgs package set. in order to build this in CI, we need to jailbreak Cabal dependency bounds & (in one case) backport a patch.

* adds a Haskell overlay for dependencies that need to be overridden in all package sets * updates `temporal-sdk` to work with newer `hs-opentelemetry-api`

`overlays.development` applies a composition of `haskellOverlays` for each GHC version we want to build/test to their respective package sets. e.g. applying this overlay means that `pkgs.haskell.packages.ghc98` will contain `temporal-sdk` (& friends).

* extracts `protogen` to its own file * restores the Haskell package matrix & local dev package deps * restores `devenv` shells for the major GHC versiosn we're tracking

jkachmar · 2025-02-13T16:10:16Z

more than likely that I will restructure this away from flake-parts in the future, so as to avoid it being required downstream by anyone consuming the flake

jkachmar force-pushed the jkachmar/flake-parts branch 6 times, most recently from bc4d6b0 to 4556a25 Compare February 11, 2025 18:58

jkachmar requested a review from iand675 February 11, 2025 18:59

jkachmar commented Feb 11, 2025

View reviewed changes

core/rust/default.nix Outdated Show resolved Hide resolved

jkachmar commented Feb 11, 2025

View reviewed changes

jkachmar force-pushed the jkachmar/flake-parts branch 2 times, most recently from 3b303d6 to c6d7011 Compare February 11, 2025 20:07

jkachmar marked this pull request as ready for review February 11, 2025 20:08

jkachmar requested a review from arianvp February 11, 2025 20:09

jkachmar commented Feb 11, 2025

View reviewed changes

.envrc Outdated Show resolved Hide resolved

jkachmar commented Feb 11, 2025

View reviewed changes

jkachmar added 8 commits February 12, 2025 13:40

fix: garnix build matrix

f9237c3

* 'aarch64' + GHC 9.8 seems to fail w/ a lot of cache misses * 'exclude' doesn't appear to union across all branches * adding 'devenv-up' to all branch 'exclude' doesn't stop it on 'main'

refactor(flake): add 'flake-parts'

8b3da60

fix(nix): add required GHC 9.10 dependencies

4f3d6b4

a few dependencies in the GHC 9.10 Haskell overlay are broken, even on the lastest unstable nixpkgs package set. in order to build this in CI, we need to jailbreak Cabal dependency bounds & (in one case) backport a patch.

refactor(nix): remove old files no longer used

b41989c

refactor(nix): move package definitions out to their own module

0529906

chore(flake): update nixpkgs input

feb07aa

* adds a Haskell overlay for dependencies that need to be overridden in all package sets * updates `temporal-sdk` to work with newer `hs-opentelemetry-api`

fix(devenv): re-enable basic devenv environment

9f1e925

jkachmar force-pushed the jkachmar/flake-parts branch 2 times, most recently from ac297f5 to 9ce150e Compare February 12, 2025 18:49

feat(nix): working dev shells

6d18846

* extracts `protogen` to its own file * restores the Haskell package matrix & local dev package deps * restores `devenv` shells for the major GHC versiosn we're tracking

jkachmar force-pushed the jkachmar/flake-parts branch from 9ce150e to 6d18846 Compare February 12, 2025 18:50

jkachmar merged commit 87de48b into main Feb 13, 2025
59 checks passed

jkachmar deleted the jkachmar/flake-parts branch February 13, 2025 16:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

big Nix refactor #56

big Nix refactor #56

jkachmar commented Feb 11, 2025 •

edited

Loading

jkachmar Feb 11, 2025

jkachmar Feb 11, 2025

jkachmar Feb 11, 2025

jkachmar commented Feb 13, 2025

big Nix refactor #56

big Nix refactor #56

Conversation

jkachmar commented Feb 11, 2025 • edited Loading

description

todo

notes

jkachmar Feb 11, 2025

Choose a reason for hiding this comment

jkachmar Feb 11, 2025

Choose a reason for hiding this comment

jkachmar Feb 11, 2025

Choose a reason for hiding this comment

jkachmar commented Feb 13, 2025

jkachmar commented Feb 11, 2025 •

edited

Loading