Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(dev): add nix-based dev-env #4726

Merged
merged 4 commits into from
Jul 22, 2024
Merged

feat(dev): add nix-based dev-env #4726

merged 4 commits into from
Jul 22, 2024

Conversation

conorsch
Copy link
Contributor

@conorsch conorsch commented Jul 18, 2024

Describe your changes

Trying to make it easier to spin up a fullnode on localhost. The team has been focused on writing tooling, like pindexer, and the manual steps to munge CometBFT configs and setup a localhost postgres database are not awesome. In addition to pd/cometbft spin-up, the dev-env includes:

  • metrics
  • postgres
  • pindexer

Not yet including grpcui, because the latest published version doesn't support the new server reflection API. Will tack that on in a dedicated follow-up.

The dev-env is strongly opinionated and uses nix [0] to manage cross-platform, polyglot dependencies. There's other great work in our ecosystem, e.g. [1], and this early push tries to be consistent with that.

Updates the relevant first-run docs. Much more work should be done on the developer-focused content in the guide overall, including an overhaul of discussing RPC endpoints and web-relevant resources, but those changes can land in the future. This PR aims to establish a basis for a sane, supported dev-env, providing a pleasant first-run experience for newcomers (and existing devs too).

[0] https://nixos.org
[1] https://github.com/informalsystems/cosmos.nix

Here's an example of what the just dev command looks like:

process-compose-nix-dev-env-screenshot

Issue ticket number and link

No issue, but this PR builds on the excellent foundation by @cratelyn in #3574.

Checklist before requesting a review

  • If this code contains consensus-breaking changes, I have added the "consensus-breaking" label. Otherwise, I declare my belief that there are not consensus-breaking changes, for the following reason:

    No changes to application code: focuses on docs and scripts for dev/ci.

Testing and review

In order to review these changes, one should check out this feature branch, build the dev docs in docs/guide/, and view them locally, then follow the instructions at http://127.0.0.1:3000/dev/dev-env.html. Pay attention to what's confusing, what's unpleasant, and write those thoughts down as part of the review.

Then proceed to http://127.0.0.1:3000/dev/devnet-quickstart.html and run just dev as recommended. Does anything break? Does it work as advertised? Is anything unclear? Try running ctrl+c to stop the setup, and then just dev to resume it. Does it come back up OK? Is that a sane behavior?

Trying to make it easier to spin up a fullnode on localhost. The team
has been focused on writing tooling, like pindexer, and the manual steps
to munge CometBFT configs and setup a localhost postgres database are
not awesome. In addition to pd/cometbft spin-up, the dev-env includes:

  * metrics
  * postgres
  * pindexer

Not yet including grpcui, because the latest published version doesn't
support the new server reflection API. Will tack that on in a dedicated
follow-up.

The dev-env is strongly opinionated and uses nix [0] to manage
cross-platform, polyglot dependencies. There's other great work in our
ecosystem, e.g. [1], and this early push tries to be consistent with
that.

Updates the relevant first-run docs. Much more work should be done on
the developer-focused content in the guide overall, including an
overhaul of discussing RPC endpoints and web-relevant resources, but
those changes can land in the future. This PR aims to establish a basis
for a sane, supported dev-env, providing a pleasant first-run experience
for newcomers (and existing devs too).

[0] https://nixos.org
[1] https://github.com/informalsystems/cosmos.nix
@zbuc
Copy link
Member

zbuc commented Jul 19, 2024

Guide was easy to follow, but I got this error when trying to run nix develop on macOS:

nix develop
error (ignored): error: end of string reached
error:
       … while calling the 'derivationStrict' builtin
         at <nix/derivation-internal.nix>:9:12:
            8|
            9|   strict = derivationStrict drvAttrs;
             |            ^
           10|

       … while evaluating derivation 'nix-shell'
         whose name attribute is located at /nix/store/iqxdbcjlg9wx8gdc8bdhy9nsd5imcbjp-source/pkgs/stdenv/generic/make-derivation.nix:334:7

       … while evaluating attribute '__impureHostDeps' of derivation 'nix-shell'
         at /nix/store/iqxdbcjlg9wx8gdc8bdhy9nsd5imcbjp-source/pkgs/stdenv/generic/make-derivation.nix:449:7:
          448|       __propagatedSandboxProfile = unique (computedPropagatedSandboxProfile ++ [ propagatedSandboxProfile ]);
          449|       __impureHostDeps = computedImpureHostDeps ++ computedPropagatedImpureHostDeps ++ __propagatedImpureHostDeps ++ __impureHostDeps ++ stdenv.__extraImpureHostDeps ++ [
             |       ^
          450|         "/dev/zero"

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error: invalid regular expression '.*.(no_lfs|param||bin)$'

@zbuc
Copy link
Member

zbuc commented Jul 19, 2024

Cool, looks good to me. I hit a few macOS-related stumbling blocks but pushed up fixes. I was able to get everything running in the end.

@conorsch
Copy link
Contributor Author

I hit a few macOS-related stumbling blocks but pushed up fixes.

Nice, thanks for taking a look! Some of the macos-specific entries broke the dev env on Linux for me, so let me tinker a bit more...

@conorsch
Copy link
Contributor Author

Nice, thanks both for testing!

@conorsch conorsch merged commit 9976905 into main Jul 22, 2024
13 checks passed
@conorsch conorsch deleted the nix-local-devnet branch July 22, 2024 19:55
conorsch added a commit to penumbra-zone/guide that referenced this pull request Jul 23, 2024
There were a few recent changes made to the guide in the protocol repo,
as of [0], and this commit pulls those changes in. At the very least,
I noticed that changes from the following PRs were missing:

* penumbra-zone/penumbra#4732
* penumbra-zone/penumbra#4726

Tried to make sure to preserve unique content updates implemented
during the mdbook -> nextra conversion.

[0] https://github.com/penumbra-zone/penumbra/tree/a2ffd8a79f1b397e81df7b7e31b256d545cdf004/docs/guide
conorsch added a commit to penumbra-zone/guide that referenced this pull request Jul 23, 2024
There were a few recent changes made to the guide in the protocol repo,
as of [0], and this commit pulls those changes in. At the very least,
I noticed that changes from the following PRs were missing:

* penumbra-zone/penumbra#4732
* penumbra-zone/penumbra#4726

Tried to make sure to preserve unique content updates implemented
during the mdbook -> nextra conversion.

[0] https://github.com/penumbra-zone/penumbra/tree/a2ffd8a79f1b397e81df7b7e31b256d545cdf004/docs/guide
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants