Add a `dune tools env` command to add dev tools to PATH #12521

gridbugs · 2025-10-01T01:30:52Z

The expected use-case is for the output of this command to be eval'd in the user's shell. In order for a terminal-based editor to run the ocamllsp dev tool, eval $(dune tools env) must be run in that terminal before launching the editor.

Fixes #10964

gridbugs · 2025-10-01T01:36:44Z

CC @art-w as this is similar to your PR #11408, but takes advantage of some more recent work to simplify the change.

@Leonidas-from-XIV I know you've been generally against this feature in discussions we've had in the past. I'd be open to adding additional mechanisms for launching programs in an environment with access to dev tools in the future, but currently I'm just trying to wrap up #10964 for the MSP of package management.

Leonidas-from-XIV · 2025-10-01T08:07:34Z

The issue I see is that this creates a huge API surface, i.e. once we implement that we have to support it. And that means dealing with things like

What if a tool is not installed? Can we install it automatically? How do we manage automatic installation, there is no mechanism to report state to the caller in a good way (bar blocking until it is done). Intercepting every binary with a shell-script that calls into dune again is exactly that kind of hacky Unix mess we should try to avoid.
What if there's multiple dune tools env invocations? One installs something, should the other one be updated?
How do you install tools into the env anyway? What happens if you run operations that modify the lockfile inside the env? If you have to restart the editor to relaunch it in a new env this does not seem like a good API.
Why are we providing users a tool which is brittle and they have to be very careful not to misuse (like using the opam env from a different switch, leading to issues). We can of course say "that's for the users to figure out" but then why are we supporting an interface that we ourselves think is problematic?

I think the suggestion in #10964 (comment) is a much better one, achieving the same without adding the trouble of managing a whole shell environment.

In my opinion, all the "env" style managers like Pythons virtualenv or OPAMs env are intensely unpleasant to use because you always need to be careful that the right environment is enabled (and if things go wrong blame it on the users for not being careful enough). Having an environment with a potentially mutable state doesn't mesh well with the stateless mode of dune pkg.

gridbugs · 2025-10-02T07:50:48Z

I think the suggestion in #10964 (comment) is a much better one, achieving the same without adding the trouble of managing a whole shell environment.

I think that's a reasonable alternative. One suggestion that would make it conceptually simple and easier to implement would be to introduce a new command rather than add this functionality to dune tools exec. The dune tools exec command is currently specifically for dev tools and that command knows how to install the specified tool, and I think making it so that that command selectively installs the specified tool if it's available and just runs the given command otherwise might have some surprising consequences. I propose adding something like dune tools cmd -- <command>. Does that sound reasonable?

Leonidas-from-XIV · 2025-10-02T08:18:13Z

What would be the semantics of dune tools cmd -- ocamlformat if ocamlformat is not installed? Would it run an ocamlformat from the system if it is not installed as dev tool but is on the system?

gridbugs · 2025-10-02T08:25:31Z

What would be the semantics of dune tools cmd -- ocamlformat if ocamlformat is not installed? Would it run an ocamlformat from the system if it is not installed as dev tool but is on the system?

Correct, it would run the ocamlformat from the system.

Leonidas-from-XIV · 2025-10-02T09:01:24Z

As long as we don't try to intercept what the user is running it seems reasonable to me (i.e. we just add the tools to the PATH and let the user do whatever they want.

The downside is that it might lead to some unexpected interactions where the user did not expect the "wrong" ocamlformat to be run.

Alizter · 2025-10-02T09:13:46Z

Do we have a which command for the dev tools, so users can at least be sure?

Leonidas-from-XIV · 2025-10-02T09:27:43Z

@Alizter Yes, dune tools which exists, you can see it used in ocamlformat-which.t.

shonfeder · 2025-10-02T21:18:39Z

@Leonidas-from-XIV

this creates a huge API surface

I do not agree with that assessment. IMO we need two things:

A way to install binaries (e.g., for dev tools).
A way for users find and invoke what has been installed.

This change addresses (2). Printing a single line with one path variable seems to be nearly the smallest API surface we could possible offer on *nix systems.

What if a tool is not installed?

IMO we should do nothing in this case. It won't be the environment, and however the user is trying to invoke the tool will just fail. I fully agree that we should not intercept every binary with a shell-script.

What if there's multiple dune tools env invocations? One installs something, should the other one be updated?

In my view, the extent of the environment update should be as minimal as possible. Dynamic updates should not be relevant. But if new bins are added to the dir(s) added to the path (or whatever) that will of course be available by normal unix means.

If users run nested invocations of the env update, they deal with the complications.

How do you install tools into the env anyway?

This is a separate question, IMO. I think we need:

A general purpose way of accessing/finding binaries installed (e.g., as dev tools). But to start with, we should at least have a way of getting access to the blessed dev tools being installed.

Why are we providing users a tool which is brittle and they have to be very careful not to misuse

IMO, this is because:

It is a widely used convention, familiar to many (if not most) devs
It works, and is the simplest alternative I know of for *nix systems. I have seen no comparably simple and usable alternative suggested in the conversations around this.

I think you are overstating the problems with tools that make it easy to update shell environments. But perhaps it is because I envision something that has very little logic to it, and is just abstracting away access to a static location (and maybe later a few other bits of information).

I understand that you don't like tools that alter shell environments (such as direnv), and no one should force you to use them, but these conventions are used widely, and in my experience they cause very little trouble when used responsibly, and give a lot of benefits.

Requiring that a user prefix any invocation of a dune installed binary with dune foo -- is user friendly, imo. But if for some reason we refuse to expose the environment in a helpful way, I (and I am assume others) will probably just cobble together our own version by doing inelegant things like PATH=$(dune foo -- echo '$PATH'):$PATH.

shonfeder · 2025-10-02T21:23:30Z

bin/tools/tools_common.ml

+    else (
+      let initial_path = Env.get Env.initial Env_path.var in
+      let new_path =
+        List.fold_left dev_tool_bin_dirs ~init:initial_path ~f:(fun acc bin_dir ->


One problem I see with this is that new env will be needed every time a new tool is installed.

Have you considered, instead, creating a single directory with all the binaries symlinked? This would ensure that once someone was "in" the dune environment, any additionally installed executables would be available immediately (as I think one would expect). It will also avoid having unnecessarily long paths.

I'm somewhat inclined to think that the desirable behavior I've described is a prerequisite for actually having an environment like this that would not be very surprising and confusing to work with. Without this simplification, I do think we run into the kinds problems @Leonidas-from-XIV is anticipating. And, on the other hand, once we have such a stable path with all the binaries, the invocation of a dune tools env operation becomes basically a trivial convenience.

Have you considered, instead, creating a single directory with all the binaries symlinked? This would ensure that once someone was "in" the dune environment, any additionally installed executables would be available immediately (as I think one would expect). It will also avoid having unnecessarily long paths.

The downside is that Dune will have to maintain a "state" of this directory, e.g. adding or removing symlinks whenever dev tools are installed, removed, updated etc. The advantage with adding directories to PATH as needed is that they're only valid until the process exits.

The downside of the former seems like a very tractable problem foe the implementation, but the upside of the latter is actually a downside for users.

I thought about it but I'm not sure exactly how to do it within the framework of dune rules (I suspect there is a way if I dig deep enough though). From a user's point of view there will be no difference though. Why do you see it as a problem to have a separate entry in PATH for each dev tool?

If/when we allow arbitrary packages to be installed as dev tools, keeping each dev tool in a separate directory will let us better handle the issue where different opam packages contain executables with the same name as each other since they won't collide in the filesystem.

Oh reading through your other comment I think I understand. It appears you believe that this command will only add entries to PATH for dev tools which are currently installed, however this command instead adds entries to PATH for all dev tools which dune knows about, regardless of whether they are installed. That way no env update is needed when a new tool is installed.

On further thought, since this does behave OK w/r/t to not having the behavior or incorrectly assumed, perhaps it is good enough for the current state of things, given that there are relatively few dev tools supported. Any improved, general solution would still be compatible from an end-user perspective IIUC.

So, with the caveat that I still think it is non ideal and I don't see how the current approach can work for general dev tooling support, I think I am ok with this approach for now if others are.

Perhaps this indicates that the proposers solution here should actually go thru the design rethinking we need in the general solution to installing dev tools?

I agree with this. When dev tools were first designed the intention was that they represented the recommended way of performing certain tasks with dune that required external tools, such as launching a top-level in the project's context, generate documentation, or start an lsp server. The current UX around dev tools isn't well suited for installing arbitrary packages as dev tools. I think we should take some time to come up with a more general solution for dev tools or to introduce running exes from abitrary packages as a difference concept to dev tools entirely (however both are out of scope of this PR).

The downside of the former seems like a very tractable problem for the implementation

It's not only that, it's also generally counter to the philosophy of package management in Dune which generally does not keep a state past the execution of a Dune command. This focus allows it to avoid all kinds of issues where the state is desynchornized from the declared configuration and keeps a focus on Dune operations being fast because they need to be quick to repeat.

So while it is possible we do need to evaluate every feature with a lens of "does it conceptually fit". Sometimes breaking that is worth it and sometimes the breakage can be contained.

I don't think that "others do it" is as good of an argument in general. e.g. Nix breaks with a lot of conventions of regular package managers and that's generally considered a good thing, otherwise one could just use these other package managers.

I am certainly not arguing that we should do things just because other systems do it a certain way. But if we introduce added complexity to users that has no precedent and has no benefit to users, I think it is worth remarking, since familiarity and precedent can sometimes be a reason to justify choosing slightly inferior options.

Speaking of nix, it uses nix-profiles/bin to provide a single directory with binaries for users. Tho, as a counterpoint to my claim, nix-shell uses an approach like the one here, adding the separate binary dir for each package from the store into the path. I am still not keen on this, but happy to deffer on that point.

It's not only that, it's also generally counter to the philosophy of package management in Dune which generally does not keep a state past the execution of a Dune command.

Why is generation of a symlink in a directory more stateful that generation of all the other artifacts produced by package management?

From the perspective of a build problem, is there any reason in principle that we couldn't we take the production of the symlinked executable(s) in a shared ./bin directory to be the ultimate build target for tools, with the rest of the artifacts just be intermediate artifacts?

I think we should take some time to come up with a more general solution for dev tools or to introduce running exes from abitrary packages as a difference concept to dev tools entirely (however both are out of scope of this PR).

Yep, makes sense to me!

Leonidas-from-XIV

Code looks quite ok. I would somewhat prefer if the feature would attempt to evaluate $SHELL instead of having to pass --fish by hand but that can be added later.

Leonidas-from-XIV · 2025-10-03T07:43:43Z

bin/tools/tools_common.ml

+        & flag
+        & info [ "fish" ] ~doc:"Print command for the fish shell raher than posix shells")
+    in
+    let _ = Common.init builder in


I think it would be good to ascribe the result type here, to make sure that _ does not get bound to a curried function application if Common.init gets extended with an additional argument.

Leonidas-from-XIV · 2025-10-03T07:44:41Z

bin/tools/tools_common.ml

+      Arg.(
+        value
+        & flag
+        & info [ "fish" ] ~doc:"Print command for the fish shell raher than posix shells")


Suggested change

& info [ "fish" ] ~doc:"Print command for the fish shell raher than posix shells")

& info [ "fish" ] ~doc:"Print command for the fish shell rather than POSIX shells")

Leonidas-from-XIV · 2025-10-03T07:54:54Z

bin/tools/tools_common.ml

+    else (
+      let initial_path = Env.get Env.initial Env_path.var in
+      let new_path =
+        List.fold_left dev_tool_bin_dirs ~init:initial_path ~f:(fun acc bin_dir ->


The downside of the former seems like a very tractable problem for the implementation

It's not only that, it's also generally counter to the philosophy of package management in Dune which generally does not keep a state past the execution of a Dune command. This focus allows it to avoid all kinds of issues where the state is desynchornized from the declared configuration and keeps a focus on Dune operations being fast because they need to be quick to repeat.

So while it is possible we do need to evaluate every feature with a lens of "does it conceptually fit". Sometimes breaking that is worth it and sometimes the breakage can be contained.

I don't think that "others do it" is as good of an argument in general. e.g. Nix breaks with a lot of conventions of regular package managers and that's generally considered a good thing, otherwise one could just use these other package managers.

gridbugs · 2025-10-07T06:14:50Z

Code looks quite ok. I would somewhat prefer if the feature would attempt to evaluate $SHELL instead of having to pass --fish by hand but that can be added later.

Something I learnt while working on the binary dune install script is that SHELL is usually not an environment variable but a shell variable, and so isn't propagated to child processes. I picked up a trick from reading how opam init is implemented which is to inspect the parent process to determine the shell, though this isn't portable and not robust to running dune from within a script. Still could be useful, but given its complexity I think we can do it in a later change.

Leonidas-from-XIV · 2025-10-07T08:48:58Z

Ah, I was not aware of that. It's a good point and I agree that this PR can go ahead without this feature.

The expected use-case is for the output of this command to be eval'd in the user's shell. In order for a terminal-based editor to run the ocamllsp dev tool, `eval $(dune tools env)` must be run in that terminal before launching the editor. Fixes ocaml#10964 Signed-off-by: Stephen Sherratt <[email protected]>

* 'main' of github.com:/ocaml/dune: (147 commits) cram test: test only parameter flags in merlin generation fix(oxcaml): import eta-expansion changes from opam-repo (ocaml#12563) address review comments Mask the path to the stdlib fix(oxcaml): generate merlin config for library parameters fix(melange + include_qualified): track correct `.cmj` dependencies in emit (ocaml#12531) refactor: remove some unused code in [Path] (ocaml#12558) dep_rules: don't run (transitive) `ocamldep` on single module buildables (ocaml#12555) fix(pkg): ignore project settings for building packages test(pkg): reproduce ocaml#12131 melange: add a test for module cycle checks (ocaml#12554) chore: lint check for new changes entries (ocaml#12553) feature(cram): allow for conflict detection (ocaml#12538) ci: update for ocaml 5.4 release (ocaml#12552) chore(script): generate changelog from structure (ocaml#12516) Reuse dependencies between project and tools (ocaml#12526) Introduce Io.overwrite_file test: fix dune install requiring a mandir Enable package management for more tests Add a `dune tools env` command to add dev tools to PATH (ocaml#12521) ...

gridbugs requested review from ElectreAAS, Leonidas-from-XIV, rgrinberg and shonfeder October 1, 2025 01:30

shonfeder reviewed Oct 2, 2025

View reviewed changes

Leonidas-from-XIV reviewed Oct 3, 2025

View reviewed changes

Leonidas-from-XIV requested a review from Alizter October 3, 2025 12:42

shonfeder approved these changes Oct 8, 2025

View reviewed changes

gridbugs force-pushed the dune-tools-env branch from f5ffc55 to 8ad1641 Compare October 8, 2025 00:26

gridbugs merged commit 4608696 into ocaml:main Oct 8, 2025
11 checks passed

gridbugs deleted the dune-tools-env branch October 8, 2025 00:28

	& info [ "fish" ] ~doc:"Print command for the fish shell raher than posix shells")
	& info [ "fish" ] ~doc:"Print command for the fish shell rather than POSIX shells")

Add a dune tools env command to add dev tools to PATH #12521

Add a dune tools env command to add dev tools to PATH #12521

Conversation

gridbugs commented Oct 1, 2025

Uh oh!

gridbugs commented Oct 1, 2025

Uh oh!

Leonidas-from-XIV commented Oct 1, 2025

Uh oh!

gridbugs commented Oct 2, 2025

Uh oh!

Leonidas-from-XIV commented Oct 2, 2025

Uh oh!

gridbugs commented Oct 2, 2025

Uh oh!

Leonidas-from-XIV commented Oct 2, 2025

Uh oh!

Alizter commented Oct 2, 2025

Uh oh!

Leonidas-from-XIV commented Oct 2, 2025

Uh oh!

shonfeder commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shonfeder Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Leonidas-from-XIV Oct 3, 2025 • edited by shonfeder Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shonfeder Oct 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Leonidas-from-XIV left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Leonidas-from-XIV Oct 3, 2025 • edited by shonfeder Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gridbugs commented Oct 7, 2025

Uh oh!

Leonidas-from-XIV commented Oct 7, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Add a `dune tools env` command to add dev tools to PATH #12521

Add a `dune tools env` command to add dev tools to PATH #12521

shonfeder commented Oct 2, 2025 •

edited

Loading

shonfeder Oct 2, 2025 •

edited

Loading

Leonidas-from-XIV Oct 3, 2025 •

edited by shonfeder

Loading

shonfeder Oct 4, 2025 •

edited

Loading

Leonidas-from-XIV Oct 3, 2025 •

edited by shonfeder

Loading