pkg: Make lock files targets

[Leonidas-from-XIV]

At this moment this PR is in the early stages of getting things to link up and doesn't actually lock.

Too early to review yet, just pushed to share the code publicly.

[rgrinberg]

I don't think you can include this information here. It makes the action non deterministic unfortunately. You could make it part of the action input if you really want it, but i'd say it's best to skip it.

[Leonidas-from-XIV]

I know. This is just for debugging, as I can't make the action rerun on changes in dune-project and want to have a way to see whether the rule was run or not. The final PR will not have this anymore.

[Leonidas-from-XIV]

I've changed it temporarily to dump the package info as dyn representation into the file, that way it should be a bit more reproducible and allow debugging whether changes in the package definition lead to re-execution of the action. Weirdly I can't get it to regenerate the file, maybe I am dereferencing the Memo.t in the wrong location and the dependency does not get registered?

[rgrinberg]

You can't use super context for package management rules. See how you've used the lower level rules api in Pkg_rules to work around this.

[Leonidas-from-XIV]

Why can't the super context be used?

[rgrinberg]

Here you are generating the rule to produce a lock directory, but to initialize the super context you need to first load the lock directory. This will cause a cycle.

Note that the super context is useless for you here anyway. It is what we use to load the environment for basic rules, but tha environment cannot impact the generation of the lock directory. Otherwise, producing lock directories wouldn't be deterministic across machines.

[Leonidas-from-XIV]

Thanks for the explanation.

I don't really need the super context anyway, I just tried to use it to get a transitive dependency on the project. While I was looking at how opam_create.ml creates the dependency, @emillon pointed out that this is most likely due to binding on Super_context.t Memo.t. But I can just as well bind on Dune_project.t list Memo.t which makes it a bit more explicit that I depend on the project definition.

However, even if I bind on it, I can't get my action to re-run when the dune-project is changed, so there's still something missing I believe.

[rgrinberg]

I don't really get this lock_dir argument. It's part of the path in the directory where we are loading the rule and we are passing it to the action?

[Leonidas-from-XIV]

It's the identifier for the lock directory we want to generate. Basically the value that lock_dir->path from dune-workspace. In most projects that will always just be dune.lock, but for simplicity I thought it would be nice if the rules would generate them in an unchanging place no matter whether you have one or multiple lock directories.

[rgrinberg]

Okay, I understand. Although note that the path in the workspace is not necessarily a path in the root. It might be something like foo/bar/dune.lock as well for example. So you will either need to escape it somehow or make a much more complex loading scheme.

[rgrinberg]

I don't think so, you can point to the directory itself.

[Leonidas-from-XIV]

I would have thought so too, but when I remove the last path element I get:

Error: No rule found for default/.lock/dune.lock (context _private)
-> required by alias dune.lock/pkg-lock

Which is surprising to me, since I believe I am attaching a rule there:

  | [ ".lock"; lock_dir ] -> Memo.return @@ setup_lock_rules ~dir ~lock_dir ~projects

[rgrinberg]

I don't fully understand the scheme of rules you're creating, but I don't see why you need more than one directory to define all of your rules. I would expect the following:

1. If you have a single lock directory, then you just need one rule inside ".lock"
2. If you multiple lock directories, they can all co-exist in .lock under different names.

You don't even need a directory per context. Just a single directory for all contexts will do.

[Leonidas-from-XIV]

The scheme is .lock/<path-of-the-lockdir>/<contents-here>. I wanted to keep it uniform, so it will always be the same, no matter if the user has one or multiple lock dirs configured in their workspace. That means that <path-of-the-lockdir> will mostly be dune.lock but I don't see an issue with that.

That way I also hope that the promote rules that promote the generated contents back into the users source tree will be simpler to write and understand as one does not need to know whether the workspace has one or multiple lock directories configured.

[rgrinberg]

You can do it like that as well. In this particular problem, the names of the targets aren't important and should be invisible to the user, so just about any scheme can be chosen. In fact, a clever approach would pick a name that is a has of the all the inputs to the solver. That way, multiple lock directories could be represented with the same target.

I don't necessarily recommend this, just pointing out that the names are completely irrelevant.
Therefore, I suggest just picking the scheme that will be the easiest to implement. If there's other layouts you'd like to experiment with, you can always do it later. Keep in mind that in an ideal implementation, none of these file targets would even be necessary and we'd just do everything in memory and internal database files. So I really don't see the need to insist on some particular naming scheme.