dev meeting 20211125

Present at the meeting

• Marek Kubica (@Leonidas-from-XIV)
• Nathan Rebours (@NathanReb)

Current plan

• Marek extract a PR that assign a {vendored} variable to the duniverse dependencies in the lockfile
• Marek finishes the hybrid mode
• Nathan adds support for passing explicit list of repos to the solver
• Nathan adds a feature to dump the local opam config to the lockfile

Meeting notes

Nathan worked on improving the opam monorepo list subcommand PR. He also fixed a bug that was initialy uncovered by this PR, that was duplicating some packages when parsing the lockfile. Finally he released version 0.2.7 with the recent additions as the next changes we plan will likely break the lockfile format and will need to be released as 0.3.

Marek worked on the hybrid mode, trying a few different approach.

We had a quick meeting yesterday to discuss how to first implement the hybrid mode. Marek suggested using an extra field rather than a variable in the source opam files so the regular opam workflow still works. This seems like a good property to enforce. By making sure that the lockfile contains everything that one needs to install, via opam (ocaml, dune, hybrid mode marked packages) or in the duniverse, and by properly flagging the duniverse packages with the variable there it is possible to run the following to setup one's dev environment:

opam switch create ./ --empty
opam monorepo pull
opam install -t --deps-only --locked ./

and still possible to have a regular opam workflow by running:

opam switch create ./

This marks a clear separation, the opam file is used for locking and for the regular opam workflow and for opam-monorepo workflows, everything goes through the lockfile.

With all this in mind we'll extract a PR to set the variable in the lockfile to already enable the above setup steps.

We also discussed how to perform locking and solving of the opam dependencies in the hybrid mode. After a long discussion we agreed that the most sensible approach would be to solve all the deps together to get a consistent solution and then do some graph traversal on the solution to determine which packages have to be installed via opam and which need to be pulled in the duniverse and mark them accordingly. This has the advantage of making sure everything can be built with the same OCaml and dune and therefore in the same switch without running the solver multiple times. The main downside is that the hybrid mode can't be used to install tools in the switch that are not co-installable with the libraries in the duniverse (e.g. if the tool depends on a different version of a library than the project itself). We agreed this was not what we're trying to solve at the moment so this is an okay limitation. There will eventually be needs for polishing this as the naive method to implement the filtering will miss some things like dependency cones intersections or might pull too much things in the duniverse in complex opam deps combination situations.

As mentioned above the next release will be 0.3, here's a list of the lockfile format changes we'll introduce:

• Each duniverse dependency in the lockfile will be marked with a {vendored} variable so it's ignored by opam. This is likely to break the format as older version of the tool might not properly parse this.
• The x-opam-monorepo-root-packages field will be renamed to x-opam-monorepo-target-packages to reflect the changes in terminology we recently introduced in the CLI and documentation.
• Optional extra fields to dump local opam config or copy the extra fields used for reproducible solve (pin-depends, x-opam-monorepo-repos).

Next steps are to finish the hybrid mode, implement the reproducible solve and dump the opam config in the lockfile for regular solve. At this point we should be able to release 3.0 and start focusing on solver failure reporting. The reproducible solve should enable running lock only using local files which in turn will allow us to write more cram tests. This will prove particularly helpful to work on solver errors.