dev meeting 20211028

Present at the meeting:

• Thomas Gazagnaire (@samoht)
• Marek Kubica (@Leonidas-from-XIV)
• Etienne Millon (@emillon)
• Nathan Rebours (@NathanReb)

Current plan

• Marek and Nathan tests the new improved local package handling before merging
• Nathan and Thomas wrap up the work on opam monorepo list command
• Etienne reach out to Hugo so he can try out the above feature
• We start working on the hybrid mode
• We start working on the extra field in opam files to set repositories to use during lock
• We start working on improved solver errors
• We publish Etienne's workflow documentation

Meeting notes

Recent work

Marek worked on the long awaited improvement of the missing opam-overlays warning. This warning used to be always emitted when the users did not add the dune-universe/opam-overlays repository to their active switch. There exists valid reasons not to add it and in particular if the solver succeeds, there's no reason to worry about those extra dune-ports. Now that we patched opam-0install-solver we are able to inspect the solver failure and emit this warning if and only if the solver failed to find a solution and among the reasons why figures a package that has no available version building with dune. Marek also worked on proper handling of local packages. opam-monorepo used to only considered the local packages that were the targets for the lock command. This has been a problem when the workspace included other opam packages that were part of the dependency cone of the target packages as opam-monorepo would then recursively include them in the duniverse resulting in a broken workspace with duplicate package definitions. All packages are now known to the solver and opam-monorepo will know to use the local versions if they happen to be required as dependencies. The PR is now ready but given that was quite a bit of work we want to properly test this before merging. Finally Marek also worked on a few CI improvements that will make working on the project a breeze.

Etienne worked on documenting opam-monorepo workflows. This will prove a very important ressource that was lacking for the tool so we should publish it ASAP. I also hope it will open the door for more documentation, adding some is always easier than writing it from scratch. He also had the chance to talk with Hugo, who works on Tezos, lately. While trying tezos + opam-monorepo Hugo ran into an issue where opam-monorepo pulled some tezos packages because it failed to detect they were present locally. This is exactly what Marek has been fixing so we should make sure he gets a chance to try it again once we have tested, merged and released this feature.

Thomas worked on two new commands for the tool. First one is a depext command that uses the opam lib to install the depexts in the lockfile. Ideally we wanted this to be done in opam as the depext command there lacks a way to just install depexts from a single opam file rather than running the solver and install transitive depexts. It's going to take a bit of time unfortunately to upstream this and this was a reasonably simple addition so we decided to add it to the tool as we do need a tool to take care of those depexts in the meantime. The second command is one that lists the packages, versions and source (tarball or git repo URL) for packages that opam-monorepo will install in the duniverse based on a lockfile. Thomas found out that new opam-monorepo users, in particular in the mirage community were quite confused as to what opam-monorepo was actually installing and this command should help with this.

Discussions and ideas

During our discussion Thomas mentioned that the first time he used opam-monorepo it was unclear that lock only worked for local packages, i.e. package that are defined in the workspace, on your file system. In the future it might be worth allowing generating lockfiles for packages from upstream repositories metadata but for now this seems like sane behaviour. It probably needs better documentation though so we should make sure we clarify this.

While talking about the "hybrid" workflow support — i.e. support for explicitly marking some dependencies (besides dune and ocaml) as to be opam installed, as opposed to being source installed in the duniverse — Thomas mentioned that we could also think about opam monorepo as a fully source based deps mgmt tool. It's definitely not ready for this in its current form but it's something we could consider at some point.

What's next

Absolute priority right now is to test and finish Marek's work on local packages handling. This has been in the works for quite a while now and we already know it will benefit some of our users starting with Sonja and Nathan for the maintenance of ppxlib. Nathan and Marek will work on this.

There is also an opened PR by thomas for a first version of the list command. Nathan will take care of the review as we all agreed the command will be useful right now even if we eventually end up moving this to opam in the future once the opam-monorepo workflow is more stable.

Thomas will give us a presentation on the mirage tool to help us better understand how it uses opam-monorepo and what could be done to improve their interactions.

Next big things to work on are:

• hybrid workflow support: using a variable to mark which dependencies in the source opam file should be pulled in the duniverse and which should be opam installed. This distinction already exists in practice, ocaml, dune and some base packages must be installed in the switch. Properly handling this will be useful for the mirage tool which has to generate different opam files to work around this limitation atm and will also improve the interaction with the main opam tool that will then be able to properly handle our lockfiles, installing only what we want it to install.
• explicit repositories: using an extra field in source opam files to specify the list of repositories opam-monorepo should use to run the lock. This, in addition to the recent support for pin-depends, will allow for reproducible lock runs. It will also allow the mirage tool and our users to use some repositories without "polluting" their opam configuration if don't want to.
• improved solver errors: the error messages from the solver are quite infamous. They are long and hard to interpret. With the recent improvements to opam-0install-solver and some work from Lucas we should be able to try filtering out irrelevant are redundant information to come up with better messages even if slightly incomplete. We should make sure to expose a flag to still get the full error, or to save it to a file somewhere in case our experimental simplification doesn't work well enough.