chore: Use Dune package management

[sabine]

This patch changes the build configuration to use Dune Package Management.

Done / To do:

• update dune-workspace to pin opam-repository just as we currently do (moved the pins from dune-project there / removed river pin, as it's no longer necessary / pin opam-repository inside dune-workspace)
• update Dockerfile
• remove .opam files
• update Makefiles to reflect using only Dune, and no longer using opam
• update CONTRIBUTING.md and HACKING.md
• update playground/ to build with Dune Developer Preview
• update the CI to use Dune developer preview (thanks Sudha for pointing this out and giving a pointer to a GH action script for Dune Developer Preview)
• update the scraper to use Dune developer preview
• Use a pinned version of dune dev preview so the ocaml.org build does not break unexpectedly. Rationale: we sometimes needs to update on short notice and can't afford to have a broken build
• How do we do power clean? That is getting back to point were dune pkg needs to be initialised. Rationale: this is needed to measure first build time without using docker
• ocaml.org build is using ~/.cache/ocamlorg is dune pkg also using ~/.cache/dune. What's the impact of deleting it?
• Create a GitHub action that uses daily dune preview and current version of ocaml.org build as a regression/blackbox test. But let's not do that with the production version
• Try to merge unrelated changes as much as possible from the patch into main to reduce PR size [on @cuihtlauac]
• Update contributor documentation to make it clear that removing dune.lock directory is enough to go back to the old build -- relevant only for open PRs / current work in progress, we should encourage people to rebase on main and use Dune package management after the merge of this patch
• Rebase tested. It should be done to minimize diff surface. Also it needs to be done again when PRs Do not use Anton's river fork #3178 Call make instead of dune build #3179 Only use Alpine 3.21 #3180 and No code blocks w/o language #3181 are merged

To test this

$ make start

$ make playground

$ make scrape

Current status

I believe this is ready to merge.

[gridbugs]

I've been using dune package management to build this project whenever I've needed to contribute to it over the past year and it's worked fine until recently. Here are some PRs/issues relating to fixing the problems I've encountered recently:

• Add overlay repo to dune-workspace #3147
• Reduce minimum version of ocaml #3146
• All the opam exec -- prefixes in the Makefile makes it hard to use with dune developer preview #3148

[gridbugs]

There's also currently an issue with the ocamlnet package source being missing. I've added it to the opam-source-archives repo and I have a PR to point the ocamlnet package at the archived source. Once the latter PR is merged, updating the opam-repository revision in this project's dune-workspace to the current tip of opam-repository will fix the problem with ocamlnet.

[Sudha247]

This is great! Thanks for the work, @sabine and @gridbugs. Upon offline discussion with Sabine, we wanted to test this on MacOS and Windows to ensure the workflow remains unaffected for contributors on those platforms. I tested with WSL2 (which is the recommended way to develop ocaml.org on Windows), and the ocaml.org project (with make start), builds and runs locally on WSL2, in this branch with dune pkg. Moreover, the testsuite passes too.

The ocaml.org playground doesn't build, as expected - as it seems to have some dependency issues. I'll try to investigate what's happening there!

[sabine]

Sounds great! Looks like we'll be migrating fully to dune pkg as soon as the playground builds.

[sabine]

Thanks @Sudha247 for sabine#53. This makes the playground dune build succeed.

One thing I observe on my machine:

Both on the main branch, and on this branch, running the playground build just deletes the .cmi files in the asset/ folder of the playground. It should be copying the .cmi files from somewhere. The script gen_stdlib.sh performs this part by looking up via opam the location of these .cmi files, so we need to either find where they reside with a dune build, or find another way to regenerate these files. 🤔

[cuihtlauac]

Replace all ````` by ````bash` in this file

[cuihtlauac]

Could the removal of the dependency on Anton's river fork be merged ahead in a separate PR?

[Leonidas-from-XIV]

Does this have any effect? The environment disappears after this call.

[cuihtlauac]

We must not download a dev preview on each build, that's brittle and wasteful.

[Leonidas-from-XIV]

Is there a need for sudo, rsync, bash or nano?

[Leonidas-from-XIV]

I don't think it is customary to depend on all these base-* packages. After all, they're just transitive dependencies of the compiler selected.

[v-gb]

I read the HACKING doc a bit quickly, and ran make watch instead of make start first:

$ make watch
dune build @run -w --force --no-buffer

File "src/ocamlorg_web/lib/dune", line 13, characters 2-15:
13 |   mirage-kv-mem))
       ^^^^^^^^^^^^^
Error: Library "mirage-kv-mem" not found.
-> required by library "ocamlorg_web" in _build/default/src/ocamlorg_web/lib
-> required by executable main in src/ocamlorg_web/bin/dune:2
-> required by _build/default/src/ocamlorg_web/bin/main.exe
-> required by alias src/ocamlorg_web/bin/run in src/ocamlorg_web/bin/dune:8
######################################################################## 100.0%                                              
Done: 95% (95/100, 5 left, 1 failed) (jobs: 1)Browserslist: caniuse-lite is outdated. Please run:
  npx update-browserslist-db@latest
  Why you should do it regularly: https://github.com/browserslist/update-db#readme

Rebuilding...

Done in 1911ms.

Shouldn't every rule that does a build depend on dune.lock, instead of just a couple of them? (or taking a step back, I thought the usual thing to do is commit the dune.lock, so I'm a bit confused about why it's being generated on the fly).