Skip to content

changelog: add a post about OCaml.org migration to Dune Developer Pre… #3152

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,174 @@
---
title: "Setting up OCaml.org as a Testing Ground for Dune Developer Preview"
tags: [dune, developer-preview, ocaml-org]
authors: ["Sabine Schmaltz", "Leandro Ostera", "Sudha Parimala"]
---

The OCaml.org website is undergoing a build setup change as it migrates to Dune Developer Preview. This migration serves a dual purpose: modernizing the website's build process to use Dune package management while providing a real-world testing environment for Dune's experimental features exposed through the [Dune Developer Preview](https://preview.dune.build?utm_source=ocaml.org&utm_medium=referral&utm_campaign=changelog).

## Why This Migration Matters

OCaml.org is one of the most visible projects in the OCaml ecosystem. By migrating it to Dune Developer Preview, we are creating an environment that will help identify issues and refine new Dune workflows before their stable release.

We are currently tracking the state of the migration in [pull request #3132](https://github.com/ocaml/ocaml.org/pull/3132). We're about to finish the work and merge the PR soon!

## Technical Challenges and Solutions

Migrating OCaml.org to use Dune Developer Preview has revealed several interesting technical aspects that other projects may encounter when adopting Dune package management:

### How to Configure Nested Dune Projects

One of the first hurdles encountered was dealing with nested Dune projects. The OCaml.org codebase contains a playground project nested within the main website project, making workspace management slightly trickier than expected.

> **Note:** This is not a standard or recommended setup, but it happens to be how OCaml.org has historically grown as a project. The playground is a part of the site that is not updated very often, and we only regenerate it (and commit the build artefacts) when the playground is updated to a new version or gets new features. Specifically, the OCaml version used in the playground does not have to be the same as the one used for the main site.

The solution we went with was to use **two separate `dune-workspace` files** - one for the main OCaml.org site and one for the playground.

**Step-by-step solution for the OCaml.org structure:**

1. **Create the main [`dune-workspace`](https://github.com/ocaml/ocaml.org/blob/main/dune-workspace)** file in the repository root:
```lisp

Check failure on line 30 in data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md

View workflow job for this annotation

GitHub Actions / lint

Fenced code blocks should be surrounded by blank lines

data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md:30 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```lisp"] https://github.com/DavidAnson/markdownlint/blob/v0.38.0/doc/md031.md
; dune-workspace (at repository root)
(lang dune 3.19)
```

2. **Create a separate [`dune-workspace`](https://github.com/ocaml/ocaml.org/blob/main/playground/dune-workspace)** file for the playground:
```lisp

Check failure on line 36 in data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md

View workflow job for this annotation

GitHub Actions / lint

Fenced code blocks should be surrounded by blank lines

data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md:36 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```lisp"] https://github.com/DavidAnson/markdownlint/blob/v0.38.0/doc/md031.md
; playground/dune-workspace
(lang dune 3.19)
```

3. **Build each project independently** by forcing the correct root:
```bash

Check failure on line 42 in data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md

View workflow job for this annotation

GitHub Actions / lint

Fenced code blocks should be surrounded by blank lines

data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md:42 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```bash"] https://github.com/DavidAnson/markdownlint/blob/v0.38.0/doc/md031.md
# Build the main OCaml.org site
dune build

# Build the playground (must use --root to force workspace detection)
cd playground
dune build --root .

# Or from anywhere in the repository
dune build --workspace playground/dune-workspace --root playground
```

4. **Lock dependencies separately** for each workspace:
```bash

Check failure on line 55 in data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md

View workflow job for this annotation

GitHub Actions / lint

Fenced code blocks should be surrounded by blank lines

data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md:55 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```bash"] https://github.com/DavidAnson/markdownlint/blob/v0.38.0/doc/md031.md
# Lock main OCaml.org dependencies
dune pkg lock

# Lock playground dependencies (from the playground directory)
cd playground
dune pkg lock --root .
```

### OCaml.org's Stable Dependency Management Setup

OCaml.org uses a fixed commit hash of opam-repository to ensure a stable environment between contributors, maintainers, and the continuous integration service. This avoids running into dependency upgrade issues during feature development or when working on bugfixes.

**Step-by-step workspace configuration for OCaml.org:**

1. **Pin opam-repository to the same specific commit** in the `dune-workspace` files:
```lisp

Check failure on line 71 in data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md

View workflow job for this annotation

GitHub Actions / lint

Fenced code blocks should be surrounded by blank lines

data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md:71 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```lisp"] https://github.com/DavidAnson/markdownlint/blob/v0.38.0/doc/md031.md
; Both dune-workspace and playground/dune-workspace use:
(repository
(name pinned_opam_repository)
(url git+https://github.com/ocaml/opam-repository#584630e7a7e27e3cf56158696a3fe94623a0cf4f))

(lock_dir
(path dune.lock)
(repositories
pinned_opam_repository
))
```

2. **Update the pinned repository commit** in both files during coordinated upgrades:
```bash

Check failure on line 85 in data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md

View workflow job for this annotation

GitHub Actions / lint

Fenced code blocks should be surrounded by blank lines

data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md:85 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```bash"] https://github.com/DavidAnson/markdownlint/blob/v0.38.0/doc/md031.md
# Update both dune-workspace files with the new commit hash

# Then regenerate both lock files
dune pkg lock
cd playground && dune pkg lock --root .
```

### Pinning Individual Dependencies

The playground has additional complexity because it requires specific versions of packages that aren't available in the standard `opam-repository` or need patches for compatibility with Dune package management. These dependencies are handled through individual pins in the playground's `dune-workspace`.

**Pin a dependency**:

```

Check failure on line 99 in data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md

View workflow job for this annotation

GitHub Actions / lint

Fenced code blocks should have a language specified

data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md:99 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"] https://github.com/DavidAnson/markdownlint/blob/v0.38.0/doc/md040.md
...

(pin
(name merlin-js)
(url "git+https://github.com/voodoos/merlin-js#3a8c83e03d629228b8a8394ecafc04523b0ab93f")
(package
(name merlin-js)))

...

(lock_dir
(repositories pinned_opam_repository)
(pins esbuild code-mirror merlin-js js-top-worker ocamlbuild ocamlfind)
(version_preference newest))
```

### Compatibility of Dependencies Using Symlinks


Check failure on line 118 in data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md

View workflow job for this annotation

GitHub Actions / lint

Multiple consecutive blank lines

data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md:118 MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2] https://github.com/DavidAnson/markdownlint/blob/v0.38.0/doc/md012.md
During the migration, we saw compatibility issues with upstream dependencies relating to the use of symlinks. For example, `merlin-js` uses symlinks in its example programs.

Currently, Dune package management doesn't support symlinks, so we removed the offending examples to enable building `merlin-js` with Dune package management from [this `merlin-js` branch](https://github.com/Sudha247/merlin-js/tree/exp).

#### The Opam-repository Overlays for OCamlBuild and OCamlFind

A similar issue relating to symlinks exists with `ocamlbuild` and `ocamlfind`; however, both packages already have patched versions compatible with dune package management and available at https://github.com/ocaml-dune/opam-overlays.

Check failure on line 125 in data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md

View workflow job for this annotation

GitHub Actions / lint

Bare URL used

data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md:125:190 MD034/no-bare-urls Bare URL used [Context: "https://github.com/ocaml-dune/..."] https://github.com/DavidAnson/markdownlint/blob/v0.38.0/doc/md034.md

So, we added the overlays repository to the `dune-workspace` files and listed both repositories within the `lock_dir` Dune stanza:

```

Check failure on line 129 in data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md

View workflow job for this annotation

GitHub Actions / lint

Fenced code blocks should have a language specified

data/changelog/posts/dune-developer-preview/2025-06-10-migrating-ocaml-org.md:129 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"] https://github.com/DavidAnson/markdownlint/blob/v0.38.0/doc/md040.md
...

(repository (name pinned_overlay_repository) (url git+https://github.com/ocaml-dune/opam-overlays#2a9543286ff0e0656058fee5c0da7abc16b8717d))

(lock_dir
(repositories pinned_opam_repository pinned_overlay_repository))
```

This way, the patched dependencies `ocamlbuild` and `ocamlfind` are picked up by Dune's solver when creating the `dune.lock` directory.

### Updating GitHub Actions to Use Dune Developer Preview

Using the GitHub Actions package available at https://github.com/ocaml-dune/setup-dune, we updated the workflows [`ci.yml`](https://github.com/ocaml/ocaml.org/blob/main/.github/workflows/ci.yml) and [`scrape.yml`](https://github.com/ocaml/ocaml.org/blob/main/.github/workflows/scrape.yml) to use Dune Package Management. The change turned out to be straightforward and we simplified the configuration in several ways: (1) instead of having explicit calls to the build tool, we invoke the existing Makefile, and (2) we do not need to specify the OCaml compiler version in the workflow anymore, since Dune manages the compiler version via `dune-project`.

### Playground: Copying .cmi Files from the OCaml Standard Library

For the playground to load the Standard Library in the browser via the generated .js bundle, we need the relevant `.cmi` files. Previously, we were using the `opam var ocaml:lib` command to discover the location from the compiler build artifacts, and we copied them to the `asset/` folder of the playground.

To replicate this behavior with Dune Developer Preview, we invoke the OCaml compiler itself to tell us the path where we can find these `.cmi` files, by invoking `ocamlopt -config-var standard_library` through Dune Developer Preview from inside the [playground's dune file](https://github.com/ocaml/ocaml.org/blob/main/playground/dune).

## Current Status and Next Steps

The migration is progressing through three key phases:

1. **Playground Integration**: Making the OCaml.org playground build successfully with Dune package management, which requires investigating and potentially fixing upstream dependency issues
2. **Documentation Updates**: Updating contributor documentation to reflect the new build process
3. **Cross-Platform Testing**: Ensuring the new workflow functions correctly on Windows and macOS

Once these phases are complete, OCaml.org will officially adopt Dune package management as its standard build method.

## Further Reading

For more detailed information about the Dune features used in this migration, consult the official Dune documentation:

- **[Dune Workspaces](https://dune.readthedocs.io/en/latest/reference/dune-workspace/index.html)** - Complete reference for `dune-workspace` file syntax and configuration
- **[Lock Directories](https://dune.readthedocs.io/en/latest/reference/dune-workspace/lock_dir.html)** - How to configure and use lock directories for reproducible builds
- **[Package Management](https://dune.readthedocs.io/en/stable/tutorials/dune-package-management/setup.html)** - Overview of Dune's package management features and workflows
- **[Pinning Dependencies](https://dune.readthedocs.io/en/stable/tutorials/dune-package-management/pinning.html)** - Guide to pinning packages to specific versions or Git commits
- **[Repository Configuration](https://dune.readthedocs.io/en/stable/tutorials/dune-package-management/repos.html)** - How to configure opam repositories in Dune workspaces

## Conclusion

Despite OCaml.org having a non-standard project setup and having to deal with custom dependencies for the playground, we found the migration to Dune package management to be mostly straightforward: OCaml.org itself didn't need us to touch any upstream dependencies, they just worked out of the box with Dune package management.

The playground uses a few custom dependencies which needed to be updated to work with Dune package management. To get an understanding of how much of the OCaml ecosystem supports Dune package management, here's an interesting read: ["Opam Health Check: or How we Got to 90+% of Packages Building with Dune Package Management"](https://tarides.com/blog/2025-06-05-opam-health-check-or-how-we-got-to-90-of-packages-building-with-dune-package-management/). That post also contains a link to the health check service, where you can get an up-to-date view of which packages build successfully.
6 changes: 6 additions & 0 deletions src/ocamlorg_frontend/pages/changelog_entry.eml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ let render (entry : Data.Changelog.t) = match entry with
</section>

<section class="pb-6 lg:pb-8">
<div class="text-content dark:text-dark-content">
<%s if List.length release.authors > 0 then String.concat ", " release.authors else "" %>
</div>
<time datetime="2023-07-14"
class="block pb-5 mt-0 mb-6 font-sans text-sm text-content dark:text-dark-content border-b-2 border-solid border-separator_20 dark:border-dark-separator_30">
<%s Utils.human_date release.date %>
Expand Down Expand Up @@ -86,6 +89,9 @@ let render (entry : Data.Changelog.t) = match entry with
</section>

<section class="pb-6 lg:pb-8">
<div class="text-content dark:text-dark-content">
<%s if List.length post.authors > 0 then String.concat ", " post.authors else "" %>
</div>
<time datetime="2023-07-14"
class="block pb-5 mt-0 mb-6 font-sans text-sm text-content dark:text-dark-content border-b-2 border-solid border-separator_20 dark:border-dark-separator_30">
<%s Utils.human_date post.date %>
Expand Down
Loading