Document ZFS macOS setup. #42
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
||
| [OCurrent](https://github.com/ocurrent/ocurrent) is an OCaml library for building incremental pipelines. Pipelines can be thought of as directed graphs with each node being some kind of job that feeds its results into the next node. For an example, have a look at [the OCaml deploy pipeline](https://deploy.ci.ocaml.org/). | ||
|
|
||
| An integral part to most of the [OCaml pipelines](https://github.com/ocurrent/overview) is something called [OCluster](https://github.com/ocurrent/ocluster) and [OBuilder](https://github.com/ocurrent/obuilder). OCluster is a library and collection of binaries for managing an infrastructure for building and scheduling jobs. These jobs could be Docker jobs or OBuilder jobs. OBuilder is quite similar to `docker build` but written in OCaml and using just a notion of *an execution environment* (on Linux [`runc`](https://github.com/opencontainers/runc)) and a *snapshotting filesystem* ([Btrfs](https://btrfs.wiki.kernel.org/index.php/Main_Page), [ZFS](https://openzfs.org/wiki/Main_Page), or an inefficient but convenient copying backend using `rsync`). |
There was a problem hiding this comment.
Suggested change
| An integral part to most of the [OCaml pipelines](https://github.com/ocurrent/overview) is something called [OCluster](https://github.com/ocurrent/ocluster) and [OBuilder](https://github.com/ocurrent/obuilder). OCluster is a library and collection of binaries for managing an infrastructure for building and scheduling jobs. These jobs could be Docker jobs or OBuilder jobs. OBuilder is quite similar to `docker build` but written in OCaml and using just a notion of *an execution environment* (on Linux [`runc`](https://github.com/opencontainers/runc)) and a *snapshotting filesystem* ([Btrfs](https://btrfs.wiki.kernel.org/index.php/Main_Page), [ZFS](https://openzfs.org/wiki/Main_Page), or an inefficient but convenient copying backend using `rsync`). | |
| An integral part to most of the [OCaml pipelines](https://github.com/ocurrent/overview) is something called [OCluster](https://github.com/ocurrent/ocluster) and [OBuilder](https://github.com/ocurrent/obuilder). OCluster is a library and collection of binaries for managing an infrastructure that builds and schedules jobs. These jobs could be Docker jobs or OBuilder jobs. OBuilder is quite similar to `docker build` but written in OCaml and using just a notion of *an execution environment* (on Linux [`runc`](https://github.com/opencontainers/runc)) and a *snapshotting filesystem* ([Btrfs](https://btrfs.wiki.kernel.org/index.php/Main_Page), [ZFS](https://openzfs.org/wiki/Main_Page), or an inefficient but convenient copying backend using `rsync`). |
|
|
||
| ### Snapshotting Filesystem | ||
|
|
||
| On macOS, there is a [port of ZFS](https://openzfsonosx.org/) that, at the time, works quite well but not perfectly. Many hours were lost debugging what ended up being small bugs in ZFS. This is what inspired adding a very portable and reliable, but slow and memory-inefficient [`rsync` store backend](https://github.com/ocurrent/obuilder/pull/88). This is what is currently used in the macOS implementation, which so far has been very reliable, and once the caches are hot, the slow down isn't too noticeable. |
There was a problem hiding this comment.
Suggested change
| On macOS, there is a [port of ZFS](https://openzfsonosx.org/) that, at the time, works quite well but not perfectly. Many hours were lost debugging what ended up being small bugs in ZFS. This is what inspired adding a very portable and reliable, but slow and memory-inefficient [`rsync` store backend](https://github.com/ocurrent/obuilder/pull/88). This is what is currently used in the macOS implementation, which so far has been very reliable, and once the caches are hot, the slow down isn't too noticeable. | |
| On macOS, there is a [port of ZFS](https://openzfsonosx.org/) that, at the time, worked quite well but not perfectly. Many hours were lost debugging what ended up being small bugs in ZFS. This is what inspired adding a very portable and reliable, but slow and memory-inefficient [`rsync` store backend](https://github.com/ocurrent/obuilder/pull/88). This is what is currently used in the macOS implementation, which so far has been very reliable, and once the caches are hot, the slow down isn't too noticeable. |
Should this be "worked" since were talking past tense "at the time"?
|
|
||
| Opam, the OCaml package manager, is happy to have multiple opam roots on a system typically in the home directory of the user (`~/.opam`). This is great as running something like `sudo -u macos-builder-705 -i /bin/bash -c 'opam install irmin'` will use `macos-builder-705`'s home directory to build `irmin`. This provides an illusion of isolation similar to `runc`. | ||
|
|
||
| Homebrew, a macOS system package manager, is not so willing to be installed anywhere. [You can do it](https://docs.brew.sh/Installation#untar-anywhere), but as they say: |
There was a problem hiding this comment.
Suggested change
| Homebrew, a macOS system package manager, is not so willing to be installed anywhere. [You can do it](https://docs.brew.sh/Installation#untar-anywhere), but as they say: | |
| Homebrew, a macOS package manager, is not so willing to be installed anywhere. [You can do it](https://docs.brew.sh/Installation#untar-anywhere), but as they say: |
Tighter, if "system" will be understood without it there
|
|
||
| We want to have multiple Homebrew installations that all believe they are installed in `/usr/local`. Without this, Homebrew can't use "bottles," which are pre-built binaries making builds much faster. | ||
|
|
||
| The proposed solution is to mount a FUSE (filesystem in userspace) filesystem onto `/usr/local` which intercepts calls and redirects them based on who the calling user is. Since macOS Catalina, this requires [System Integrity Protection (SIP) to be disabled](https://developer.apple.com/documentation/security/disabling_and_enabling_system_integrity_protection). |
There was a problem hiding this comment.
Suggested change
| The proposed solution is to mount a FUSE (filesystem in userspace) filesystem onto `/usr/local` which intercepts calls and redirects them based on who the calling user is. Since macOS Catalina, this requires [System Integrity Protection (SIP) to be disabled](https://developer.apple.com/documentation/security/disabling_and_enabling_system_integrity_protection). | |
| The proposed solution is to mount a FUSE (filesystem in userspace) onto `/usr/local` which intercepts calls and redirects them based on who the calling user is. Since macOS Catalina, this requires [System Integrity Protection (SIP) to be disabled](https://developer.apple.com/documentation/security/disabling_and_enabling_system_integrity_protection). |
Since the F in FUSE stands for filesystem, does it need to specify a filesystem as well?
|
|
||
| ### Rsyncing to Home | ||
|
|
||
| With the scheme described above, each user's home directory is easy to find, as it is based on the `uid`. One problem though is that for each build step (`run`, `copy`...) we take a snapshot so we can later restore from it. This means we need to sync the home directory with the snapshot. This is also important for packages that are not relocatable (at the time of writing `ocamlfind` is an example). |
There was a problem hiding this comment.
Suggested change
| With the scheme described above, each user's home directory is easy to find, as it is based on the `uid`. One problem though is that for each build step (`run`, `copy`...) we take a snapshot so we can later restore from it. This means we need to sync the home directory with the snapshot. This is also important for packages that are not relocatable (at the time of writing `ocamlfind` is an example). | |
| With the scheme described above, each user's home directory is easy to find, as it is based on the UID. One problem though is that for each build step (`run`, `copy`...) we take a snapshot so we can later restore from it. This means we need to sync the home directory with the snapshot. This is also important for packages that are not relocatable (at the time of writing OCamlFind is an example). |
Should these be in monospace or be the acronym and proper name, respectively?
| This includes: | ||
|
|
||
| - Having both `opam.2.0.X` and `opam.2.1.X` installed and ready to use by sym-linking to `/usr/local/bin/opam` (a.k.a. `~/local/bin/opam`). | ||
| - Being clever (thanks @kit-ty-kate) with the `.bash_profile` script to reuse the name of the base image (i.e., `(from "macos-homebrew-ocaml-4.11")`) to setup things like the path to the system compiler. This means the OBuilder specs don't need to worry about the OCluster worker implementation details. |
There was a problem hiding this comment.
Suggested change
| - Being clever (thanks @kit-ty-kate) with the `.bash_profile` script to reuse the name of the base image (i.e., `(from "macos-homebrew-ocaml-4.11")`) to setup things like the path to the system compiler. This means the OBuilder specs don't need to worry about the OCluster worker implementation details. | |
| - Being clever (thanks @kit-ty-kate) with the `.bash_profile` script that reuses the base image name (i.e., `(from "macos-homebrew-ocaml-4.11")`) to setup things like the path to the system compiler. This means that the OBuilder specs don't need to worry about the OCluster worker implementation details. |
| sudo rsync -aHq /Users/mac705/ /Users/macos-homebrew-ocaml-4.14 | ||
| ``` | ||
|
|
||
| And use this from stage in your spec file `(from macos-homebrew-ocaml-4.14)`. Note that the OCluster PR still uses the Docker fetcher in the PR, but not on the machines. The machines have the following small patch that hopefully in the future can be removed. |
There was a problem hiding this comment.
Suggested change
| And use this from stage in your spec file `(from macos-homebrew-ocaml-4.14)`. Note that the OCluster PR still uses the Docker fetcher in the PR, but not on the machines. The machines have the following small patch that hopefully in the future can be removed. | |
| And use this from the stage in your spec file `(from macos-homebrew-ocaml-4.14)`. Note that the OCluster PR still uses the Docker fetcher in the PR, but not on the machines. The machines have the following small patch that hopefully in the future can be removed. |
Should this be "the stage" or "a stage"?
|
|
||
| ## Starting and Stopping | ||
|
|
||
| `ocluster-worker` is run via LaunchAgent. |
There was a problem hiding this comment.
Suggested change
| `ocluster-worker` is run via LaunchAgent. | |
| `ocluster-worker` is run via LaunchAgent. |
Should this be OCluster-Worker or the command ocluster-worker. Seems like the latter. It was written Ocluster-worker, but since it's being run via LaunchAgent, I figured it was a comman: ocluster-worker.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Promotes the current ZFS based implementation to the main README.md for the repository and keeps the previous rsync version in README-rsync.md for reference.