Skip to content

Commit

Permalink
implement persistent storage for services
Browse files Browse the repository at this point in the history
The manifest.yaml now has a storage section where requests for
additional named partitions can be added, e.g.

storage:
  - label: zot-data
    persistent: true
    nsgroup: "zot"
    size: 30G

Each target also has a storage section specifying a list of
labels to mount and destinations at which to mount them.  A
nonpersistent storage label can be used if some target expects
to use a lot of temporary space with which it doesn't want to fill
up the shared /scratch-writes.  Persistent volumes can be used
to maintain data across reboots.

Signed-off-by: Serge Hallyn <[email protected]>
  • Loading branch information
hallyn committed Dec 23, 2023
1 parent 89f3a13 commit b14e85e
Show file tree
Hide file tree
Showing 14 changed files with 453 additions and 106 deletions.
8 changes: 6 additions & 2 deletions cmd/trust/launch.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ var launchCmd = cli.Command{
Usage: "Serial number UUID to assign to the machine, empty to use a random UUID",
Value: "",
},
cli.BoolFlag{
Name: "debug",
Usage: "show console during provision and install",
},
cli.BoolFlag{
Name: "skip-provisioning",
Usage: "Skip provisioning the machine",
Expand Down Expand Up @@ -142,7 +146,7 @@ func doLaunch(ctx *cli.Context) error {
}
}()

if err := m.RunProvision(); err != nil {
if err := m.RunProvision(ctx.Bool("debug")); err != nil {
return errors.Wrapf(err, "Failed to run provisioning ISO")
}

Expand All @@ -151,7 +155,7 @@ func doLaunch(ctx *cli.Context) error {
return nil
}

if err := m.RunInstall(); err != nil {
if err := m.RunInstall(ctx.Bool("debug")); err != nil {
return errors.Wrapf(err, "Failed to run install ISO")
}

Expand Down
74 changes: 74 additions & 0 deletions docs/storage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Storage for targes

Following is an example manifest.yaml showing how to specify storage
for targets:

```
storage:
- label: zot-data
persistent: true
nsgroup: "zot"
size: 30G
- label: zot-config
persistent: true
nsgroup: "zot"
size: 1G
- label: zot-tmp
persistent: false
nsgroup: "zot"
size: 1G
- label: nginx-data
persistent: true
nsgroup: "zot"
size: 1G
targets:
- service_name: zot
source: docker://zothub.io/machine/bootkit/demo-zot:0.0.4-squashfs
version: 1.0.0
nsgroup: zot
storage:
- dest: /zot
label: zot-data
- dest: /etc/zot
label: zot-config
- dest: /tmp
label: zot-tmp
- service_name: nginx
source: docker://zothub.io/machine/bootkit/demo-nginx:0.0.4-squashfs
version: 1.0.0
nsgroup: zot
storage:
- dest: /data/zot
label: zot-data
- dest: /var/lib/www
label: nginx-data
```

When a target starts up, its rootfs is an overlay of a writeable tmpfs
over the source OCI image (which itself is an overlay of dmverity-protected
squashfs images). The writeable overlays are all in a shared partition
mounted at /scratch-writes. In order to provide persistent storage
across boots, shared storage between containers, or a larger private
ephemeral storage which does not risk filling up /scratch-writes,
extra storage can be requested.

In the above example, four additional storage volumes are requested. The
30G volume called zot-data will be persistent, so its contents will be
saved across boots. In contrast, zot-tmp is not persistent, so its contents
will be deleted across reboots. All four are in the 'nsgroup zot', which
both of the targets, zot and nginx, run in. The nsgroup is a named
user namespace mapping, so uid 0 will be represented by the same host
uid (for instance 100000) for all.

Note that if nginx were not placed into nsgroup 'zot', it would still
be able to mount zot-data, however all files would appear as
owned by nobody:nogroup, and nginx would get the world access rights.

Each target now has an optional storage section, where it can
specify which volumes it should mount, and where.

On boot, the machine will first create the storage volumes, and uid-shift
them if needed. If a non-persistent volume already exists, it will be
deleted and recreated.

All storage volumes are created as ext4 filesystems.
44 changes: 22 additions & 22 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ require (
github.com/fatih/color v1.15.0
github.com/foxboron/go-uefi v0.0.0-20230218004016-d1bb9a12f92c
github.com/go-git/go-git/v5 v5.4.2
github.com/google/uuid v1.3.0
github.com/google/uuid v1.5.0
github.com/jsipprell/keyctl v1.0.4
github.com/lxc/lxd v0.0.0-20230130192612-1e882f91a2da
github.com/msoap/byline v1.1.1
Expand All @@ -22,7 +22,7 @@ require (
github.com/project-machine/machine v0.1.2
github.com/project-stacker/stacker v0.21.2
github.com/rekby/gpt v0.0.0-20200614112001-7da10aec5566
github.com/stretchr/testify v1.8.3
github.com/stretchr/testify v1.8.4
github.com/urfave/cli v1.22.12
golang.org/x/sys v0.15.0
golang.org/x/text v0.14.0
Expand All @@ -49,10 +49,10 @@ require (
github.com/containers/ocicrypt v1.1.7 // indirect
github.com/containers/storage v1.45.4 // indirect
github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
github.com/cyberphone/json-canonicalization v0.0.0-20220623050100-57a0ce2678a7 // indirect
github.com/cyphar/filepath-securejoin v0.2.4 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/docker/distribution v2.8.2+incompatible // indirect
github.com/docker/docker v24.0.7+incompatible // indirect
github.com/docker/docker-credential-helpers v0.7.0 // indirect
Expand Down Expand Up @@ -85,8 +85,8 @@ require (
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/google/go-containerregistry v0.14.0 // indirect
github.com/gorilla/mux v1.8.0 // indirect
github.com/gorilla/websocket v1.5.0 // indirect
github.com/gorilla/mux v1.8.1 // indirect
github.com/gorilla/websocket v1.5.1 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/imdario/mergo v0.3.15 // indirect
Expand All @@ -95,18 +95,18 @@ require (
github.com/json-iterator/go v1.1.12 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/kevinburke/ssh_config v0.0.0-20201106050909-4977a11b4351 // indirect
github.com/klauspost/compress v1.16.3 // indirect
github.com/klauspost/cpuid/v2 v2.2.4 // indirect
github.com/klauspost/compress v1.17.4 // indirect
github.com/klauspost/cpuid/v2 v2.2.6 // indirect
github.com/klauspost/pgzip v1.2.6-0.20220930104621-17e8dac29df8 // indirect
github.com/leodido/go-urn v1.2.4 // indirect
github.com/letsencrypt/boulder v0.0.0-20230213213521-fdfea0d469b6 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/martinjungblut/go-cryptsetup v0.0.0-20220520180014-fd0874fd07a6 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.19 // indirect
github.com/mattn/go-runewidth v0.0.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.15 // indirect
github.com/miekg/pkcs11 v1.1.1 // indirect
github.com/minio/sha256-simd v1.0.0 // indirect
github.com/minio/sha256-simd v1.0.1 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/moby/sys/mountinfo v0.6.2 // indirect
Expand All @@ -116,7 +116,7 @@ require (
github.com/opencontainers/runc v1.1.5 // indirect
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
github.com/pborman/uuid v1.2.1 // indirect
github.com/pelletier/go-toml/v2 v2.0.8 // indirect
github.com/pelletier/go-toml/v2 v2.1.1 // indirect
github.com/pkg/xattr v0.4.9 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/proglottis/gpgme v0.1.3 // indirect
Expand All @@ -131,8 +131,8 @@ require (
github.com/sigstore/fulcio v1.1.0 // indirect
github.com/sigstore/rekor v1.2.0 // indirect
github.com/sigstore/sigstore v1.6.4 // indirect
github.com/sirupsen/logrus v1.9.0 // indirect
github.com/spf13/afero v1.9.3 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/spf13/afero v1.11.0 // indirect
github.com/stefanberger/go-pkcs11uri v0.0.0-20201008174630-78d3cae3a980 // indirect
github.com/syndtr/gocapability v0.0.0-20200815063812-42c35b437635 // indirect
github.com/theupdateframework/go-tuf v0.5.2 // indirect
Expand All @@ -151,16 +151,16 @@ require (
go.mozilla.org/pkcs7 v0.0.0-20210826202110-33d05740a352 // indirect
golang.org/x/arch v0.3.0 // indirect
golang.org/x/crypto v0.17.0 // indirect
golang.org/x/exp v0.0.0-20230321023759-10a507213a29 // indirect
golang.org/x/mod v0.10.0 // indirect
golang.org/x/net v0.17.0 // indirect
golang.org/x/sync v0.2.0 // indirect
golang.org/x/exp v0.0.0-20231206192017-f3f8817b8deb // indirect
golang.org/x/mod v0.14.0 // indirect
golang.org/x/net v0.19.0 // indirect
golang.org/x/sync v0.5.0 // indirect
golang.org/x/term v0.15.0 // indirect
golang.org/x/tools v0.7.0 // indirect
golang.org/x/tools v0.16.1 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 // indirect
google.golang.org/grpc v1.56.3 // indirect
google.golang.org/protobuf v1.30.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20231212172506-995d672761c0 // indirect
google.golang.org/grpc v1.60.0 // indirect
google.golang.org/protobuf v1.31.0 // indirect
gopkg.in/go-jose/go-jose.v2 v2.6.1 // indirect
gopkg.in/square/go-jose.v2 v2.6.0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
Expand Down
Loading

0 comments on commit b14e85e

Please sign in to comment.