Skip to content

Commit

Permalink
Merge pull request #75 from vigh-m/core-kit-readme
Browse files Browse the repository at this point in the history 
Adding a README.md and updating the BUILDING.md
  • Loading branch information
@vigh-m
vigh-m authored Sep 12, 2024
2 parents 10dd6b5 + e8af5ff commit fd5148f
Show file tree
Hide file tree
Showing 2 changed files with 76 additions and 190 deletions.
243 changes: 53 additions & 190 deletions BUILDING.md
View file
Original file line number Diff line number Diff line change
@@ -1,244 +1,107 @@
# Building Bottlerocket
# How to build the Bottlerocket core kit

If you'd like to build your own image instead of relying on an Amazon-provided image, follow these steps.
You can skip to the [setup guide for Kubernetes](QUICKSTART-EKS.md) or the [setup guide for Amazon ECS](QUICKSTART-ECS.md) to use an existing image in Amazon EC2.
(We're still working on other use cases!)

## Build an image

### Dependencies
If you'd like to build your own copy of the core kit for local development, follow these steps.

## Dependencies
#### System Requirements

The build process artifacts and resulting images can consume in excess of 80GB in the local directory.

The build process is also fairly demanding on your CPU, since we build all included software from scratch.
(The first time. Package builds are cached, and only changes are built afterward.)
The build process is also fairly demanding on your CPU, since we build all included software from scratch. This is only done the first time. Package builds are cached, and only changes are built afterward.
The build scales well to 32+ cores.
The first time you build, the fastest machines can take about 12 minutes while slower machines with only a couple cores can take 3-4 hours.

The first time you build, the fastest machines can take about 20 minutes while slower machines with only a couple cores can take 3-4 hours.
#### Linux

The build system requires certain operating system packages to be installed.

Ensure the following OS packages are installed:

Ensure the following packages are installed:
##### Ubuntu

```shell
apt install build-essential openssl libssl-dev pkg-config liblz4-tool
```

##### Fedora

```shell
yum install make automake gcc openssl openssl-devel pkg-config lz4 perl-FindBin perl-lib
```


#### Rust

The build system is based on the Rust language.
We recommend you install the latest stable Rust using [rustup](https://rustup.rs/), either from the official site or your development host's package manager.
Rust 1.51.0 or higher is required.

To organize build tasks, we use [cargo-make](https://sagiegurari.github.io/cargo-make/).
To get it, run:

```shell
cargo install cargo-make
```

#### Docker
### OCI Artifacts

Building a kit results in building OCI artifacts, there are two ways to build these artifacts: `crane` or `docker` with the `containerd-snapshotter` feature enabled.

Bottlerocket uses [Docker](https://docs.docker.com/install/#supported-platforms) to orchestrate package and image builds.
We recommend using `crane` (and `krane`) over `docker` as it has shown better performance in our testing.

We recommend Docker 20.10.10 or later.
#### Docker
We recommend [Docker](https://docs.docker.com/install/#supported-platforms) 20.10.10 or later. The default seccomp policy of older versions of Docker do not support the `clone3` syscall in recent versions of Fedora or Ubuntu, on which the Bottlerocket SDK is based.
Builds rely on Docker's integrated BuildKit support, which has received many fixes and improvements in newer versions.
The default seccomp policy of older versions of Docker do not support the `clone3` syscall in recent versions of Fedora or Ubuntu, on which the Bottlerocket SDK is based.

You'll need to have Docker installed and running, with your user account added to the `docker` group.
Docker's [post-installation steps for Linux](https://docs.docker.com/install/linux/linux-postinstall/) will walk you through that.

> Note: If you're on a newer Linux distribution using the unified cgroup hierarchy with cgroups v2, you may need to disable it to work with current versions of runc.
> You'll know this is the case if you see an error like `docker: Error response from daemon: OCI runtime create failed: this version of runc doesn't work on cgroups v2: unknown.`
> Set the kernel parameter `systemd.unified_cgroup_hierarchy=0` in your boot configuration (e.g. GRUB) and reboot.
### Build process

To build an image, run:

```shell
cargo make
You'll also need to enable the containerd-snapshotter and buildkit features for your docker daemon. This is required to ensure docker compatibility with OCI Images (which kits are stored in).
The following configuration is needed in your `/etc/docker/daemon.json`
```json
{
"features": {
"buildkit": true,
"containerd-snapshotter": true
}
}
```
#### Crane
[Crane](https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md) is a tool for interacting with remote images and registries.. It does not require a daemon and thus you don't need the above Docker features to use it. Twoliter supports utilizing `crane` (or `krane`) instead of `docker` if it is installed.

This will build an image for the default variant (a recent `aws-k8s-*`, see the `BUILDSYS_VARIANT` variable in [Makefile.toml](Makefile.toml) to find the current default variant).
All packages will be built in turn, and then compiled into an `img` file in the `build/images/` directory.

The version number in [Release.toml](Release.toml) will be used in naming the file, and will be used inside the image as the release version.
If you're planning on [publishing your build](PUBLISHING.md), you may want to change the version.
The installation instructions for [crane](https://github.com/google/go-containerregistry/tree/main/cmd/crane) should help you set it up for use with Twoliter.

To build an image for a different variant, run:
## Build the core kit

```shell
cargo make -e BUILDSYS_VARIANT=my-variant-here
Building the core kit can be done by using the makefile targets.
```

To build an image for a different architecture, run:

```shell
cargo make -e BUILDSYS_ARCH=my-arch-here
make ARCH=<architecture>
```

If you want to limit the build concurrency, set `BUILDSYS_JOBS` (the default is `8`):
## Publish the Kit
After the kit has been built you can then publish the kit image to your private registry. This will allow you to consume it to build and test a variant.

```shell
cargo make -e BUILDSYS_JOBS=4
```

(You can use variant and arch arguments together, too.)

#### Package licenses
### Use a private registry for development
It is recommended that you have some form of protected container registry to use for testing.
For testing purposes you can either utilize mutable tags to allow overriding of multiple versions of a core kit as you test, or you can use immutable tags and continuously bump the core kit version via the `Twoliter.toml`.

Most packages will include license files extracted from upstream source archives.
However, in some rare cases there are multiple licenses that could apply to a package.
Bottlerocket's build system uses the `Licenses.toml` file in conjunction with the `licenses` directory to configure the licenses used for such special packages.
Here is an example of a simple `Licenses.toml` configuration file:

```toml
[package]
spdx-id = "SPDX-ID"
licenses = [
{ path = "the-license.txt" }
]
### Configure Infra.toml
An `Infra.toml` file needs to be created and should have a definition of your vendor (container registry) in order to publish the kits you build. To do so make sure that the `Infra.toml` has the below.
```

In the previous example, it is expected that the file `the-license.txt` is present in `licenses`.
You can retrieve the licenses from a remote endpoint, or the local filesystem if you specify the `license-url` field:

```toml
[package]
spdx-id = "SPDX-ID AND SPDX-ID-2" # Package with multiple licenses
licenses = [
# This file is copied from a file system, and will be saved as `path`
{ license-url = "file:///path/to/spdx-id-license.txt", path = "spdx-id-license.txt" },
# This file is fetched from an https endpoint, and will be saved as `path`
{ license-url = "https://localhost/spdx-id-license-v2.txt", path = "spdx-id-license-2.txt" }
]
```

#### NVIDIA variants

If you want to build any of the NVIDIA variants, you can follow these steps to prepare a `Licenses.toml` file using the [License for customer use of NVIDIA software](https://www.nvidia.com/en-us/drivers/nvidia-license/):

1. Create a `Licenses.toml` file in your Bottlerocket root directory, with the following content:

```toml
[nvidia]
spdx-id = "LicensesRef-NVIDIA-Customer-Use"
licenses = [
{ path = "LICENSE", license-url = "https://www.nvidia.com/en-us/drivers/nvidia-license/" }
]
[vendor.<vendor-name>]
registry = "####.dkr.ecr.us-west-2.amazonaws.com"
```

2. Fetch the licenses with this command:

```shell
cargo make -e BUILDSYS_UPSTREAM_LICENSE_FETCH=true fetch-licenses
After the kit has been built locally, the kit can be published to the provided vendor in `Infra.toml`. To do this, you will need docker credentials with ECR access. You can do this with,
```

3. Build your image, setting the `BUILDSYS_UPSTREAM_SOURCE_FALLBACK` flag to `true`, if you haven't cached the driver's sources:

```shell
K8S_VERSION=1.24
cargo make \
-e BUILDSYS_VARIANT=aws-k8s-${K8S_VERSION}-nvidia \
-e BUILDSYS_UPSTREAM_SOURCE_FALLBACK="true"
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin ####.dkr.ecr.us-west-2.amazonaws.com
```

### Register an AMI

To use the image in Amazon EC2, we need to register the image as an AMI.

To do this, you'll need to have your AWS account credentials setup on your system.
There are lots of ways to do this; one method is using [the `aws` CLI](https://aws.amazon.com/cli/) via its `configure` command with your user's access and secret keys.
If you're using an EC2 instance, the [EC2 instance's IAM role](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) will be used automatically if available.

For a simple start, pick an [EC2 region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions), then run:

```shell
cargo make -e PUBLISH_REGIONS=your-region-here ami
Finally, publishing the core kit images can be handled by the makefile target.
```

Note that the task ("ami") must come **after** the arguments to `cargo make` that are specified with `-e`.

Your new AMI ID will be printed after it's registered.

If you built your image for a different architecture or variant, just use the same arguments here:

```shell
cargo make -e PUBLISH_REGIONS=your-region-here -e BUILDSYS_VARIANT=my-variant-here ami
make publish VENDOR=<vendor-name>
```
At this point, there should be a core kit image in your private registry which can be consumed when building a variant to test and validate.

(There's a lot more detail on building and managing AMIs in the [PUBLISHING](PUBLISHING.md) guide.)

## Use your image
## Consuming the published kit image
This section will cover building a variant to test a build of the core kit as done above. Please note this section does not cover the complete complexity of testing a change to Bottlerocket. For this see the [BUILDING](https://github.com/bottlerocket-os/bottlerocket/blob/develop/BUILDING.md) section in the [Bottlerocket](https://github.com/bottlerocket-os/bottlerocket/) repository.

See any of the setup guides tailored to the various execution environments for information on running Bottlerocket images:

* [Setup guide for Kubernetes](QUICKSTART-EKS.md)
* [Setup guide for Amazon ECS](QUICKSTART-ECS.md)
* [Setup guide for VMware](QUICKSTART-VMWARE.md)
* [Setup guide for QEMU/KVM](QUICKSTART-LOCAL.md)

## Publish your image

See the [PUBLISHING](PUBLISHING.md) guide for information on deploying Bottlerocket images and repositories.

## Building out-of-tree kernel modules

To further extend Bottlerocket, you may want to build extra kernel modules.
The specifics of building an out-of-tree module will vary by project, but the first step is to download the "kmod kit" that contains the kernel headers and toolchain you'll need to use.

### Downloading the kmod kit

kmod kits are included in the official Bottlerocket repos starting with Bottlerocket v1.0.6.
Let's say you want to download the kit for building x86_64 modules for v1.11.0 and variant aws-k8s-1.24.

First, you need tuftool:
```shell
cargo install tuftool
### Configure Twoliter.toml
To consume a private copy of the Bottlerocket core kit with your changes built into it, you need to define the vendor that points to your container registry in `Twoliter.toml` and adjust the core kit dependency:
```

Next, you need the Bottlerocket root role, which is used by tuftool to verify the kmod kit.
This will download and verify the root role itself:
```shell
curl -O "https://cache.bottlerocket.aws/root.json"
sha512sum -c <<<"2ff1fbf99b20dd7ff5d2c84243a8e3b51701183b1f524b7d470a6b7a9b0172fbb36a0949b7e586ab7ccb6e348eb77125d6ed9fd1a638f4381e4f3f084ff38596 root.json"
[vendor.my-vendor]
registry = "####.dkr.ecr.us-west-2.amazonaws.com"
[[kit]]
name = "bottlerocket-core-kit" # Name of your ECR repo
version = "2.x.y" # your version tag you want to test
vendor = "my-vendor"
```

Next, set your desired parameters, and download the kmod kit:
```shell
ARCH=x86_64
VERSION=v1.11.0
VARIANT=aws-k8s-1.24
OUTDIR="${VARIANT}-${VERSION}"

tuftool download "${OUTDIR}" --target-name ${VARIANT}-${ARCH}-kmod-kit-${VERSION}.tar.xz \
--root ./root.json \
--metadata-url "https://updates.bottlerocket.aws/2020-07-07/${VARIANT}/${ARCH}/" \
--targets-url "https://updates.bottlerocket.aws/targets/"
Any time you change the vendor or version of the kit above you need to run `twoliter update` to update the `Twoliter.lock`
```

### Using the kmod kit

To use the kmod kit, extract it, and update your PATH to use its toolchain:
```shell
tar xf "${VARIANT}-${ARCH}-kmod-kit-${VERSION}.tar.xz"

export CROSS_COMPILE="${ARCH}-bottlerocket-linux-musl-"
export KERNELDIR="${PWD}/${VARIANT}-${ARCH}-kmod-kit-${VERSION}/kernel-devel"
export PATH="${PWD}/${VARIANT}-${ARCH}-kmod-kit-${VERSION}/toolchain/usr/bin:${PATH}"
./tools/twoliter/twoliter update
```

Now you can compile modules against the kernel headers in `${KERNELDIR}`.
23 changes: 23 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Bottlerocket Core Kit
This is the core kit for [Bottlerocket](https://github.com/bottlerocket-os/bottlerocket).
It includes many common dependencies for downstream package and variant builds.

## Contents
The core kit includes:
* Shared libraries such as glibc and libz
* Management daemons such as systemd and dbus-broker
* Agents for settings API and in-place updates

### Availability
The [Bottlerocket core kit](https://gallery.ecr.aws/bottlerocket/bottlerocket-core-kit) is available through Amazon ECR Public.

### Development
The core kit can be built on either an **x86_64** or an **aarch64** host. To do this you can use the following commands.
```shell
make
```
OR
```shell
make ARCH=<aarch64, x86_64>
```
See the [BUILDING](BUILDING.md) guide for more details.

0 comments on commit fd5148f

Please sign in to comment.