Garden Architecture
Garden is a simple single-host OCI container manager implementing the Garden API.
There are two Garden deployments scenarios:
- bosh job - as part of a CF deployments
- standalone server - the only user known to this deployment scenario is Concourse CI. However, Concourse are moving away from Garden and are advising their users to migrate to
containerd.
Another aspect of Garden deployment scenario is rootful vs. rootless, i.e. whether Garden server runs as the root user or as a non-privileged one. Running as root is the default; native rootless support never got out of the internal experiment phase. A rootless-like support was introduced via BPM, this feature however is still experimental - cf-deployment experimental ops file - and will probably remain experimental forever.
Garden consists of API and backend (Guardian).
The API and the Garden client provide the capability to manage containers and perform actions such as run process, attach to a process, manage IO, manage network, get metrics, etc. The API talks to the Guardian server over tcp sockets or HTTP, and more precisely to the Gardener backend.
Guardian consists of:
- Gardener - the backend, handles incoming requests
- Containerizer - manages containers
- Volumizer - takes care of containers filesystem
- Networker - deals with containers network configuration
The containerizer uses the OCIRuntime interface to manage containers. The interface has two implementations:
-
RunRunC- calls the runc binary -
RunContainerd- talks to the containerd server
The RunRunC implementation of the the OCIRuntime interface uses the runc external binary to manage containers. In fact, the path to the binary is configured in the Garden configuration and therefore RunRunC can talk to any binary that implements the runc binary interface, such as winc to support windows containers.
Da Doo ron ron sounds like dadoo runs runc which makes dadoo a perfect name for a component that invokes the runc binary. It also acts as a shim between Garden and the container itself. For every process running in a Garden container there is a running dadoo process which is the parent of the container process. The dadoo process does not exit when Guardian shuts down thus allowing it to connect to processes that already exist upon garden server restart.
The depot is a container state storage located at /var/vcap/data/garden/depot on the host machine. Whenever a container is created, or a process is run, Guardian would put container related data (such as the container config.json, the container creation time, exit code, etc.) there.
As the name suggests, RunContainerd talks to containerd. Under the hood it invokes the containerd client.
We did not want to spread containerd types all over the place in Guardian, that is why we came up with the nerd which translates Golang and OCI runtime primitives into invocations of the containerd client.
The Networker configures container network settings so that containers can access the network and receive incoming requests. There are two Networker interface implementations - Kawasaki and the external networker
Kawasaki is named after Guy Kawasaki, an evangelist and author of books and essays about networking (but as in social networks, not Ethernet). Kawasaki is an amazing networker! Later on the container networking team in the CF Foundation created Ducati, the next gen CF container networking solution. The name was a humorous response to Kawasaki. Kawasaki is a naive implementation which talks to iptables on the host machine to configure firewall rules for containers. It is turned off in CF-Deployment, we mainly use it for test purposes.
The external networker is a facility to invoke an external networking binary (its path is configured in the Garden configuration). In CF Deployment this is the runc-cni binary, part of cf-networking.
In rootful deployments Garden invokes the network binary directly, this is however not possible in bpm deployments due to linux namespaces limitations. Instead, in bpm context the external networker would talk to the network plugin shim via the shim client which implements the same binary interface as the cf-networker binary (runc-cni). This comment from Ed King explains why we need this infrastructure.
The Volumizer manages container filesystem via fetching, unpacking and mounting layers of root filesystem images, such as docker ones. Similarly to the networker, the Volumizer calls out to a binary (its path is configured in the Garden configuration) via the image plugin. We are aware of two binaries that implement the binary interface used by the image plugin:
- grootfs, used for linux containers
-
groot-windows, used for windows containers.
groot-windowsis built around groot, a framework for building custom image plugins for Garden.