Seekable OCI and the Kubernetes Runtime Class

[ollypom]

You can use a Kubernetes Runtime Class to choose which pods are sent to the SOCI snapshotter and which stay with the default (overlayfs) snapshotter, as mentioned by @Kern-- . The feature is available in containerd 1.7, and yesterday the EKS AMI started shipping with containerd 1.7 🎉, so I wanted to quickly write up my experience of getting started with Kubernetes Runtime Class and SOCI on EKS.

Environment

• EKS Cluster using Kubernetes 1.28
• One node in a node group using the latest AMI 1.28.3-20231201
• soci-snapshotter-grpc v0.4.0 binary on the node and managed / started by systemd
• containerd config (/etc/containerd/config.toml)
  • Set the global snapshotter for CRI as overlayfs (this is the default for all pods)
  • A dedicated soci runtime class in containerd which uses the soci snapshotter

my containerd config

version = 2

root = "/var/lib/containerd"
state = "/run/containerd"

[grpc]
address = "/run/containerd/containerd.sock"

[plugins."io.containerd.grpc.v1.cri".containerd]
# Set the default snapshotter to overlayfs
snapshotter = "overlayfs"
# disable_snapshot_annotations disables to pass additional annotations (image
# related information) to snapshotters. These annotations are required by
# stargz snapshotter (https://github.com/containerd/stargz-snapshotter)
disable_snapshot_annotations = false
# default_runtime_name is the default runtime name to use.
default_runtime_name = "runc"
# discard_unpacked_layers allows GC to remove layers from the content store after
# successfully unpacking these layers to the snapshotter.
discard_unpacked_layers = true

# Hard set the pause container
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/pause:3.5"

# Taken from the EKS Optimized AMI
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"

# Create a dedicated runtime class for SOCI
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.soci]
runtime_type = "io.containerd.runc.v2"
snapshotter = "soci"

# Taken from the EKS Optimized AMI
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

# Taken from the EKS Optimized AMI
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true

# Taken from the EKS Optimized AMI
[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"

# Configure the SOCI Snapshotter as a Remote Snapshotter for containerd
[plugins]
[proxy_plugins.soci]
type = "snapshot"
address = "/run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock"

• SOCI config (/etc/soci-snapshotter-grpc/config.toml)
  • Using the containerd content store under the k8s.io namespace for my SOCI artifacts (not mandatory).

my soci-snapshotter config

# config/fs.go FSConfig
http_cache_type=""
filesystem_cache_type=""
resolve_result_entry=0 # Actually zero
debug=false
allow_no_verification=true
# disable_verification=false
# Causes TestRunWithDefaultConfig to break, but
# fine to use in /etc/soci-snapshotter-grpc-config.toml
max_concurrency=0  # Actually zero
no_prometheus=false
mount_timeout_sec=0
fuse_metrics_emit_wait_duration_sec=0

## config/config.go Config

metrics_address=""
metrics_network="" # Uses default metrics network
# no_prometheus=true # Defined above, can't be redeclared
debug_address=""
metadata_store="db"

[http]
MaxRetries=0
MinWaitMsec=0
MaxWaitMsec=0
DialTimeoutMsec=0
ResponseHeaderTimeoutMsec=0
RequestTimeoutMsec=0

#
## config/fs.go
#

[blob]
valid_interval=0
check_always=false
fetching_timeout_sec=0
force_single_range_mode=false
# max_retries=0 # Set by http.
# min_wait_msec=0 # Set by http.
# max_wait_msec=0 # Set by http.
max_span_verification_retries=0 # Actually zero

[directory_cache]
max_lru_cache_entry=0 # Actually zero
max_cache_fds=0 # Actually zero
sync_add=false
direct=true

[fuse]
attr_timeout=0
entry_timeout=0
negative_timeout=0
log_fuse_operations=false

[background_fetch]
disable=false
silence_period_msec=0
fetch_period_msec=0
max_queue_size=0
emit_metric_period_sec=0

[content_store]
type="containerd" # will set to 'soci' by default
namespace="k8s.io" # will set to 'default' by default

#
## config/resolver.go
#

[resolver]
  [resolver.host]

#
## config/service.go
#

[kubeconfig_keychain]
enable_keychain=false
kubeconfig_path=""

[cri_keychain]
enable_keychain=false
image_service_path="" # Uses default image service address

[snapshotter]
min_layer_size=0 # Actually zero
allow_invalid_mounts_on_restart=false

#
## service/resolver/cri.go
#

[registry]
config_path=""
mirrors={}
configs={}

[Mirror]
endpoint={}

[RegistryConfig]

[auth]
username=""
password=""
auth=""
identitytoken=""

[tls]
insecure_skip_verify=false
ca_file=""
cert_file=""
key_file=""

• Using the docker-ecr-credential-helper so that the soci snapshotter can authenticate to Amazon ECR. I created a Docker credentials file (/root/.docker/config.json)

my credentials config

{ "credsStore":"ecr-login" }

• A created a Kubernetes runtime class for SOCI

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: soci-runtime
handler: soci

Failed Experiment

• Pod has runtimeClassName set to soci-runtime

Pod Spec

apiVersion: v1
kind: Pod
metadata:
  name: nginxsoci
  namespace: default
spec:
  runtimeClassName: soci-runtime 
  containers:
  - name: nginx
    image: 111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest
    ports:
    - containerPort: 80

The pod fails with a CreateContainerError

NAMESPACE     NAME                       READY   STATUS                 RESTARTS   AGE    IP             NODE                                        NOMINATED NODE   READINESS GATES
default       nginxsoci                  0/1     CreateContainerError   0          8s     10.1.219.241   ip-10-1-146-99.eu-west-1.compute.internal   <none>           <none>

Looking in the containerd logs

 "msg": "CreateContainer within sandbox \\\"87aaddba77d41a454f77463e01c27d4f30f8591d6a0e23abc27cc966a0ce7dbf\\\" for &ContainerMetadata{Name:nginx,Attempt:0,} failed\" error=\"rpc error: code = NotFound desc = failed to create containerd container: error unpacking image: failed to extract layer sha256:569368f508c5ea612bb63fcb7b6a8d41549bbfdf87c352bf24a6bdc81ebad241: failed to get reader from content store: content digest sha256:0b4a6f011995244a95bff79a1298e83d230bc0aa22871a9c510745cafebec227: not found"

So it appears containerd is unable to extract the base layer of my image (this was odd as containerd should not be doing image management). Checking the soci-snapshotter logs they are empty... This container image is not passed to the soci snapshotter :(

Successful Experiment

• Pod has runtimeClassName set to soci-runtime
• Pod has a runtime-handler annotation set to the runtime in my containerd config soci

Pod Spec

apiVersion: v1
kind: Pod
metadata:
  name: nginxsoci
  namespace: default
  annotations:
    io.containerd.cri.runtime-handler: "soci"
spec:
  runtimeClassName: soci-runtime
  containers:
  - name: nginx
    image: 111222333444.dkr.ecr.eu-west-1.amazonaws.com/nginx-demo:latest
    ports:
    - containerPort: 80

The pod is now running!!!

NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE     IP             NODE                                        NOMINATED NODE   READINESS GATES
default       nginxsoci                  1/1     Running   0          61s     10.1.221.130   ip-10-1-146-99.eu-west-1.compute.internal   <none>           <none>

Checking the soci snapshotter logs, there are log messages! containerd has delegated image management for this pod to the soci-snapshotter :)

[Kern--]

Something I missed when I initially suggested using SOCI in a dedicated runtime class is that k8s only very recently added alpha support for runtime-class-aware images (kubernetes/enhancements#4216) and it's not yet supported in containerd (containerd/containerd#9377).

The problem is that while you can specify a snapshotter to use in a runtime class, kubelet/CRI don't know about it without this new feature. To them an image is independent of runtime classes. So, if you create a pod with a SOCI runtime class, all the images will be pulled with the SOCI snapshotter. Later, if you try to create a pod with the default runtime class (which uses OverlayFS), then kubelet/CRI will see that the pause image (for example) already exists so they will skip the image pull. When kubelet tries to launch it, however, the image won't be found because containerd is expecting it in the OverlayFS snapshotter, but it's actually in the SOCI snapshotter.

This makes runtime classes with SOCI pretty much identical to just setting SOCI as the default snapshotter for all pods because you can only successfully use one runtime class snapshotter per node. Thankfully the problem is already known upstream and there is work being done to resolve the issue.

[dims]

we would need containerd/containerd#5105 as well i think!

[dims]

I experimented a bit in awslabs/amazon-eks-ami@master...dims:amazon-eks-ami:support-for-soci-snapshotter for starters nerdctl in AL/AL2023 are stale, so please pull new 1.7.2 at least if someone is trying this out. (bunch of my notes are here)