Is it possible to configure a pod to spin up containers on the host machine?

[artburkart]

I'm struggling to prove out an example where I make a container in a Kubernetes pod spin up another container on a host machine running Bottlerocket. Naturally, some variation of it is possible using the "admin" and "control" containers, but I'm starting to wonder whether it's possible in the context of Kubernetes.

Thanks for your help!

UPDATE:
The question I should've asked was whether I could use ctr to create a container inside of a container on Bottlerocket.

[bcressey]

You can map in the host's /usr/bin/apiclient and /run/api.sock to a privileged pod, then use the settings API to start a static pod:

apiclient set \
  kubernetes.static-pods.my-pod.manifest=${BASE64_BLOB} \
  kubernetes.static-pods.my-pod.enabled=true

kubelet will register the static pod as a mirror pod when it connects to the Kubernetes API server.

To create a "real" pod, you'd need to run your pod with a service account and role that allowed pod creation. I haven't tried this, but I expect the role would be something like the pod-reader example with mutating verbs added.

[bcressey]

@samuelkarp tells me that containerd isn't really designed to do what you're trying to do; there's a fundamental expectation that the client will run in the same namespaces as the daemon.

[artburkart]

That's fair. Maybe I'm barking up the wrong tree.

[artburkart]

Yeah, so I definitely thought the industry had gone in the direction of interacting with the host's containerd socket to create containers, but as it turns out, I was totally wrong about that! 😅

Back in 2015, I was trying desperately to get DinD to work, and it simply would not. And I seemed to recall that it was strongly discouraged back then. However, now it seems people use it regularly, and that's actually the root of the problem I'm hitting right now.

It seems that I can't get containerd to create containers from within a container on Bottlerocket that normally can on other operating systems. Do you know if there's been any word about why that might be? Meanwhile, I'm trying to see if I can provide a real test case that proves the point.

[artburkart]

Also, it might be appropriate for me to open a separate question, since my original question here has been answered (thank you very much!).

[artburkart]

I don't think my problem is related to Bottlerocket at all. I just did a little test to verify that I could indeed create containers using containerd inside of a container running on Bottlerocket.

helm repo add concourse https://concourse-charts.storage.googleapis.com/
helm install --create-namespace --namespace test-case -f <(echo '{"concourse": {"worker": {"runtime": "containerd"}}}') test-case concourse/concourse
kubectl -n test-case exec -it test-case-worker-0 -- bash 
/usr/local/concourse/bin/ctr --address "/run/containerd/containerd.sock" images pull docker.io/library/redis:alpine
/usr/local/concourse/bin/ctr --debug --address "/run/containerd/containerd.sock" containers list
/usr/local/concourse/bin/ctr --debug --address "/run/containerd/containerd.sock" containers create docker.io/library/redis:alpine redis
/usr/local/concourse/bin/ctr --debug --address "/run/containerd/containerd.sock" containers delete redis

The above commands should work on any old Kubernetes cluster. I've tried them on several at this point without issue. I've confirmed that I can create a container inside of a container on Bottlerocket.

Sorry for dragging you along with me through this saga. Thank you for your patience. I'm going to mark this as resolved.