runnc allocates to much Space on /run

[timstoffel]

Hi,

i using nabla on docker. If I create too many nabla containers (python-tornado) at once, I get the error from the eleventh:
Error response from daemon: write /var/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/cdf3e4285f5025503cee1371efaea1438d8fc01228adbea0b1bf23439dfad6cc/config.json: no space left on device: unknown.

It seems like /run is getting full.

Filesystem      Size  Used Avail Use% Mounted on
udev            7.8G     0  7.8G   0% /dev
tmpfs           1.6G  1.6G   28K 100% /run
/dev/nvme0n1p2  218G   16G  191G   8% /
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/nvme0n1p1  511M   22M  490M   5% /boot/efi
tmpfs           1.6G     0  1.6G   0% /run/user/1000

If i inspect the /run, i get the following:

4.0K    /run/blkid
0       /run/docker/netns
0       /run/docker/runtime-runnc/moby/78d3c23d287d458792a060809e6868111c34621cda91e632be23f50a74c2ab53
125M    /run/docker/runtime-runnc/moby/40d75a82307992072cddbcbbbf01f88f52aebca9871b8a773ed5dae0cb1f73f3
127M    /run/docker/runtime-runnc/moby/5781e45ef08f95d86c41d7ddcfbed7d957163334fd4453c0e5ee3c53aaf0c77f
127M    /run/docker/runtime-runnc/moby/b7a345bae853ac236d5cac4982eaa58b7433a2509ba2be257d88cab9d9e4a5de
127M    /run/docker/runtime-runnc/moby/87ce9bb5c4926b01f53c71b28e5d11b768a536a067a5b5e7496e5e28ce32dfc2
127M    /run/docker/runtime-runnc/moby/61a1533f8d4cd6876f692bffc9ee61b0b3f60b257ec0a80418ada3d5af732a70
127M    /run/docker/runtime-runnc/moby/554c1dcf3149dc3cb2f409a37cf4eecfb28cbf2442799c852061d0b4753c3b45
127M    /run/docker/runtime-runnc/moby/8f97002f8ffe02dc8975064d44de7bed03daa30d3a54960f3a972bc1cb3ce62d
127M    /run/docker/runtime-runnc/moby/23b3f7c9d744bceaa3f7e74f7cbdd9c888e205f8fa69f49ce1e3b442fc8c2be6
127M    /run/docker/runtime-runnc/moby/dd518c88d16b5b23004338c30b02aaeadd4ed8da0bc41763590b2ba8c4985e83
127M    /run/docker/runtime-runnc/moby/e8bb84bb3311cd88389070f8c8418186f0e5326c2574de8ab0410f9c286e8ede
127M    /run/docker/runtime-runnc/moby/76fe4e3a1805d5805d483b684648702cc48508f561cdf88dd4bf7568bc05588a
127M    /run/docker/runtime-runnc/moby/1648a72a1312112f1553cb2aa865402801cebe9d965952fea6422aa081d96b69
1.5G    /run/docker/runtime-runnc/moby
1.5G    /run/docker/runtime-runnc

It looks to me like the files under /run/docker/runtime-runnc/moby/ are filling up the drive.

/run on this system now has 1.6 GB which is exactly 10% of the available RAM.

What can I do to solve the problem?
Can I change the path?

Why does runnc write the files to a tmpfs?

Kind Regrads

Tim

[lumjjb]

Yea you can change the path by changing the configuration of the docker daemon so that it points to a disk backed file system.

Why does runnc write the files to a tmpfs?

this is actually done by the runtime, when performing unpacking of the file system. It is a combination of the runtime storage driver and runnc creating the ISO on top of the container runtime bundle

[timstoffel]

Hi!

Thanks for the answer

I thought for the daemon I could change the path /var/lib/docker/. But this path does not point to /run, does it?

[lumjjb]

Hmm yea it looks like its not configurable.. can you try to use another storage driver like overlay2 and see if it helps, it should instead mount something from /var/lib/docker instead of copying things over.

[timstoffel]

Hi

tanks for your answer.

I am writing a scientific paper comparing the management of Nabla, Kata, Kata Firecracker and gVisor.

Kata FC is currently limited to the devicemapper as storage-driver. Therefore I use this one. If I configure a different storage-driver for Nabla, the results are probably difficult to compare.