Skip to content
This repository has been archived by the owner on Apr 24, 2023. It is now read-only.

for containerd/cri please include note about /var/lib dir #88

Open
perezjasonr opened this issue Feb 11, 2021 · 1 comment
Open

for containerd/cri please include note about /var/lib dir #88

perezjasonr opened this issue Feb 11, 2021 · 1 comment

Comments

@perezjasonr
Copy link

I noticed that there are notes about changing the parser to cri if you are using containerd or cri-o.
but there is not an example manifest for it (daemonset, etc) and it doesn't say anywhere to also modify the volumes (i presume this is also not accounted for in the helm chart).

On a node running containerd as the runtime for k8s, there is no /var/lib/docker, but there is /var/lib/containerd, yet the ds will still mount (or try to) the docker dir.

ls -la /var/lib/docker
ls: cannot access '/var/lib/docker': No such file or directory
[root@ip-10-42-10-221 ~]# ls -la /var/lib/container*
/var/lib/containerd:
total 8
drwx------. 10 root root 4096 Jan 20 22:27 .
drwxr-xr-x. 30 root root 4096 Jan 20 22:27 ..
drwxr-xr-x.  3 root root   20 Jan 20 22:27 io.containerd.content.v1.content
drwx--x--x.  2 root root   21 Jan 20 22:27 io.containerd.metadata.v1.bolt
drwx--x--x.  2 root root    6 Jan 20 22:27 io.containerd.runtime.v1.linux
drwx--x--x.  2 root root    6 Jan 20 22:27 io.containerd.runtime.v2.task
drwxr-xr-x.  2 root root    6 Jan 20 22:27 io.containerd.snapshotter.v1.btrfs
drwx------.  3 root root   23 Jan 20 22:27 io.containerd.snapshotter.v1.native
drwx------.  3 root root   23 Jan 20 22:27 io.containerd.snapshotter.v1.overlayfs
drwx------.  2 root root    6 Jan 20 22:27 tmpmounts

/var/lib/containers:
total 4
drwxr-xr-x.  4 root root   37 Jan 20 22:27 .
drwxr-xr-x. 30 root root 4096 Jan 20 22:27 ..
drwxr-xr-x.  2 root root    6 Sep 23 16:19 sigstore
drwx------.  9 root root  169 Jan 20 22:27 storage
@z0rc
Copy link

z0rc commented May 21, 2021

AFAIK there is no need to mount /var/lib/containerd when running using cri runtime. /var/lib/docker mount point is needed when using dockerd runtime, because /var/log/containers/*.log are actually symlinks, real log files are in /var/lib/docker/containers/*/*.log.

With cri runtime /var/log/containers/*.log are still symlinks, but this time real files are in /var/log/pods/*.log.

solsson added a commit to Yolean/fluent-bit-kubernetes-logging that referenced this issue Aug 11, 2021
Displaying unparsed log entries using
kubectl apply -k base; kubectl rollout status daemonset fluent-bit; kubectl logs -f -l app.kubernetes.io/name=fluent-bit

Issues like fluent#88 and fluent/fluent-bit#1902 (comment)
indicate that depending on /var/log/containers symlinks cause
quite a few issues.

/var/log/pods/ is the path stated in
https://github.com/kubernetes/kubernetes/blob/v1.22.0/pkg/kubelet/kuberuntime/kuberuntime_manager.go#L63
and I've verified on GKE cos-containerd, GKE ubuntu-dockerd and k3s
that the path contains the actual files, not symliks.

Also using /var/log/pods makes it trivial to exclude logs from any
container named fluent-bit. Doing so reduces the risk of endless log loops.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants