infra-toolkit busybox tar causes k8s node OOM

[bizrad]

We are using the cosmos operator in our k8s cluster. When downloading the snapshot in the init container that runs ghcr.io/strangelove-ventures/infra-toolkit:v0.1.6 our k8s node with 256GB of ram gets OOM killed. Before this happens this can be observer by watching free -h under "buff/cache". This is memory used by the kernel to cache disk I/O operations and therefore not included in pod memory limits or usage tracking.

The infra-toolkit image is based on busybox and uses a stripped down version of tar. Our snapshot is compressed with lz4 and is being extracted by this script in the init container.

I was able to run a test and simply switching to an Ubuntu image with GNU tar in place and running the same commands resolved this issue. GNU tar is better suited for handling large archives with optimized memory usage.

Would it be possible to switch the image to GNU tar and put this in the path before busybox tar?

Alternatively, is it possible to swap out the init container image?

[agouin]

Thank you for the issue! We also use lz4 compressed snapshots, so I am surprised that we have not run into this ever for our deployments. We will review the switch to gnu tar in infra-toolkit and publish a new operator version if accepted.

is it possible to swap out the init container image?

Yes, this is possible with the strategic merge patching of the spec.podTemplate, e.g.:

apiVersion: cosmos.strange.love/v1
kind: CosmosFullNode
spec:
  podTemplate:
    initContainers:
       - name: snapshot-restore
         image: $CUSTOM_IMAGE_REPO:$CUSTOM_IMAGE_TAG

[bizrad]

It's possible the reason could be differences in resources. I'm seeing DL speed of about 800Mb/s. This was seen on a mostly unused node using 8 of 48 cores at idle. Since swapping out tar makes the difference I'm guessing it's most likely that the lz4 decompress is faster than busybox tar can handle in our environment.

[nourspace]

@pharr117 did we finalise the test in https://github.com/strangelove-ventures/infra/pull/3336?

[nourspace]

Ping @pharr117