Docker Container Infinitely Grows

[coryjreid]

Problematic Behavior

Something within the Docker container appears to be writing to the container's filesystem causing it to grow infinitely in size, resulting in the container's overlay2 file to eventually fill the disk.

Expected Behavior

Container doesn't write to un-mounted directories within the container (or, at least not continually).

Background

This VM only hosts this Docker container and an NGINX server which is a forward-proxy for some streaming URLs that need to remain private and has 32GB of disk space allocated to it.

In the below output you can see me run du -cs /var/lib/docker/* back-to-back with a period of only a few seconds between the two and the size of the overlay2 folder growing in that short time (the bot is actively streaming in Discord). You can also see I only have the one Docker container running.

root@muse:/var/lib/docker# du -cs /var/lib/docker/*
108     /var/lib/docker/buildkit
52      /var/lib/docker/containers
4       /var/lib/docker/engine-id
1988    /var/lib/docker/image
52      /var/lib/docker/network
3488800 /var/lib/docker/overlay2
16      /var/lib/docker/plugins
4       /var/lib/docker/runtimes
4       /var/lib/docker/swarm
4       /var/lib/docker/tmp
28      /var/lib/docker/volumes
3491060 total
root@muse:/var/lib/docker# du -cs /var/lib/docker/*
108     /var/lib/docker/buildkit
52      /var/lib/docker/containers
4       /var/lib/docker/engine-id
1988    /var/lib/docker/image
52      /var/lib/docker/network
3489024 /var/lib/docker/overlay2
16      /var/lib/docker/plugins
4       /var/lib/docker/runtimes
4       /var/lib/docker/swarm
4       /var/lib/docker/tmp
28      /var/lib/docker/volumes
3491284 total
root@muse:/var/lib/docker# docker ps -a
CONTAINER ID   IMAGE                    COMMAND                  CREATED        STATUS       PORTS     NAMES
2ee1175cc193   codetheweb/muse:latest   "docker-entrypoint.s…"   22 hours ago   Up 3 hours             muse
root@muse:/var/lib/docker#

This StackOverflow post suggested looking into files being written in the container by running docker container diff <container_name/id>, which I did, and you can see the output here.

For the sake of completeness, my compose file is:

version: '3.4'

services:
  muse:
    container_name: muse
    image: codetheweb/muse:latest
    restart: always
    network_mode: host # allows access to NGINX proxy on same VM
    volumes:
      - ./data:/data
    environment:
      - DISCORD_TOKEN=***
      - YOUTUBE_API_KEY=***
      - SPOTIFY_CLIENT_ID=***
      - SPOTIFY_CLIENT_SECRET=***

[codetheweb]

Interesting, it looks like tsx may be misbehaving. Could you check out the built image in the PR above and see if that fixes it?

If tsx is the root issue, maybe I'll switch back to pre-transpiling for faster startup times as well.

[coryjreid]

It appears the issue still exists, albeit the rate does seem slower.

I cloned the repo, checked out the branch, and ran docker build -t test . from the repository root. Then I modified my compose file to use the specific image that was built. If this fails to properly test please do let me know (I'm not super savvy on Docker).

Below you can see relevant command outputs.

root@muse:~# du -cs /var/lib/docker/*
3188    /var/lib/docker/buildkit
40      /var/lib/docker/containers
4       /var/lib/docker/engine-id
2004    /var/lib/docker/image
52      /var/lib/docker/network
3217924 /var/lib/docker/overlay2
16      /var/lib/docker/plugins
4       /var/lib/docker/runtimes
4       /var/lib/docker/swarm
4       /var/lib/docker/tmp
28      /var/lib/docker/volumes
3223268 total
root@muse:~# du -cs /var/lib/docker/*
3188    /var/lib/docker/buildkit
40      /var/lib/docker/containers
4       /var/lib/docker/engine-id
2004    /var/lib/docker/image
52      /var/lib/docker/network
3218516 /var/lib/docker/overlay2
16      /var/lib/docker/plugins
4       /var/lib/docker/runtimes
4       /var/lib/docker/swarm
4       /var/lib/docker/tmp
28      /var/lib/docker/volumes
3223860 total
root@muse:~# docker diff muse
A /data
C /tmp
A /tmp/yarn--1693530312744-0.025729490385082876
A /tmp/yarn--1693530312744-0.025729490385082876/node
A /tmp/yarn--1693530312744-0.025729490385082876/yarn
A /tmp/capacitor-8424f91a84d4c944638c8e3e9e185a58.tmp
A /tmp/envset-database-url-24d42b41.sh
C /root
C /root/.cache
C /root/.cache/checkpoint-nodejs
A /root/.cache/checkpoint-nodejs/prisma-82996626
A /root/.cache/checkpoint-nodejs/signature
C /root/.npm
A /root/.npm/_update-notifier-last-checked
A /root/.npm/_cacache
A /root/.npm/_cacache/content-v2
A /root/.npm/_cacache/content-v2/sha512
A /root/.npm/_cacache/content-v2/sha512/eb
A /root/.npm/_cacache/content-v2/sha512/eb/8d
A /root/.npm/_cacache/content-v2/sha512/eb/8d/75c8d2d874c09ea42b0dc0cca972c460d74f1913183dc199c436110fcdd69bb03cb992e5c99140c6f1628f7bcbdaf6cc3412985de0294cd9d                                                                                                                                                              9c8fc5749c6
A /root/.npm/_cacache/index-v5
A /root/.npm/_cacache/index-v5/46
A /root/.npm/_cacache/index-v5/46/e0
A /root/.npm/_cacache/index-v5/46/e0/f061ea30b750e78f6672057dcdbe3e1e78ac7b48c4593ab70b28b49f02bd
A /root/.npm/_cacache/tmp
C /root/.npm/_logs
A /root/.npm/_logs/2023-09-01T01_05_13_158Z-debug-0.log
root@muse:~# docker images
REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
test         latest    d25ee43030e2   8 minutes ago   1.13GB
root@muse:~# docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED         STATUS         PORTS     NAMES
bbf601cc6ca7   d25ee43030e2   "docker-entrypoint.s…"   7 minutes ago   Up 7 minutes             muse

[codetheweb]

The comment here has a image tag that you can pull directly to get the PR code instead of building it yourself. But yes, the new output looks correct (no more esbuild caching).

I'll try to reproduce this locally.

[codetheweb]

I left Muse running for about 10m. I observed overlay2 increasing by about 100MB at the begining but otherwise didn't grow, so having a hard time reproducing this.

When you're observing the behavior, could you go into the container with docker exec -it <container-id> /bin/bash, run du -h /tmp, and post the output?

[coryjreid]

So we've been playing music with the bot for several hours a day every day and the overlay2 file has now grown to over 8GB. While streaming music the behavior continues. Output sample:

root@muse:~# du -cs /var/lib/docker/*
3188    /var/lib/docker/buildkit
40      /var/lib/docker/containers
4       /var/lib/docker/engine-id
2004    /var/lib/docker/image
52      /var/lib/docker/network
8686732 /var/lib/docker/overlay2
16      /var/lib/docker/plugins
4       /var/lib/docker/runtimes
4       /var/lib/docker/swarm
4       /var/lib/docker/tmp
28      /var/lib/docker/volumes
8692076 total
root@muse:~# du -cs /var/lib/docker/*
3188    /var/lib/docker/buildkit
40      /var/lib/docker/containers
4       /var/lib/docker/engine-id
2004    /var/lib/docker/image
52      /var/lib/docker/network
8688028 /var/lib/docker/overlay2
16      /var/lib/docker/plugins
4       /var/lib/docker/runtimes
4       /var/lib/docker/swarm
4       /var/lib/docker/tmp
28      /var/lib/docker/volumes
8693372 total

The output you requested is as follows:

root@muse:/tmp# du -h /tmp
12K     /tmp/yarn--1693530312744-0.025729490385082876
12K     /tmp/yarn--1693530219317-0.01351705037614681
2.2M    /tmp/v8-compile-cache-0/10.2.154.13-node.9
2.2M    /tmp/v8-compile-cache-0
2.7G    /tmp

Running without -h:

root@muse:/tmp# du /tmp
12      /tmp/yarn--1693530312744-0.025729490385082876
12      /tmp/yarn--1693530219317-0.01351705037614681
2160    /tmp/v8-compile-cache-0/10.2.154.13-node.9
2164    /tmp/v8-compile-cache-0
2749288 /tmp
root@muse:/tmp# du /tmp
12      /tmp/yarn--1693530312744-0.025729490385082876
12      /tmp/yarn--1693530219317-0.01351705037614681
2160    /tmp/v8-compile-cache-0/10.2.154.13-node.9
2164    /tmp/v8-compile-cache-0
2749364 /tmp

I notice /tmp is still growing. I'm going to investigate further and will post another update.

[coryjreid]

I ran a couple du /tmp/* commands sequentially (while the bot is actively streaming) and saved the output. Then I later diffed the output.

root@muse:/tmp# du /tmp/* > output1
root@muse:/tmp# du /tmp/* > output2
root@muse:/tmp# diff output1  output2
39c39
< 10764 /tmp/capacitor-e54e7e9b8489f3f714a47bf8ea19e8a2.tmp
---
> 10860 /tmp/capacitor-e54e7e9b8489f3f714a47bf8ea19e8a2.tmp
44a45
> 4     /tmp/output1

[coryjreid]

And the command which launched the process which is generating the data:

/usr/local/bin/node --require /usr/app/node_modules/tsx/dist/suppress-warnings.cjs --loader file:///usr/app/node_modules/tsx/dist/loader.js src/scripts/migrate-and-start.ts

I captured this by installing procps and running ps -p <pid> -o command.

[codetheweb]

root@muse:/tmp# du -h /tmp
12K     /tmp/yarn--1693530312744-0.025729490385082876
12K     /tmp/yarn--1693530219317-0.01351705037614681
2.2M    /tmp/v8-compile-cache-0/10.2.154.13-node.9
2.2M    /tmp/v8-compile-cache-0
2.7G    /tmp

This is strange, du reports /tmp is 2.7 GB but there's no file/directory that big? 🤔

root@muse:/tmp# du /tmp/* > output1
root@muse:/tmp# du /tmp/* > output2
root@muse:/tmp# diff output1  output2
39c39
< 10764 /tmp/capacitor-e54e7e9b8489f3f714a47bf8ea19e8a2.tmp
---
> 10860 /tmp/capacitor-e54e7e9b8489f3f714a47bf8ea19e8a2.tmp
44a45
> 4     /tmp/output1

Was this when you were observing the 2.7 GB /tmp directory or had you restarted Muse between runs? In theory the temp file for the capacitor package shouldn't infinitely grow, but here it's only 11 MB.