docker builder failed to call diskusage: DeadlineExceeded: context deadline exceeded

[savemetenminutes]

Description

Recently my docker builds started occurring what appears to be disk space issues. The first time I encountered this issue I went into the Docker Desktop UI -> Builds -> Builder Settings and for the currently active builder called desktop-linux I saw a image-like message saying "Storage error" and "docker builder failed to call diskusage: DeadlineExceeded: context deadline exceeded" and "20GB storage limit". The build of one of the services kept failing with "4.746 : You don't have enough free space in /var/cache/apt/archives/.". Since there was an update I went ahead and installed it and now I get some sort of pie chart diagram in the desktop-linux builder info, which again says "Storage limit 20GB", but also "Used 121.9GB", "Shared 117.3GB" with the pie chart reading "59.3MB Local sources" and "121.8GB Regular". Since I'm using the WSL2 integration for Docker Desktop I went into the Debian WSL console and ran df -h, which reports the following:

Filesystem      Size  Used Avail Use% Mounted on
none             14G  4.0K   14G   1% /mnt/wsl
none            931G  595G  336G  64% /usr/lib/wsl/drivers
none             14G     0   14G   0% /usr/lib/wsl/lib
/dev/sdc        2.9T  1.7T  1.2T  60% /
none             14G   76K   14G   1% /mnt/wslg
rootfs           14G  1.9M   14G   1% /init
none             14G     0   14G   0% /dev
none             14G  4.0K   14G   1% /run
none             14G     0   14G   0% /run/lock
none             14G     0   14G   0% /run/shm
none             14G     0   14G   0% /run/user
tmpfs            14G     0   14G   0% /sys/fs/cgroup
none             14G   76K   14G   1% /mnt/wslg/versions.txt
none             14G   76K   14G   1% /mnt/wslg/doc
drvfs           931G  595G  336G  64% /mnt/c
drvfs           7.3T  1.2T  6.1T  17% /mnt/d
drvfs           931G  235G  696G  26% /mnt/e
drvfs           3.7T  2.4T  1.4T  64% /mnt/f
/dev/sde        251G  238G  177M 100% /mnt/wsl/docker-desktop-data/isocache
drvfs           931G  595G  336G  64% /Docker/host
none             14G  8.0K   14G   1% /mnt/wsl/docker-desktop/shared-sockets/host-services
/dev/sdf       1007G   57M  956G   1% /mnt/wsl/docker-desktop/docker-desktop-user-distro
/dev/loop0      463M  463M     0 100% /mnt/wsl/docker-desktop/cli-tools

I know I have plenty of space on the WSL2 vhdx as I have a 4TB dedicated drive reserved specifically for the WSL virtual disk with 1.32TB of free space left and I see no reason to limit any disk space for purposes of Docker cache, images and volume storage.

I also tried the self-diagnose tool as explained here:
https://docs.docker.com/desktop/troubleshoot/overview/#self-diagnose-tool
In powershell after running
& "C:\Program Files\Docker\Docker\resources\com.docker.diagnose.exe" check
the output is

Starting diagnostics

com.docker.diagnose.exe : [2024-09-11T09:40:33.688851700Z][com.docker.diagnose.exe.ipc] c1a5278a-diagnose -> \\.\pipe\dockerBackendApiServer 
BackendAPI
At line:1 char:1
+ & "C:\Program Files\Docker\Docker\resources\com.docker.diagnose.exe"  ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: ([2024-09-11T09:...rver BackendAPI:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
[2024-09-11T09:40:33.699823400Z][com.docker.diagnose.exe.ipc] (21bd8fee) c1a5278a-diagnose C->S BackendAPI POST /idle/make-busy
[2024-09-11T09:40:33.701460000Z][com.docker.diagnose.exe.ipc] (21bd8fee) c1a5278a-diagnose C<-S dc927013-BackendAPI POST /idle/make-busy 
(1.6366ms): 0xc0006246f0
[2024-09-11T09:40:34.708119700Z][com.docker.diagnose.exe.ipc] (21bd8fee-1) c1a5278a-diagnose C->S BackendAPI GET /idle
[2024-09-11T09:40:34.711749900Z][com.docker.diagnose.exe.ipc] (21bd8fee-1) c1a5278a-diagnose C<-S dc927013-BackendAPI GET /idle (3.4234ms): 
{"apisInFlight":{"/v1.46/images/get from Docker-Client/unknown-version (windows)":2},"booted":true,"busyReason":["2 calls to /v1.46/images/get 
from Docker-Client/unknown-version (windows)","timed activities: 
map[/idle/make-busy:28991669000]"],"timedActivities":{"/idle/make-busy":"float64"},"vmRunning":true}
[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[SKIP] DD0002: does the bootloader have virtualization enabled?
[SKIP] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[SKIP] DD0022: is the Virtual Machine Platform Windows Feature enabled?
[SKIP] DD0021: is the WSL 2 Windows Feature enabled?
[PASS] DD0024: is WSL installed?
[PASS] DD0025: are WSL distros installed?
[PASS] DD0026: is the WSL LxssManager service running?
[PASS] DD0029: is the WSL 2 Linux filesystem corrupt?
[PASS] DD0035: is the VM time synchronized?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0031: does the Docker API work?
[PASS] DD0013: is the $PATH ok?
[PASS] DD0003: is the Docker CLI working?
[PASS] DD0005: is the user in the docker-users group?
[PASS] DD0038: is the connection to Docker working?
[PASS] DD0014: are the backend processes running?
[PASS] DD0007: is the backend responding?
[SKIP] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[SKIP] DD0006: is the Docker Desktop Service responding?
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0033: does the host have Internet access?
[PASS] DD0002: does the bootloader have virtualization enabled?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0022: is the Virtual Machine Platform Windows Feature enabled?
[PASS] DD0021: is the WSL 2 Windows Feature enabled?
[PASS] DD0024: is WSL installed?
[PASS] DD0025: are WSL distros installed?
[PASS] DD0026: is the WSL LxssManager service running?
[PASS] DD0029: is the WSL 2 Linux filesystem corrupt?
[PASS] DD0035: is the VM time synchronized?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0031: does the Docker API work?
[PASS] DD0032: do Docker networks overlap with host IPs?
No fatal errors detected.

Reproduce

My docker compose stack is quite big (100+ services) and over time there are updates to many of the images and apt packages within the images and it needs to be rebuilt.

1. docker compose build

Expected behavior

docker compose build should succeed

docker version

Client:
 Version:           20.10.5+dfsg1
 API version:       1.41
 Go version:        go1.15.15
 Git commit:        55c4c88
 Built:             Mon May 30 18:34:49 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Desktop  ()
 Engine:
  Version:          27.2.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       3ab5c7d
  Built:            Tue Aug 27 14:15:15 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.20
  GitCommit:        8fc6bcff51318944179630522a095cc9dbf9f353
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.16.2-desktop.1)
  compose: Docker Compose (Docker Inc., v2.29.2-desktop.2)
  debug: Get a shell into any image or container (Docker Inc., 0.0.34)
  desktop: Docker Desktop commands (Alpha) (Docker Inc., v0.0.15)
  dev: Docker Dev Environments (Docker Inc., v0.1.2)
  extension: Manages Docker extensions (Docker Inc., v0.2.25)
  feedback: Provide feedback, right in your terminal! (Docker Inc., v1.0.5)
  init: Creates Docker-related starter files for your project (Docker Inc., v1.3.0)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scout: Docker Scout (Docker Inc., v1.13.0)
WARNING: Plugin "/usr/local/lib/docker/cli-plugins/docker-scan" is not valid: failed to fetch metadata: fork/exec /usr/local/lib/docker/cli-plugins/docker-scan: no such file or directory

Server:
 Containers: 52
  Running: 0
  Paused: 0
  Stopped: 52
 Images: 1592
 Server Version: 27.2.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
  WARNING: You're not using the default seccomp profile
   Profile: unconfined
 Kernel Version: 5.15.90.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 27.41GiB
 Name: docker-desktop
 ID: 36c7edfc-a3e7-4b9f-9304-729c5228b84a
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
  com.docker.desktop.address=unix:///var/run/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Diagnostics ID

432BCC77-B0B5-4D17-BCC8-CEA9C3669229/20240911093329

Additional Info

No response

[crazy-max]

Thanks for your report, can you take a screenshot of desktop-linux builder in Builders settings like:

Fyi you can increase storage limit for this builder by updating the Daemon configuration in "Docker Engine" settings:

Can you also post the docker compose build logs output and output of command docker buildx du?

[savemetenminutes]

IMPORTANT NOTE: This screenshot was taken after I executed docker system prune --volumes so it will definitely show different info from the time of the occurrence of the issue. After running the abovementioned command the build completed successfully.

Thanks for pointing out the engine configuration, which increases the available builder storage space. I will definitely use it to mitigate any potential issues in the future. At this point I'm more interested in assisting with diagnosing and fixing what appears to be a recently introduced issue with unreliable handling of available storage - something that has never occurred to me in the past. Hopefully we can get to the bottom of why this "gc" (garbage collector?) needs such setting and is not able to clear unused storage data automatically in order to free up space for current build operations. Is the storage being used up for container images, some sort of cache or something else entirely? If it is related to images I would assume that there was no hard limit on how much space the Docker service could use in the past as I've been using this large compose stack for 4 years already.