Deleted builds still visible in build list

[jeremiah-snee-openx]

Description

Deleted builds still visible in build list. Links take you to error saying file deleted

Reproduce

1. configure docker build cloud
2. use desktop-linux builder
3. delete build entry from ui

Expected behavior

deleted entries should be removed

docker version

Client:
 Version:           27.4.0
 API version:       1.47
 Go version:        go1.22.10
 Git commit:        bde2b89
 Built:             Sat Dec  7 10:35:43 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.37.1 (178610)
 Engine:
  Version:          27.4.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       92a8393
  Built:            Sat Dec  7 10:38:33 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.21
  GitCommit:        472731909fa34bd7bc9c087e4c27943f9835f111
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    27.4.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.5.1
    Path:     /Users/local.user/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.2-desktop.1
    Path:     /Users/local.user/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.31.0-desktop.2
    Path:     /Users/local.user/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     /Users/local.user/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Beta) (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/local.user/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/local.user/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/local.user/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/local.user/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/local.user/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/local.user/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.1
    Path:     /Users/local.user/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 27.4.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 7.751GiB
 Name: docker-desktop
 ID: a7579eab-2a1a-4fa2-b80c-01654c467fa0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/local.user/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

B4613923-37A4-4A0D-B817-686C29DE1B7A/20250103205341

Additional Info

No response

[crazy-max]

Thanks for raising this issue, I'm taking a look.

[crazy-max]

@jeremiah-snee-openx What is the record ID for which this issue occurs? Looking at your logs it seems to be oa31nlsosezye22qnt4a6ozv3:

2025-01-03 20:53:32 [W] error streaming build desktop-linux/desktop-linux/oa31nlsosezye22qnt4a6ozv3: rpc error: code = Unknown desc = ref oa31nlsosezye22qnt4a6ozv3 is deleted: file does not exist

Seems the ref is effectively deleted but is still part of the events list. I think the record remains on moby side while it doesn't have any blobs attached but this has been fixed by moby/buildkit#5378.

Can you provide a new diagnostics ID please? I would like to check if somehow the record got GCed later on.

Can you also show the output of docker buildx ls?

[jeremiah-snee-openx]

I don't remember seeing this issue before I configured the docker build cloud

Generating new diagnostic ID.
B4613923-37A4-4A0D-B817-686C29DE1B7A/20250106161618

The build entry is still visible in the ui.

[crazy-max]

Would you mind making a screenshot of the build view with the incriminated build please?

[jeremiah-snee-openx]

All builds with default builder have been deleted.

Seems like original build finally disappeared so regenerating diagnostic ID.

B4613923-37A4-4A0D-B817-686C29DE1B7A/20250106205628

[crazy-max]

Ok I see now that the record is attached to a Docker Build cloud instance so maybe the blobs attached to this record have been removed. @fiam @silvin-lubecki @sgarcez Any idea?

[fiam]

There's a known issue in cloud that could be related to this, although we're not positive. We will deploy a fix soon and we'll update this thread.