Fleet showing stale data upon destroying ~1000 instanced services

[aaronjwood]

We're seeing a strange issue where we are trying to destroy around 1000 instanced services in a cluster. What's happening is when we try to go and destroy all of the instances there are around 40 that get destroyed but that don't seem to properly register that action in the etcd database. Doing a fleetctl list-units and looking for this service will show you the 40 units that are supposedly still "running" but if you try to destroy them fleet complains that the service doesn't exist.

If we go to the nodes directly and look for the service file under /var/run/fleet/units it's not there. The service is also not really running (which is good and is what we'd want) but fleetctl list-units is what shows the service existing and running.

Is there any way to prevent this from happening or to resolve this issue without restarting fleet on each node that is affected? I believe we're now manually taking it out of etcd using etcdctl to resolve the issue.