intermittently getting "Cannot connect to the Docker daemon at unix:///var/run/docker.sock"

[AurimasNav]

Hi, every few runs or so we are seeing this error:

ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Rerunning the jobs is usually successful.

I've also seen some people reporting same problem here - #2999, but the original problem was different and question is marked as answered.

values.yaml for our arc runner set helm installation:

githubConfigUrl: https://github.com/<org>
controllerServiceAccount:
  namespace: arc
  name: arc-gha-rs-controller
githubConfigSecret: arc-runner-set
maxRunners: 2
minRunners: 1
runnerGroup: "default"
runnerScaleSetName: "custom"
containerMode:
  type: dind
template:
  spec:
    hostNetwork: true
    containers:
    - name: runner
      image: some.azurecr.io/custom-actions-runner:latest
      command: ["/home/runner/run.sh"]
    imagePullSecrets:
    - name: acr-connectivity-pull
image:
  actionsRunnerImagePullSecrets:
  - name: acr-connectivity-pull

the Dockerfile file for custom-actions-runner image:

FROM ghcr.io/actions/actions-runner:latest

USER root

RUN echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf.d/00-docker
RUN echo 'APT::Install-Recommends "0";' >> /etc/apt/apt.conf.d/00-docker

RUN <<EOF
DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y -qq wget
EOF

RUN <<EOF
. /etc/os-release
wget -q https://packages.microsoft.com/config/ubuntu/$VERSION_ID/packages-microsoft-prod.deb
dpkg -i packages-microsoft-prod.deb
rm packages-microsoft-prod.deb
DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y -qq powershell
EOF

RUN <<EOF
wget https://github.com/GitTools/GitVersion/releases/download/5.12.0/gitversion-linux-x64-5.12.0.tar.gz
tar -xvf gitversion-linux-x64-5.12.0.tar.gz
rm gitversion-linux-x64-5.12.0.tar.gz
mv gitversion /usr/local/bin
EOF

RUN <<EOF
#allow other users to execute tools from /usr/local/bin
chmod -R o+rx /usr/local/bin
EOF

RUN wget -O - https://aka.ms/InstallAzureCLIDeb | bash

RUN <<EOF
wget --secure-protocol=TLSv1_2 --https-only https://get.opentofu.org/install-opentofu.sh -O install-opentofu.sh
chmod +x install-opentofu.sh
./install-opentofu.sh --install-method deb
rm install-opentofu.sh
EOF

WORKDIR /opt/kustomize

RUN <<EOF
wget -O - "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash
ln -s /opt/kustomize/kustomize /usr/local/bin/kustomize
EOF

WORKDIR /opt/argocd

RUN <<EOF
wget https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64 -O argocd-linux-amd64
install -m 555 argocd-linux-amd64 argocd
rm argocd-linux-amd64
ln -s /opt/argocd/argocd /usr/local/bin/argocd
EOF

RUN <<EOF
DEBIAN_FRONTEND=noninteractive
apt-get install ca-certificates -y -qq
update-ca-certificates
echo "deb [trusted=yes] https://apt.fury.io/bearer/ /" | tee -a /etc/apt/sources.list.d/fury.list
apt-get update
apt-get install bearer
EOF

WORKDIR /
USER runner

[AurimasNav]

Created an issue #3794