Docker socket error with agent-k8s

[briananstett]

I'm trying to use the new K8s agent for my Scalr agent pool but have not been successful. The daemonset pods are crashlooping with the following error.

{"logger_name": "tacoagent.cmd", "version": "0.1.33", "backend": "docker", "event": "Starting Scalr agent", "level": "info", "timestamp": "2023-06-09 15:03:53.916851"}
{"logger_name": "tacoagent.cmd", "event": "Cannot connect to the docker daemon. Error while fetching server API version: Cannot connect to the Docker API at unix:///var/run/docker.sock. Is the scalr-docker service running?", "level": "error", "timestamp": "2023-06-09 15:03:53.921063"}
{"logger_name": "tacoagent.worker.worker", "event": "Cannot connect to the Docker daemon (unix:///var/run/docker.sock). Please check that the Docker daemon is running.", "level": "critical", "timestamp": "2023-06-09 15:03:53.921502"}

I'm not sure if it's intended but I noticed the k8s agent daemonset doesn't utilize any type of dind configuration like the docker-agent does. From my understanding, the K8s agent shouldn't be trying to utilze docker but rather schedule new pods for Sclar runs.

Kubernetes Provider: EKS
Kubernetes version: 1.26 (v1.26.4-eks-0a21954)

Steps to reproduce

helm repo add scalr-agent-helm https://scalr.github.io/agent-helm/
helm repo update

helm install scalr-agent-pool scalr-agent-helm/agent-k8s --set agent.url=<> --set agent.token=<>

[jbilliau-rcd]

This is happening to us as well, on EKS 1.24. Is it maybe because 1.24+ is containerD, and docker(moby) is deprecated and removed? If the agent is expecting docker, that isn't gonna work.

[maratkomarov]

Hey @briananstett and @jbilliau-rcd

Thank you for being willing to be early adopters.

The Helm chart still goes with the older appVersion: 0.1.33. We'll address this issue.

Currently, please enforce --set image.tag="0.1.35" in the helm install ... command, and you'll get the latest version, supporting terraform runs in Kubernetes.

Cheers.

[jbilliau-rcd]

awesome, I'll give that a shot, thanks @maratkomarov . Also, just curious; why the decision to go with a Daemonset over a Deployment in agent-k8s vs agent-docker? Why would this be something we run one pod per node? Doesn't that mean we need an entire node per job, instead of a Deployment where I can spin up 10 pods on a single node?

[maratkomarov]

@jbilliau-rcd I think most of your questions can answer this deployment diagram https://github.com/Scalr/agent-helm/tree/master/charts/agent-k8s#agent-k8s

A single Node can host many jobs. How many jobs Kubernetes would schedule on a single Node is driven by Terraform Run pod's resource requests and Node's allocatable resources. Terraform runs are just regular Pods, and you can change their resources/limits via the Chart's values.yaml

We went with a DaemonSet primarily because of the need to manage Pod's local mounts, namely a per-Node terraform plugin cache of public providers downloaded from registry.terraform.io. Cause the end goal is to start jobs as quickly as possible.

[jbilliau-rcd]

thanks for the response @maratkomarov. The diagram just shows, to me anyway, that a Daemonset spins up other pods for the actual Terraform builds and applys, and that makes sense. However, a Deployment could do the exact same thing. I just find Daemonset odd becuase all of our other Daemonsets are truly things you need per node; aws-node (cni for allocating IP's), splunk forwarder (for collecting logs of every container via stdout), Dynatrace one-agent (for injecting all pods on that node), etc., you get the picture.

Nothing about this scalr-agent is node-specific. If I look at at a very similiar thing we are also POC'ing, github action self-hosted runners in k8s, they are Deployments. Problem with Daemonsets is, because they are PER-NODE, say I have 30-40 nodes....I now have 30-40 scalr runners when I might need 1-5 tops?

Now, I do see your point of wanting local mounts for the sake of caching TF providers, though you could just achieve that 1.) with a emptydir volume on a Deployment if the user doesn't care about that or 2.) if they do, use a StatefulSet with an underlying volume, for example, EBS. We use that with Prometheus for similar reasons.

Having said all of that, I would normally just use the the agent-docker helm chart since it works the way I want it to, but then your agent-k8s readme says "It has many advantages over the agent-docker chart and would eventually replace it.". What are these advantages? How soon will agent-docker be deprecated?

[graemechristie]

This does seem to be an odd choice of architecture. Typically if you wanted some kind of shared file caching across pod instances you would use a statefulset .. if you particulary want to use the node's filesystem for caching/sharing those files you would use local persistent volumes but I imagine allowing any generic PVC volume to be used would be preferable, as you could then share this data across all your workers, not just the ones running on the same node .. and leaning too heavily on a nodes local storage can be problematic with large volumes of files ..). Other similar agent/runners implementations I have seen (the Github actions runner controller, mend/whitesource for e.g.) would typically stand up controller pods in a deployment (synonymous with your agent running on the node currently) .. and then those would instantiate StatefulSets (somewhat like an operator pattern).

I have not jumped in and deployed this yet, so I'm not sure what exactly might not be tenable in our environment .. but I am a bit reluctant to kick that off as this architecture does seem to be bit out of line with the common use of daemonsets .. using them to address challenges for which statefulsets and PVC's were developed.

Don't get me wrong though - I would be thrilled to see some kind of a controller -> worker architecture where the run workloads are run in pods scheduled natively in kubernetes .. curently we run our scalr agents in kubernetes using Sysbox to run the executing pod in a userspace docker container (essentially a docker-in-docker scenario ..but run sanely using userspace isolation) .. but that is not ideal.

[graemechristie]

Nothing about this scalr-agent is node-specific. If I look at at a very similiar thing we are also POC'ing, github action self-hosted runners in k8s, they are Deployments. Problem with Daemonsets is, because they are PER-NODE, say I have 30-40 nodes....I now have 30-40 scalr runners when I might need 1-5 tops?

Presumably you would run a separate node pool just for scalr agents, and only attach the Daemonset to those nodes. This does still seem to be a non-typical usage of the damonset architecture though

[jbilliau-rcd]

Yeah that sounds like a giant pain...we have over 200 clusters, I'd have to go tell our devops teams to go create all new managed node groups (we on EKS), configure scaling to 0 when not in use, JUST to run a Daemonset that could easily just be a Deployment and avoid all of that. Which is exactly what the agent-docker helm chart is!

Hopefully it stays around for a while, because as it stands, I am choosing to use that over the agent-k8s one due to this decision. If the deployment of a runner requires me to think about, stand up, maintain, and configure nodes JUST to run the runner....it's dead in the water.

[maratkomarov]

Thank you, everyone, for your feedback. We greatly appreciate it, and based on your input, we are currently working on a few minor improvements. But before I go there, a few words about the runner design.

One of the main objectives in developing the Kubernetes native runner was to leverage the advantages of the Kubernetes scheduler for workload placement. When combined with Autoscaler, it provides the most efficient utilization of compute resources. In contrast, the previous iteration of the Runner on Docker lacked resource-aware scheduling capabilities.

While we have been moving the runner towards Kubernetes, we still maintain support for Docker within a single codebase, albeit with some trade-offs. One of these trade-offs is the usage of host mounts, which, as you've noticed, goes against Kubernetes patterns.

Yes, we have decided to use a DaemonSet for workers instead of an agent sidecar for every terraform run. This approach allows us to achieve a lower resource footprint per Node, which is especially important considering that the Agent is written in Python.

With all that being said, I would like to share the list of planned improvements:

• First, we will update the Helm chart to separate the task consumer and worker components. The task consumer will run in a Deployment, while the worker will run in a DaemonSet.

• Secondly, we will expose both components' 'nodeSelector' settings in the values.yaml file. This will effectively enable you to run the worker in a separate node pool, which can initially have 0 nodes. If having a dedicated node pool is not feasible, you can label some of the nodes in the default node pool and place the workers there.

We hope that these planned improvements will address the concerns raised and provide a more optimized and flexible solution. Thank you again for your feedback, and please feel free to reach out if you have any further questions or suggestions.

[jbilliau-rcd]

hey @maratkomarov , thanks for your response! Question, what is the difference between the "task consumer" and "worker" in your first bullet? Is the consumer (daemonset) the pod that will accept/process the job, but then launch another pod, the worker (deployment), to actually run it? In which case, would the expectation be the consumer would never need more than 1 pod per node, since it's just a "manager" of sorts?

That is better....though still not ideal. I don't need more than 1 manager, especially not per node.

That being said, if you still plan on maintaining agent-docker, I won't bug you too much....if my use-case doesn't fit with your vision/future of the agent-k8s, I'll just use the agent-docker and stop my complaining :)

[maratkomarov]

@jbilliau-rcd Consumer (deployment) pulls a job from Scalr, spawns a Pod to execute a terraform workload, and then worker (daemonset) communicates with the Pod to get the terraform work done. You only need 1 consumer per agent pool, the rest are workers.

[rtfee]

@graemechristie @briananstett @jbilliau-rcd please see the improvements in the latest release: https://updates.scalr.io/release/ZXac5-scalr-agent-august-release-notes

[c4po]

if we don't have a dedicated node pool for sclar agent, the "worker" pod will be run on all nodes in my cluster. And we will run out of agent seat.
is it possible only register the controller to agent pool and not count the workers?