-
Notifications
You must be signed in to change notification settings - Fork 1
Development Setup
tl;dr: Execute the hack/bootstrap-demo-env.sh script in the repository, which automates the steps below 🥳
Create a new Colima instance, with the qemu virtualization backend (note: the disk will automatically grow if needed):
$ brew install colima docker
$ colima start --cpu 4 --memory 12 --disk 20 --network-address --vm-type qemuOptionally, you can append a name for the new instance at the end of the command line.
In order to tear down an existing Colima instance (optionally passing the name of the instance):
$ colima deleteInstall minikube:
$ brew install minikubeCreate a cluster:
$ minikube start --driver=docker --cpus=max --memory=max # tweak resource limits as desiredTest your setup by running a simple pod:
$ kubectl run --attach -it --rm --restart=Never --image hello-world test
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(arm64v8)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
pod "test" deletedIf you want to build custom Docker images to run in the Kubernetes cluster, you should set the environment variables to connect your host to the Docker Engine running inside the Minikube VM (so you don't need to sideload images from the host using minikube load):
$ eval $(minikube docker-env)Minikube comes with an addon that runs a local instance of the Docker Registry. Before you can use it, you need to enable it once:
$ minikube addons enable registryIf you want to be able to have stable image names prefixed with localhost:5000/, you can run a small proxy container that forwards requests to port 5000 on your machine to the registry container (this needs to be started whenever needed, the container won't stay around):
$ docker run --rm -it --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:$(minikube ip):5000"Install the manifests for Kueue:
VERSION=v0.8.1
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/$VERSION/manifests.yamlSet up a single cluster queue, local queue, and a few default priority classes for quickstart (might have to wait a few moments after the previous command to allow all Kueue components to start running):
$ cat single-clusterqueue-setup.yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
name: "default-flavor"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
name: "cluster-queue"
spec:
namespaceSelector: {} # match all.
# Allow preemption of workloads by higher priority ones
preemption:
reclaimWithinCohort: Any
borrowWithinCohort:
policy: LowerPriority
maxPriorityThreshold: 100
withinClusterQueue: LowerPriority
resourceGroups:
- coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
flavors:
- name: "default-flavor"
resources:
- name: "cpu"
nominalQuota: 4
- name: "memory"
nominalQuota: 6Gi
- name: "nvidia.com/gpu"
nominalQuota: 1
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
namespace: "default"
name: "user-queue"
spec:
clusterQueue: "cluster-queue"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: WorkloadPriorityClass
metadata:
name: background
value: 1
description: "Background (=lowest) priority"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: WorkloadPriorityClass
metadata:
name: development
value: 100
description: "Development priority"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: WorkloadPriorityClass
metadata:
name: production
value: 1000
description: "Production priority"
$ kubectl create -f single-clusterqueue-setup.yaml
resourceflavor.kueue.x-k8s.io/default-flavor created
clusterqueue.kueue.x-k8s.io/cluster-queue created
localqueue.kueue.x-k8s.io/user-queue created
workloadpriorityclass.kueue.x-k8s.io/background created
workloadpriorityclass.kueue.x-k8s.io/development created
workloadpriorityclass.kueue.x-k8s.io/production createdYou can check the resources have been created correctly:
$ kubectl get clusterqueue
NAME COHORT PENDING WORKLOADS
cluster-queue 0
$ kubectl get -A localqueue
NAMESPACE NAME CLUSTERQUEUE PENDING WORKLOADS ADMITTED WORKLOADS
default user-queue cluster-queue 0 0
$ kubectl get resourceflavor
NAME AGE
default-flavor 4m53s
$ kubectl get workloadpriorityclasses
NAME VALUE
background 1
development 100
production 1000We can now submit a Kubernetes Job to Kueue for execution:
$ kubectl create -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
generateName: sample-job-
namespace: default
labels:
kueue.x-k8s.io/queue-name: user-queue
spec:
parallelism: 1
completions: 3
suspend: true
template:
spec:
containers:
- name: dummy-job
image: alpine:latest
args: ["sleep", "30"]
resources:
requests:
cpu: 1
memory: "200Mi"
restartPolicy: Never
EOF
job.batch/sample-job-q442s createdMonitor its execution in a local queue:
$ kubectl get localqueue
NAME CLUSTERQUEUE PENDING WORKLOADS ADMITTED WORKLOADS
user-queue cluster-queue 0 1
$ kubectl get workload
NAME QUEUE ADMITTED BY AGE
job-sample-job-q442s-bcaa7 user-queue cluster-queue 55s
$ kubectl describe workload
Name: job-sample-job-q442s-bcaa7
Namespace: default
Labels: kueue.x-k8s.io/job-uid=9942277f-b504-4100-9f9e-a19147b1d6a8
Annotations: <none>
API Version: kueue.x-k8s.io/v1beta1
Kind: Workload
Metadata:
Creation Timestamp: 2024-04-04T09:34:16Z
Finalizers:
kueue.x-k8s.io/resource-in-use
Generation: 1
Owner References:
API Version: batch/v1
Block Owner Deletion: true
Controller: true
Kind: Job
Name: sample-job-q442s
UID: 9942277f-b504-4100-9f9e-a19147b1d6a8
Resource Version: 1968
UID: 47099990-5e59-4199-9ce2-c2d92de534ab
Spec:
Active: true
Pod Sets:
Count: 1
Name: main
Template:
Metadata:
Spec:
Containers:
Args:
sleep
30
Image: alpine:latest
Image Pull Policy: Always
Name: dummy-job
Resources:
Requests:
Cpu: 1
Memory: 200Mi
Termination Message Path: /dev/termination-log
Termination Message Policy: File
Dns Policy: ClusterFirst
Restart Policy: Never
Scheduler Name: default-scheduler
Security Context:
Termination Grace Period Seconds: 30
Priority: 0
Priority Class Source:
Queue Name: user-queue
Status:
Admission:
Cluster Queue: cluster-queue
Pod Set Assignments:
Count: 1
Flavors:
Cpu: default-flavor
Memory: default-flavor
Name: main
Resource Usage:
Cpu: 1
Memory: 200Mi
Conditions:
Last Transition Time: 2024-04-04T09:34:16Z
Message: Quota reserved in ClusterQueue cluster-queue
Reason: QuotaReserved
Status: True
Type: QuotaReserved
Last Transition Time: 2024-04-04T09:34:16Z
Message: The workload is admitted
Reason: Admitted
Status: True
Type: Admitted
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal QuotaReserved 89s kueue-admission Quota reserved in ClusterQueue cluster-queue, wait time since queued was 0s
Normal Admitted 89s kueue-admission Admitted by ClusterQueue cluster-queue, wait time since reservation was 0sSeeing that the job has been admitted to the queue, we can see the pods that actually run the workload:
$ kubectl get jobs
NAME COMPLETIONS DURATION AGE
sample-job-q442s 3/3 104s 2m10s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
sample-job-q442s-6mcv8 0/1 Completed 0 48s
sample-job-q442s-dqvxw 0/1 Completed 0 83s
sample-job-q442s-gkmbs 0/1 Completed 0 118sIn order to submit Ray compute jobs, the Kuberay operator needs to be installed in the cluster first.
$ brew install helm # if necessary
$ helm repo add kuberay https://ray-project.github.io/kuberay-helm/
$ helm repo update
$ helm install kuberay-operator kuberay/kuberay-operator