Skip to content

Commit

Permalink
add statefulset github runners
Browse files Browse the repository at this point in the history
  • Loading branch information
sergelogvinov committed Dec 7, 2023
1 parent 624fcfd commit 33b69ea
Show file tree
Hide file tree
Showing 12 changed files with 404 additions and 20 deletions.
2 changes: 1 addition & 1 deletion charts/github-actions-runner/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,5 @@ maintainers:
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 1.4.4
version: 1.5.0
appVersion: 2.311.0
20 changes: 14 additions & 6 deletions charts/github-actions-runner/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# github-actions-runner

![Version: 1.4.4](https://img.shields.io/badge/Version-1.4.4-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 2.311.0](https://img.shields.io/badge/AppVersion-2.311.0-informational?style=flat-square)
![Version: 1.5.0](https://img.shields.io/badge/Version-1.5.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 2.311.0](https://img.shields.io/badge/AppVersion-2.311.0-informational?style=flat-square)

Github Actions with container registry and mirrors

Expand Down Expand Up @@ -71,19 +71,23 @@ nodeSelector:
| imagePullSecrets | list | `[]` | |
| nameOverride | string | `""` | |
| fullnameOverride | string | `""` | |
| maxRunners | int | `8` | |
| minRunners | int | `1` | |
| autoscaling.enabled | bool | `false` | |
| autoscaling.minReplicas | int | `1` | |
| autoscaling.maxReplicas | int | `9` | |
| autoscaling.targetUtilizationPercentage | int | `90` | |
| autoscaling.scaleDown.stabilizationWindowSeconds | int | `600` | |
| autoscaling.scaleUp.stabilizationWindowSeconds | int | `30` | |
| runnerGroup | string | `"default"` | |
| runnerScaleSetName | string | `""` | |
| runnerVersion | string | `"0.6.1"` | |
| githubConfigUrl | string | `"https://github.com/..."` | |
| githubConfigSecret | object | `{}` | |
| controllerServiceAccount.name | string | `"arc"` | |
| dind.enabled | bool | `true` | |
| dind.image.repository | string | `"docker"` | |
| dind.image.pullPolicy | string | `"IfNotPresent"` | |
| dind.image.tag | string | `"23.0-dind"` | |
| dind.image | object | `{"pullPolicy":"IfNotPresent","repository":"docker","tag":"24.0-dind"}` | Docker in Docker image. ref: https://hub.docker.com/_/docker/tags?page=1&name=dind |
| dind.resources | object | `{"limits":{"cpu":1,"memory":"1Gi"},"requests":{"cpu":"500m","memory":"256Mi"}}` | Resource requests and limits. ref: https://kubernetes.io/docs/user-guide/compute-resources/ |
| dind.extraVolumeMounts | list | `[]` | Additional container volume mounts. |
| dind.extraVolumes | list | `[]` | Additional volumes. |
| dind.persistence | object | `{"accessModes":["ReadWriteOnce"],"annotations":{},"enabled":false,"size":"100Gi"}` | Persistence parameters for source code ref: https://kubernetes.io/docs/user-guide/persistent-volumes/ |
| mirrors.enabled | bool | `true` | |
| mirrors.image.repository | string | `"ghcr.io/project-zot/zot-linux-amd64"` | |
Expand All @@ -99,6 +103,7 @@ nodeSelector:
| registry.image.repository | string | `"registry"` | |
| registry.image.pullPolicy | string | `"IfNotPresent"` | |
| registry.image.tag | float | `2.8` | |
| registry.ingress | object | `{"annotations":{"nginx.ingress.kubernetes.io/proxy-body-size":0},"className":"","enabled":false,"hosts":[],"tls":[]}` | Registry ingress parameters ref: http://kubernetes.io/docs/user-guide/ingress/ |
| registry.resources | object | `{"limits":{"cpu":1,"memory":"512Mi"},"requests":{"cpu":"100m","memory":"128Mi"}}` | Resource requests and limits. ref: https://kubernetes.io/docs/user-guide/compute-resources/ |
| registry.extraVolumeMounts | list | `[]` | |
| registry.extraVolumes | list | `[]` | |
Expand Down Expand Up @@ -129,8 +134,11 @@ nodeSelector:
| service | object | `{"ipFamilies":["IPv4"]}` | Service parameters ref: https://kubernetes.io/docs/user-guide/services/ |
| resources | object | `{"requests":{"cpu":"200m","memory":"256Mi"}}` | Resource requests and limits. ref: https://kubernetes.io/docs/user-guide/compute-resources/ |
| persistence | object | `{"accessModes":["ReadWriteOnce"],"annotations":{},"enabled":false,"size":"8Gi"}` | Persistence parameters for source code ref: https://kubernetes.io/docs/user-guide/persistent-volumes/ |
| extraVolumeMounts | list | `[]` | Additional container volume mounts. |
| extraVolumes | list | `[]` | Additional volumes. |
| nodeSelector | object | `{}` | Node labels for pod assignment. ref: https://kubernetes.io/docs/user-guide/node-selection/ |
| tolerations | list | `[]` | Tolerations for pod assignment. ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ |
| podAntiAffinityPreset | string | `"soft"` | Anti-affinity for pod assignment. options: soft, hard |
| affinity | object | `{}` | Affinity for pod assignment. ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity |

----------------------------------------------
Expand Down
19 changes: 18 additions & 1 deletion charts/github-actions-runner/templates/NOTES.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,22 @@
{{- if not (.Capabilities.APIVersions.Has "actions.github.com/v1alpha1") }}
1. Do not forget to add this rules to prometheus adapter:
{{- $configUrl := trimSuffix "/" .Values.githubConfigUrl }}
{{- if eq (len (regexSplit "/" $configUrl -1)) 4 }}
- seriesQuery: '{__name__="github_runner_organization_status",app_kubernetes_io_name="{{ include "github-actions-runner.fullname" . }}"}'
{{- else }}
- seriesQuery: '{__name__="github_runner_status",app_kubernetes_io_name="{{ include "github-actions-runner.fullname" . }}"}'
{{- end }}
resources:
overrides:
kubernetes_namespace: {resource: "namespace"}
app_kubernetes_io_instance: {group: "apps", resource: "deployment"}
name:
as: "github_actions_worker_usage_ratio"
metricsQuery: 'sum(<<.Series>>{<<.LabelMatchers>>,busy="true"}) BY (<<.GroupBy>>,repo) / sum(<<.Series>>{}) BY (<<.GroupBy>>,repo) * 100 OR on () topk(1, <<.Series>>{<<.LabelMatchers>>})'

{{- end }}
{{- if .Values.registry.enabled }}
1. Clean registry
2. Clean registry manually

kubectl -n {{ .Release.Namespace }} create job --from=cronjob/{{ include "github-actions-runner.fullname" . }}-registry-cleaner {{ include "github-actions-runner.fullname" . }}-registry-cleaner-manual
{{- end }}
8 changes: 6 additions & 2 deletions charts/github-actions-runner/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,9 @@ Return a soft podAffinity/podAntiAffinity definition
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels: {{- (include "github-actions.selectorLabels" .) | nindent 10 }}
matchLabels:
{{- (include "github-actions-runner.selectorLabels" .) | nindent 10 }}
app.kubernetes.io/component: autoscaling-runner-set
namespaces:
- {{ .Release.Namespace | quote }}
topologyKey: kubernetes.io/hostname
Expand All @@ -87,7 +89,9 @@ Return a hard podAffinity/podAntiAffinity definition
{{- define "affinities.pods.hard" -}}
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels: {{- (include "github-actions.selectorLabels" .) | nindent 8 }}
matchLabels:
{{- (include "github-actions-runner.selectorLabels" .) | nindent 8 }}
app.kubernetes.io/component: autoscaling-runner-set
namespaces:
- {{ .Release.Namespace | quote }}
topologyKey: kubernetes.io/hostname
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
{{- if not (.Capabilities.APIVersions.Has "actions.github.com/v1alpha1") }}
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ include "github-actions-runner.fullname" . }}
namespace: {{ .Release.Namespace }}
labels:
{{- include "github-actions-runner.labels" . | nindent 4 }}
app.kubernetes.io/component: autoscaling-runner-set
spec:
serviceName: {{ include "github-actions-runner.fullname" . }}
{{- if not .Values.autoscaling.enabled }}
replicas: {{ max 1 .Values.minRunners }}
{{- end }}
updateStrategy:
type: RollingUpdate
podManagementPolicy: Parallel
minReadySeconds: 5
selector:
matchLabels:
{{- include "github-actions-runner.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: autoscaling-runner-set
template:
metadata:
annotations:
kubectl.kubernetes.io/default-container: runner
{{- if .Values.mirrors.enabled }}
checksum/config: {{ toJson .Values.mirrors.registry | sha256sum }}
{{- end }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "github-actions-runner.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: autoscaling-runner-set
spec:
enableServiceLinks: false
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "github-actions-runner.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
terminationGracePeriodSeconds: 3600
dnsConfig:
options:
- name: ndots
value: "3"
containers:
{{- if .Values.dind.enabled }}
- name: {{ .Chart.Name }}-dind
securityContext:
privileged: true
seccompProfile:
type: Unconfined
image: "{{ .Values.dind.image.repository }}:{{ .Values.dind.image.tag }}"
imagePullPolicy: {{ .Values.dind.image.pullPolicy }}
command: ["dockerd"]
args:
- --config-file=/etc/docker-config/daemon.json
volumeMounts:
- name: config
mountPath: /etc/docker-config
- name: tls
mountPath: /etc/docker-tls
- name: docker
mountPath: /var/lib/docker
- name: run
mountPath: /run
- name: modules
mountPath: /lib/modules
readOnly: true
{{- if .Values.dind.extraVolumeMounts }}
{{- toYaml .Values.dind.extraVolumeMounts | nindent 12 }}
{{- end }}
resources:
{{- toYaml .Values.dind.resources | nindent 12 }}
{{- end }}
- name: runner
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
command: ["/bin/bash", "/entrypoint.sh"]
env:
- name: ACTIONS_RUNNER_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
value: "false"
- name: RUNNER_WORK_FOLDER
value: /builds
- name: RUNNER_ORGANIZATION_URL
valueFrom:
secretKeyRef:
name: {{ include "github-actions-runner.fullname" . }}-github
key: RUNNER_ORGANIZATION_URL
optional: true
- name: RUNNER_REPOSITORY_URL
valueFrom:
secretKeyRef:
name: {{ include "github-actions-runner.fullname" . }}-github
key: RUNNER_REPOSITORY_URL
optional: true
- name: GITHUB_ACCESS_TOKEN
valueFrom:
secretKeyRef:
name: {{ include "github-actions-runner.fullname" . }}-github
key: GITHUB_ACCESS_TOKEN
envFrom:
- secretRef:
name: "{{ include "github-actions-runner.fullname" . }}"
lifecycle:
postStart:
exec:
command: ["/bin/sh", "/etc/docker-config/mirrors.sh"]
volumeMounts:
- name: config
mountPath: /etc/docker-config
- name: tls
mountPath: /etc/docker-tls
{{- if .Values.extraVolumeMounts }}
{{- toYaml .Values.extraVolumeMounts | nindent 12 }}
{{- end }}
- name: builds
mountPath: /builds
- name: run
mountPath: /run
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
affinity:
podAntiAffinity: {{- include "affinities.pods" . | nindent 10 }}
{{- with .Values.affinity }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
volumes:
- name: run
emptyDir:
medium: Memory
- name: config
configMap:
name: {{ include "github-actions-runner.fullname" . }}
- name: tls
secret:
secretName: {{ include "github-actions-runner.fullname" . }}-client-tls
defaultMode: 288
{{- if .Values.dind.enabled }}
- name: modules
hostPath:
path: /lib/modules
- name: docker
{{- if and .Values.dind.persistence.enabled .Values.dind.persistence.storageClass }}
ephemeral:
volumeClaimTemplate:
metadata:
labels:
{{- include "github-actions-runner.labels" . | nindent 22 }}
{{- with .Values.dind.persistence.annotations }}
annotations:
{{- range $key, $value := . }}
{{ $key }}: {{ $value }}
{{- end }}
{{- end }}
{{- include "volumeClaimTemplate.spec.render" .Values.dind.persistence | nindent 14 }}
{{- else }}
emptyDir: {}
{{- end }}
{{- end }}
- name: builds
{{- if and .Values.persistence.enabled .Values.persistence.storageClass }}
ephemeral:
volumeClaimTemplate:
metadata:
labels:
{{- include "github-actions-runner.labels" . | nindent 22 }}
{{- with .Values.persistence.annotations }}
annotations:
{{- range $key, $value := . }}
{{ $key }}: {{ $value }}
{{- end }}
{{- end }}
{{- include "volumeClaimTemplate.spec.render" .Values.persistence | nindent 14 }}
{{- else }}
emptyDir: {}
{{- end }}
{{- if .Values.extraVolumes }}
{{- toYaml .Values.extraVolumes | nindent 8 }}
{{- end }}
{{- if .Values.dind.extraVolumes }}
{{- toYaml .Values.dind.extraVolumes | nindent 8 }}
{{- end }}
{{- end }}
14 changes: 10 additions & 4 deletions charts/github-actions-runner/templates/autoscalingrunnerset.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ spec:
{{- with .Values.runnerScaleSetName }}
runnerScaleSetName: {{ . }}
{{- end }}
maxRunners: {{ default 2 .Values.maxRunners | int }}
minRunners: {{ default 0 .Values.minRunners | int }}
maxRunners: {{ default 2 .Values.autoscaling.maxReplicas | int }}
minRunners: {{ default 0 .Values.autoscaling.maxReplicas | int }}

template:
metadata:
Expand Down Expand Up @@ -74,6 +74,9 @@ spec:
- name: modules
mountPath: /lib/modules
readOnly: true
{{- if .Values.dind.extraVolumeMounts }}
{{- toYaml .Values.dind.extraVolumeMounts | nindent 12 }}
{{- end }}
resources:
{{- toYaml .Values.dind.resources | nindent 12 }}
{{- end }}
Expand Down Expand Up @@ -105,7 +108,7 @@ spec:
- name: tls
mountPath: /etc/docker-tls
{{- if .Values.extraVolumeMounts }}
{{ toYaml .Values.extraVolumeMounts | nindent 12 }}
{{- toYaml .Values.extraVolumeMounts | nindent 12 }}
{{- end }}
- name: builds
mountPath: /builds
Expand Down Expand Up @@ -177,6 +180,9 @@ spec:
emptyDir: {}
{{- end }}
{{- if .Values.extraVolumes }}
{{ toYaml .Values.extraVolumes | nindent 8 }}
{{- toYaml .Values.extraVolumes | nindent 8 }}
{{- end }}
{{- if .Values.dind.extraVolumes }}
{{- toYaml .Values.dind.extraVolumes | nindent 8 }}
{{- end }}
{{- end }}
2 changes: 1 addition & 1 deletion charts/github-actions-runner/templates/configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ data:
enabled: true
http:
addr: :8443
host: https://{{ include "github-actions-runner.fullname" . }}-registry.{{ .Release.Namespace }}.svc
# host: https://{{ include "github-actions-runner.fullname" . }}-registry.{{ .Release.Namespace }}.svc
tls:
certificate: /etc/registry-tls/tls.crt
key: /etc/registry-tls/tls.key
Expand Down
Loading

0 comments on commit 33b69ea

Please sign in to comment.