Document Tekton Pipeline support with pod integration #3898
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
release-note-none
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: kannon92 The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
cncf-cla: yes
✅ Deploy Preview for kubernetes-sigs-kueue ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
kannon92
changed the title
tenzen-y
reviewed
Jan 2, 2025
| date: 2024-01-03 | ||
| weight: 7 | ||
| description: > | ||
| Integrate Kueue with TektonCD Pipelines. |
There was a problem hiding this comment.
Suggested change
| Integrate Kueue with TektonCD Pipelines. | |
| Integrate Kueue with Tekton Pipelines. |
Isn't it the appropriate name is Tekton Pipeline instead of TektonCD Pipeline?
https://github.com/tektoncd/pipeline?tab=readme-ov-file#-tekton-pipelines
| Integrate Kueue with TektonCD Pipelines. | ||
| --- | ||
|
|
||
| This page shows how to leverage Kueue's scheduling and resource management capabilities when running [TektonCD pipelines](https://tekton.dev/docs/). |
There was a problem hiding this comment.
Suggested change
| This page shows how to leverage Kueue's scheduling and resource management capabilities when running [TektonCD pipelines](https://tekton.dev/docs/). | |
| This page shows how to leverage Kueue's scheduling and resource management capabilities when running [Tekton pipelines](https://tekton.dev/docs/). |
Comment on lines
+13
to
+14
| As of the writing of this doc, Kueue doesn't support TektonCD Pipelines [Pipelines](https://github.com/tektoncd/pipeline/blob/main/docs/README.md) resources directly, | ||
| but you can take advantage of the ability for Kueue to [manage plain pods](/docs/tasks/run_plain_pods) to integrate them. |
There was a problem hiding this comment.
Suggested change
| As of the writing of this doc, Kueue doesn't support TektonCD Pipelines [Pipelines](https://github.com/tektoncd/pipeline/blob/main/docs/README.md) resources directly, | |
| but you can take advantage of the ability for Kueue to [manage plain pods](/docs/tasks/run_plain_pods) to integrate them. | |
| We demonstrate how to support scheduling Tekton Pipelines Tasks in Kueue based on the [Plain Pod](/docs/tasks/run_plain_pods) integration, where every Pod from a Pipeline is represented as a single independent Plain Pod. |
I would like to keep mentioning only end user-facing descriptions here.
Comment on lines
+18
to
+20
| 1. By default, the integration for `v1/pod` is not enabled. | ||
| Learn how to [install Kueue with a custom manager configuration](/docs/installation/#install-a-custom-configured-released-version) | ||
| and enable the `pod` integration. |
There was a problem hiding this comment.
Suggested change
| 1. By default, the integration for `v1/pod` is not enabled. | |
| Learn how to [install Kueue with a custom manager configuration](/docs/installation/#install-a-custom-configured-released-version) | |
| and enable the `pod` integration. | |
| 1. Learn how to [install Kueue with a custom manager configuration](/docs/installation/#install-a-custom-configured-released-version). | |
| 2. Follow the steps in [Run Plain Pods](docs/tasks/run/plain_pods/#before-you-begin) to learn how to enable and configure the `v1/pod` integration. | |
| 3. Check [Administrator cluster quotas](/docs/tasks/manage/administer_cluster_quotas/) for details on the initial Kueue step. |
Could you delegate the duplicated descriptions to the existing documentation?
Comment on lines
+22
to
+51
| The pod integration for TektonCD Pipelines could look like: | ||
|
|
||
| ```yaml | ||
| apiVersion: config.kueue.x-k8s.io/v1beta1 | ||
| kind: Configuration | ||
| integrations: | ||
| frameworks: | ||
| - "pod" | ||
| podOptions: | ||
| # You can change namespaceSelector to define in which | ||
| # namespaces kueue will manage the tektoncd pods. | ||
| namespaceSelector: | ||
| matchExpressions: | ||
| - key: kubernetes.io/metadata.name | ||
| operator: NotIn | ||
| values: [ kube-system, kueue-system ] | ||
| # Tekton pipelines uses the app.kubernetes.io/managed-by label to | ||
| # keep track of pods it manages. We will use that as a hint for Kueue | ||
| # to find Tekton pods. | ||
| podSelector: | ||
| matchExpressions: | ||
| - key: app.kubernetes.io/managed-by | ||
| operator: In | ||
| values: [ "tekton-pipelines" ] | ||
| ``` | ||
|
|
||
| 2. Pods that belong to other API resources managed by Kueue are excluded from being queued by `pod` integration. | ||
| For example, pods managed by `batch/v1.Job` won't be managed by `pod` integration. | ||
|
|
||
| 3. Check [Administer cluster quotas](/docs/tasks/administer_cluster_quotas) for details on the initial Kueue setup. |
There was a problem hiding this comment.
Once we describe the basic Kueue setting the above, we can get rid of these configurations.
Comment on lines
+40
to
+45
| # to find Tekton pods. | ||
| podSelector: | ||
| matchExpressions: | ||
| - key: app.kubernetes.io/managed-by | ||
| operator: In | ||
| values: [ "tekton-pipelines" ] |
There was a problem hiding this comment.
I do not think that we should configure this Pod selector since this selector prevents the management of Pod except for the Tekton Pipeline Pods.
Comment on lines
+55
to
+76
| ## Motivation | ||
|
|
||
| Let's say you are providing a build platform for a software delivery service. | ||
| During a busy time of a year, you have many engineers creating build pipelines and your cluster is at capacity due to the number of builds created. | ||
| One can use Kueue to limit the number of pods that are created by using the pod integration. | ||
| Kueue provides quota limits on core resources so one could create a clusterqueue that limits the number of pods. | ||
|
|
||
| ```yaml | ||
| apiVersion: kueue.x-k8s.io/v1beta1 | ||
| kind: ClusterQueue | ||
| metadata: | ||
| name: "cluster-queue" | ||
| spec: | ||
| namespaceSelector: {} # match all. | ||
| resourceGroups: | ||
| - coveredResources: ["pods"] | ||
| flavors: | ||
| - name: "default-flavor" | ||
| resources: | ||
| - name: "pods" | ||
| nominalQuota: 2 | ||
| ``` |
There was a problem hiding this comment.
Suggested change
| ## Motivation | |
| Let's say you are providing a build platform for a software delivery service. | |
| During a busy time of a year, you have many engineers creating build pipelines and your cluster is at capacity due to the number of builds created. | |
| One can use Kueue to limit the number of pods that are created by using the pod integration. | |
| Kueue provides quota limits on core resources so one could create a clusterqueue that limits the number of pods. | |
| ```yaml | |
| apiVersion: kueue.x-k8s.io/v1beta1 | |
| kind: ClusterQueue | |
| metadata: | |
| name: "cluster-queue" | |
| spec: | |
| namespaceSelector: {} # match all. | |
| resourceGroups: | |
| - coveredResources: ["pods"] | |
| flavors: | |
| - name: "default-flavor" | |
| resources: | |
| - name: "pods" | |
| nominalQuota: 2 | |
| ``` |
I do not think that we want to mention this here since this is end users like batch users' documentation.
| As a simple example, we will define two tasks named sleep and hello: | ||
|
|
||
| Tasks: | ||
|
|
There was a problem hiding this comment.
Could you move Tekton specific manifests to https://github.com/kubernetes-sigs/kueue/tree/main/site/static/examples?
|
|
||
| This will inject the kueue label on every pod of the pipeline. Kueue will gate the pods once you are over the quota limits. | ||
|
|
||
| ## c. How it works |
There was a problem hiding this comment.
I think that we should remove this section since this page does not indicate the concept and mechanism. We just describe the user's documentation step by step.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind documentation
What this PR does / why we need it:
Example integration with TektonCD
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?