Skip to content

Commit

Permalink
[TEP-0044]: Refocus on Execution Options
Browse files Browse the repository at this point in the history
This commit renames TEP-0044 to "Data Locality and Pod Overhead in Pipelines" and re-focuses it on giving
users more control over how their Pipelines are executed. There are two reasons for this change:

- The problems identified with how Tasks are executed (pod overhead and difficulty passing data between Tasks)
are relevant only when executing multiple Tasks together, not when a single Task is executed in a TaskRun.
- The title "Decoupling Scheduling from Execution" implies that we must choose a runtime-based configuration
for Pipeline execution. This commit clarifies that while execution options must be configurable at runtime,
we may choose to also make them configurable at authoring time in the future.
  • Loading branch information
lbernick authored and tekton-robot committed Feb 2, 2022
1 parent 3f950fe commit b50e8f5
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 10 deletions.
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
---
status: proposed
title: Decouple Task Composition from Scheduling
title: Data Locality and Pod Overhead in Pipelines
creation-date: '2021-01-22'
last-updated: '2021-12-07'
last-updated: '2022-01-26'
authors:
- '@bobcatfish'
- '@lbernick'
---

# TEP-0044: Decouple Task Composition from Scheduling
# TEP-0044: Data Locality and Pod Overhead in Pipelines

<!-- toc -->
- [Summary](#summary)
Expand All @@ -26,14 +26,12 @@ authors:
## Summary

As stated in Tekton's [reusability design principles](https://github.com/tektoncd/community/blob/main/design-principles.md#reusability),
Pipelines and Tasks are meant to capture authoring-time concerns, and to be reusable in a variety of execution contexts.
PipelineRuns and TaskRuns should be able to control execution without the need to modify the corresponding Pipeline or Task.

Pipelines and Tasks should be reusable in a variety of execution contexts.
However, because each TaskRun is executed in a separate pod, Task and Pipeline authors indirectly control the number of pods used in execution.
This introduces both the overhead of extra pods and friction associated with moving data between Tasks.

This TEP lists the pain points associated with running each TaskRun in its own pod and describes the current features that mitigate these pain points.
It explores several options for decoupling Task composition and TaskRun scheduling but does not yet propose a preferred solution.
It explores several additional execution options for Pipelines but does not yet propose a preferred solution.

## Motivation

Expand All @@ -57,8 +55,8 @@ This could be storage within a cluster, like a PVC, configmap, or secret, or rem

Workspaces make it easier to "shuttle" data through a Pipeline by abstracting details of data storage out of Pipelines and Tasks.
They currently support only forms of storage within a cluster (PVCs, configmaps, secrets, and emptydir).
They're an important puzzle piece in decoupling Task composition and scheduling, but they don't address the underlying problem
that some form of external data storage is needed to pass artifacts between TaskRuns.
Abstracting data storage out of Pipeline and Task definitions helps make them more reusable, but doesn't address
the underlying problem that some form of external data storage is needed to pass artifacts between TaskRuns.

The need for data storage locations external to pods introduces friction in a few different ways.
First, moving data between storage locations can incur monetary cost and latency.
Expand Down Expand Up @@ -157,6 +155,8 @@ See [TEP-0074](./0074-deprecate-pipelineresources.md) for the deprecation plan f
- This requirement is being included because we could choose a solution that doesn't
address the above use case; for example in PipelineResources, you can have a
storage "output" but if the steps fail, the "output" pipelineresource will not run
- Any configuration for the execution of a Pipeline must be modifiable at runtime.
- We may explore adding authoring time configuration in the future after gathering feedback on runtime configuration.

## Design details

Expand Down
2 changes: 1 addition & 1 deletion teps/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ This is the complete list of Tekton teps:
|[TEP-0040](0040-ignore-step-errors.md) | Ignore Step Errors | implemented | 2021-08-11 |
|[TEP-0041](0041-tekton-component-versioning.md) | Tekton Component Versioning | implementable | 2021-04-26 |
|[TEP-0042](0042-taskrun-breakpoint-on-failure.md) | taskrun-breakpoint-on-failure | implemented | 2021-12-10 |
|[TEP-0044](0044-decouple-task-composition-from-scheduling.md) | Decouple Task Composition from Scheduling | proposed | 2021-12-07 |
|[TEP-0044](0044-data-locality-and-pod-overhead-in-pipelines.md) | Data Locality and Pod Overhead in Pipelines | proposed | 2022-01-26 |
|[TEP-0045](0045-whenexpressions-in-finally-tasks.md) | WhenExpressions in Finally Tasks | implemented | 2021-06-03 |
|[TEP-0046](0046-finallytask-execution-post-timeout.md) | Finally tasks execution post pipelinerun timeout | implemented | 2021-12-14 |
|[TEP-0047](0047-pipeline-task-display-name.md) | Pipeline Task Display Name | proposed | 2021-02-10 |
Expand Down

0 comments on commit b50e8f5

Please sign in to comment.