-
Notifications
You must be signed in to change notification settings - Fork 104
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: Best practices for handling flow outputs (#2199)
- Loading branch information
1 parent
1b3fca0
commit d465eee
Showing
1 changed file
with
60 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
--- | ||
title: Managing and purging flow outputs | ||
icon: /docs/icons/best-practices.svg | ||
--- | ||
|
||
Best practices for handling flow outputs incl. purging large outputs and conditionally returning outputs. | ||
|
||
## Conditionally returning outputs | ||
|
||
When you have a flow that can return different outputs based on some condition, you can use a Pebble expression in the `outputs` section to conditionally return the output of task `A` if it wasn't skipped, and the output of task `B` otherwise. | ||
|
||
```yaml | ||
id: conditionallyReturnOutputs | ||
namespace: company.team | ||
|
||
inputs: | ||
- id: runTask | ||
type: BOOLEAN | ||
defaults: true | ||
|
||
tasks: | ||
- id: taskA | ||
runIf: "{{ inputs.runTask }}" | ||
type: io.kestra.plugin.core.debug.Return | ||
format: Hello World! | ||
|
||
- id: taskB | ||
type: io.kestra.plugin.core.debug.Return | ||
format: Fallback output | ||
|
||
outputs: | ||
- id: flowOutput | ||
type: STRING | ||
value: "{{ tasks.taskA.state != 'SKIPPED' ? outputs.taskA.value : outputs.taskB.value }}" | ||
``` | ||
## Purging large outputs | ||
When you have a flow that generates large outputs that don't need to be kept after the execution is completed, you can use the `io.kestra.plugin.core.storage.PurgeExecutionFiles` task to purge the outputs of the current execution. | ||
|
||
In the example below, the flow downloads a large file from an HTTP API and uploads it to an S3 bucket. After the file is uploaded to S3, it's no longer needed, so the `PurgeExecutionFiles` task is used to delete the file from the internal storage. | ||
|
||
```yaml | ||
id: extractLoadPurge | ||
namespace: company.team | ||
tasks: | ||
- id: extract | ||
type: io.kestra.plugin.core.http.Download | ||
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv | ||
- id: load | ||
type: io.kestra.plugin.aws.s3.Upload | ||
from: "{{ outputs.extractLargeFile.uri }}" | ||
bucket: myBucket | ||
key: largeFiles/orders.csv | ||
- id: purge | ||
type: io.kestra.plugin.core.storage.PurgeCurrentExecutionFiles | ||
``` |