Skip to content

Commit

Permalink
feat(docs): Task Runs Page (#1931)
Browse files Browse the repository at this point in the history
* fix: add task runs page

* fix: add more to task run page

* fix: taskrun expression

* fix: add error block WIP

* fix: move task run object

* fix: remove error placeholder
  • Loading branch information
wrussell1999 authored Nov 14, 2024
1 parent 0720bf3 commit 1281c23
Show file tree
Hide file tree
Showing 5 changed files with 217 additions and 86 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@ tasks:
For processing items, or forwarding processing to a subflow, [ForEachItem](#foreachitem) is better suited.

::next-link
[ForEach Task documentation](https://kestra.io/plugins/core/tasks/flow/io.kestra.plugin.core.flow.foreach)
[ForEach Task documentation](/plugins/core/tasks/flow/io.kestra.plugin.core.flow.foreach)
::

### ForEachItem
Expand Down
211 changes: 211 additions & 0 deletions content/docs/04.workflow-components/01.tasks/02.taskruns.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
---
title: Task Runs
icon: /docs/icons/flow.svg
---

A Task Run is a single run of an individual task within an [Execution](../03.execution.md), where an Execution is a single run of a flow. This means an Execution can have many Task Runs.

Each Task Run has associated data such as:

- Execution ID
- State
- Start Date
- End Date

## Attempts

Each task run can have one or more attempts. Most task runs will have only one attempt, but you can configure [retries](../12.retries.md) for a task.
If retries have been configured, a task failure will generate new attempts until the retry `maxAttempt` or `maxDuration` threshold is hit.

## States

Similar to Executions, Task Runs cans be in a particular state.

| State | Description |
| - |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `CREATED` | The Execution or Task Run is waiting to be processed. This state usually means that the Execution is in a queue and is yet to be started. |
| `RUNNING` | The Execution or Task Run is currently being processed. |
| `SUCCESS` | The Execution or Task Run has been completed successfully. |
| `WARNING` | The Execution or Task Run exhibited unintended behavior, but the execution continued and was flagged with a warning. |
| `FAILED` | The Execution or Task Run exhibited unintended behavior that caused the execution to fail. |
| `RETRYING` | The Execution or Task Run is currently being [retried](./12.retries.md). |
| `RETRIED` | An Execution or Task Run exhibited unintended behavior, stopped, and created a new execution as defined by its [flow-level retry policy](./12.retries.md#flow-level-retries). The policy was set to the `CREATE_NEW_EXECUTION` behavior. |
| `KILLING` | A command was issued that asked for the Execution or Task Run to be killed. The system is in the process of killing the associated tasks. |
| `KILLED` | An Execution or Task Run was killed (upon request), and no more tasks will run. |

::alert{type="info"}
For a detailed overview of how each Task Run transition through different states, see the [States](../17.states.md#how-are-task-run-states-different-from-execution-states) page.
::

## Expression

You can access information about a taskrun using the `{{ taskrun }}` expression.


This example returns the information from `{{ taskrun }}`:
```yaml
id: taskrun
namespace: company.team

tasks:
- id: return
type: io.kestra.plugin.core.debug.Return
format: "{{ taskrun }}"
```
The logs show the following:
```json
{
"id": "61TxwXQjkXfwTd4ANK6fhv",
"startDate": "2024-11-13T14:38:38.355668Z",
"attemptsCount": 0
}
```

Some [Flowable Tasks](./00.flowable-tasks.md), such as [ForEach](./00.flowable-tasks.md) and [ForEachItem](./00.flowable-tasks.md#foreachitem), group tasks together. You can use the expression `{{ taskrun.value }}` to access the value for that task run.

In the example below, `foreach` will iterate twice over the values `[1, 2]`:

```yaml
id: loop
namespace: company.team

tasks:
- id: foreach
type: io.kestra.plugin.core.flow.ForEach
values: [1, 2]
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "{{ taskrun.value }}"
```
This outputs two separate log tasks, one with `1` and the other with `2`.

You can also use the `{{ parents }}` expression to access a task run value from a parent task. Here's an example of it with `ForEach`:

```yaml
id: loop
namespace: company.team
tasks:
- id: foreach
type: io.kestra.plugin.core.flow.ForEach
values: [1, 2]
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "{{ taskrun.value }}"
- id: if
type: io.kestra.plugin.core.flow.If
condition: "{{ true }}"
then:
- id: log_parent
type: io.kestra.plugin.core.log.Log
message: "{{ parents }}"
```
This will iterate through the `log` and `if` tasks twice as there are two items in `values` property.

The `log_parent` task outputs the task run value produced by `foreach` on the first iteration:

```json
[
{
"taskrun": {
"value": "1"
}
}
]
```

::collapse{title="Task Run JSON Object Example"}
```json
{
"id": "5cBZ1JF8kim8fbFg13bumX",
"executionId": "6s1egIkxu3gpzzILDnyxTn",
"namespace": "io.kestra.tests",
"flowId": "each-sequential-nested",
"taskId": "1-1_return",
"parentTaskRunId": "5ABxhOwhpd2X8DtwUPKERJ",
"value": "s1",
"attempts": [
{
"metrics": [
{
"name": "length",
"tags": {
"format": "{{task.id}} > {{taskrun.value}} ⬅ {{taskrun.startDate}}"
},
"value": 45.0,
"type": "counter"
},
{
"name": "duration",
"tags": {
"format": "{{task.id}} > {{taskrun.value}} ⬅ {{taskrun.startDate}}"
},
"type": "timer",
"value": "PT0.007213673S"
}
],
"state": {
"current": "SUCCESS",
"histories": [
{
"state": "CREATED",
"date": "2021-05-04T12:02:54.121836Z"
},
{
"state": "RUNNING",
"date": "2021-05-04T12:02:54.121841Z"
},
{
"state": "SUCCESS",
"date": "2021-05-04T12:02:54.131892Z"
}
],
"duration": "PT0.010056S",
"endDate": "2021-05-04T12:02:54.131892Z",
"startDate": "2021-05-04T12:02:54.121836Z"
}
}
],
"outputs": {
"value": "1-1_return > s1 ⬅ 2021-05-04T12:02:53.938333Z"
},
"state": {
"current": "SUCCESS",
"histories": [
{
"state": "CREATED",
"date": "2021-05-04T12:02:53.938333Z"
},
{
"state": "RUNNING",
"date": "2021-05-04T12:02:54.116336Z"
},
{
"state": "SUCCESS",
"date": "2021-05-04T12:02:54.144135Z"
}
],
"duration": "PT0.205802S",
"endDate": "2021-05-04T12:02:54.144135Z",
"startDate": "2021-05-04T12:02:53.938333Z"
}
}
```
::

Read more about it on the [Expression Usage](../../expressions/02.expression-usage.md#parent-tasks-with-flowable-tasks) page.

## Task Runs Page (EE)

::alert{type="info"}
This feature is only available on the Enterprise Edition
::

If you have Kestra setup using the [Kafka and Elasticsearch backend](../../07.architecture/index.md#architecture-with-kafka-and-elasticsearch-backend), you can view Task Runs in the UI.

![taskrun_view](/docs/workflow-components/taskrun_view.png)

It's similar to the [Execution View](../../08.ui/02.executions.md) but only shows Task Runs.
4 changes: 3 additions & 1 deletion content/docs/04.workflow-components/03.execution.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,11 @@ Each task run has associated data such as:
- Start Date
- End Date

Read more about Task Runs on the [dedicated docs page](./01.tasks/02.taskruns.md).

## Attempts

Each task run can have one or more attempts. Most task runs will have only one attempt, but you can configure retries for a task.
Each task run can have one or more attempts. Most task runs will have only one attempt, but you can configure [retries](./12.retries.md) for a task.
If retries have been configured, a task failure will generate new attempts until the retry `maxAttempt` or `maxDuration` threshold is hit.

## Outputs
Expand Down
86 changes: 2 additions & 84 deletions content/docs/14.best-practices/0.flows.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,92 +51,10 @@ In most cases, there is a `stores` property that can handle a large volume of da


## Parallel Task

Using the [EachParallel](/plugins/core/tasks/flows/io.kestra.plugin.core.flow.EachParallel) task or the [Parallel](/plugins/core/tasks/flows/io.kestra.plugin.core.flow.Parallel) task is a convenient way to optimize flow duration, but keep in mind that, by default, **all parallel tasks are launched at the same time** (unless you specify the `concurrent` property). The only limit will be the number of worker threads you have configured.
Using the [Parallel](/plugins/core/tasks/flows/io.kestra.plugin.core.flow.Parallel) task is a convenient way to optimize flow duration, but keep in mind that, by default, **all parallel tasks are launched at the same time** (unless you specify the `concurrent` property). The only limit will be the number of worker threads you have configured.

Keep this in mind, because you cannot allow parallel tasks to reach the limit of external systems, such as connection limits or quotas.


## Duration of Tasks

By default, Kestra **never limits the duration** (unless specified explicitly on the task's documentation) of the tasks. If you have a long-running process or an infinite loop, the tasks will never end. We can control the timeout on RunnableTask with the property `timeout` that takes a [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration like `PT5M` for a duration of 5 minutes.

## Task run JSON

Here is an example of a TaskRun JSON object:
```json
{
"id": "5cBZ1JF8kim8fbFg13bumX",
"executionId": "6s1egIkxu3gpzzILDnyxTn",
"namespace": "io.kestra.tests",
"flowId": "each-sequential-nested",
"taskId": "1-1_return",
"parentTaskRunId": "5ABxhOwhpd2X8DtwUPKERJ",
"value": "s1",
"attempts": [
{
"metrics": [
{
"name": "length",
"tags": {
"format": "{{task.id}} > {{taskrun.value}} ⬅ {{taskrun.startDate}}"
},
"value": 45.0,
"type": "counter"
},
{
"name": "duration",
"tags": {
"format": "{{task.id}} > {{taskrun.value}} ⬅ {{taskrun.startDate}}"
},
"type": "timer",
"value": "PT0.007213673S"
}
],
"state": {
"current": "SUCCESS",
"histories": [
{
"state": "CREATED",
"date": "2021-05-04T12:02:54.121836Z"
},
{
"state": "RUNNING",
"date": "2021-05-04T12:02:54.121841Z"
},
{
"state": "SUCCESS",
"date": "2021-05-04T12:02:54.131892Z"
}
],
"duration": "PT0.010056S",
"endDate": "2021-05-04T12:02:54.131892Z",
"startDate": "2021-05-04T12:02:54.121836Z"
}
}
],
"outputs": {
"value": "1-1_return > s1 ⬅ 2021-05-04T12:02:53.938333Z"
},
"state": {
"current": "SUCCESS",
"histories": [
{
"state": "CREATED",
"date": "2021-05-04T12:02:53.938333Z"
},
{
"state": "RUNNING",
"date": "2021-05-04T12:02:54.116336Z"
},
{
"state": "SUCCESS",
"date": "2021-05-04T12:02:54.144135Z"
}
],
"duration": "PT0.205802S",
"endDate": "2021-05-04T12:02:54.144135Z",
"startDate": "2021-05-04T12:02:53.938333Z"
}
}
```
By default, Kestra **never limits the duration** (unless specified explicitly on the task's documentation) of the tasks. If you have a long-running process or an infinite loop, the tasks will never end. We can control the timeout on [Runnable Tasks](../04.workflow-components/01.tasks/01.runnable-tasks.md) with the property [timeout](../04.workflow-components/13.timeout.md) that takes a [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration like `PT5M` for a duration of 5 minutes.
Binary file added public/docs/workflow-components/taskrun_view.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 1281c23

Please sign in to comment.