Skip to content

Commit

Permalink
fix: namespace examples
Browse files Browse the repository at this point in the history
  • Loading branch information
wrussell1999 committed Nov 12, 2024
1 parent 4e65e37 commit 8e37e4a
Showing 1 changed file with 87 additions and 46 deletions.
133 changes: 87 additions & 46 deletions content/docs/05.concepts/02.namespace-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ namespace: company.team

tasks:
- id: for_each_row
type: io.kestra.plugin.core.flow.EachSequential
value: "{{ trigger.rows }}"
type: io.kestra.plugin.core.flow.ForEach
values: "{{ trigger.rows }}"
tasks:
- id: return
type: io.kestra.plugin.core.debug.Return
Expand All @@ -37,8 +37,8 @@ triggers:
type: io.kestra.plugin.jdbc.clickhouse.Trigger
interval: "PT5M"
url: jdbc:clickhouse://127.0.0.1:56982/
username: ch_user
password: ch_passwd
username: "{{ secret('CLICKHOUSE_USERNAME') }}"
password: "{{ secret('CLICKHOUSE_PASSWORD') }}"
sql: "{{ read('queries/my_query.sql') }}" # 🚀 The read() function reads the content of the file as a string!
fetchType: FETCH
```
Expand Down Expand Up @@ -88,46 +88,6 @@ tasks:

The `Execute` button allows you to run your flow directly from the Code Editor. Click on the `Execute` button to run your flow. You should then see the Execution being created in a new browser tab and once you navigate to the `Logs` tab, you should see a friendly message ``Hello from the Editor!`` in the logs.

To make the example more concrete, here is a simple `weather.py` script that reads a secret to talk to a Weather Data API:

```python
import requests
api_key = '{{ secret("WEATHER_DATA_API_KEY") }}'
url = f"https://api.openweathermap.org/data/2.5/weather?q=Paris&APPID={api_key}"
weather_data = requests.get(url)
print(weather_data.json())
```

And here is the flow that uses the script:
```yaml
id: weather_data
namespace: company.team
tasks:
- id: get_weather
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
include:
- scripts/weather.py
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/pydata:latest
commands:
- python scripts/weather.py
```

#### `namespaceFiles` property

The example above uses the `include` field to only allow the `scripts/weather.py` file to be accessible by the task.

We can control what namespace files are available to our flow with the `namespaceFiles` property.

`namespaceFiles` has 3 attributes:
- `enabled`: when set to true enables all files in that namespace to be visible to the task
- `include`: allows you to specify files you want to be accessible by the task
- `exclude`: allows you to specify files you don't want to be accessible by the task

### GitHub Actions CI/CD

You can leverage our official GitHub Action called [deploy-action](https://github.com/kestra-io/deploy-action) to synchronize your Git repository with a given namespace. This is useful if you want to orchestrate complex Python modules, dbt projects, Terraform or Ansible infrastructure, or any other project that contains code and configuration files with potentially multiple nested directories and files.
Expand Down Expand Up @@ -231,19 +191,100 @@ Then, open the Kestra UI at `http://localhost:28080` and create a new flow with

## How to use Namespace Files in your flows

There are multiple ways to use Namespace Files in your flows. You can use the `read()` function to read the content of a file as a string, or you can point to the file path in the supported tasks.
There are multiple ways to use Namespace Files in your flows. You can use the `read()` function to read the content of a file as a string, point to the file path in the supported tasks or use a dedicated task to retrieve it as an output.

Usually, pointing to a file location, rather than reading the file's content, is required when you want to use a file as an input to a CLI command, e.g. in a `Commands` task such as `io.kestra.plugin.scripts.python.Commands` or `io.kestra.plugin.scripts.node.Commands`. In all other cases, the `read()` function can be used to read the content of a file as a string e.g. in `Query` or `Script` tasks.

You can also use the `io.kestra.plugin.core.flow.WorkingDirectory` task to read namespace files there and then use them in child tasks that require reading the file path in CLI commands e.g. `python scipts/hello.py`.


### The `read()` function

Note how the script in the first section used the `read()` function to read the content of the `scripts/hello.py` file as a string using the expression `"{{ read('scripts/hello.py') }}"`. It'a important to remeber that this function reads **the content of the file as a string**. Therefore, you should use that function only in tasks that expect a string as an input, e.g., `io.kestra.plugin.scripts.python.Script` or `io.kestra.plugin.scripts.node.Script`, rather than `io.kestra.plugin.scripts.python.Commands` or `io.kestra.plugin.scripts.node.Commands`.

The `read()` function allows you to read the content of a Namespace File stored in the Kestra's internal storage backend. The `read()` function takes a single argument, which is the absolute path to the file you want to read. The path must point to a file stored in the **same namespace** as the flow you are executing.

In this example, we have a namespace file called `example.txt` that contains the text `Hello, World!`. We can print the content to the logs by using `{{ read('example.txt') }}`:

```yaml
id: files
namespace: company.team
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: "{{ read('example.txt') }}"
```

### `namespaceFiles.enabled` on supported tasks

With supported tasks, such as the `io.kestra.plugin.scripts` group, we can access files using their path and enabling the task to read namespace files.

Here is a simple `weather.py` script that reads a secret to talk to a Weather Data API:

```python
import requests
api_key = '{{ secret("WEATHER_DATA_API_KEY") }}'
url = f"https://api.openweathermap.org/data/2.5/weather?q=Paris&APPID={api_key}"
weather_data = requests.get(url)
print(weather_data.json())
```

And here is the flow that uses the script:
```yaml
id: weather_data
namespace: company.team
tasks:
- id: get_weather
type: io.kestra.plugin.scripts.python.Commands
namespaceFiles:
enabled: true
include:
- scripts/weather.py
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/pydata:latest
commands:
- python scripts/weather.py
```

#### `namespaceFiles` property

The example above uses the `include` field to only allow the `scripts/weather.py` file to be accessible by the task.

We can control what namespace files are available to our flow with the `namespaceFiles` property.

`namespaceFiles` has 3 attributes:
- `enabled`: when set to true enables all files in that namespace to be visible to the task
- `include`: allows you to specify files you want to be accessible by the task
- `exclude`: allows you to specify files you don't want to be accessible by the task

### Namespace Tasks

You can use the Namespace Tasks to upload, download and delete tasks in Kestra.

In the example below, we have a namespace file called `example.ion` that we want to convert to a csv file. We can use the `DownloadFiles` task to generate an output that contains the file so we can easily pass it dynamically to the `IonToCsv` task.

```yaml
id: files
namespace: company.team
tasks:
- id: namespace
type: io.kestra.plugin.core.namespace.DownloadFiles
namespace: company.team
files:
- example.ion
- id: ion_to_csv
type: io.kestra.plugin.serdes.csv.IonToCsv
from: "{{ outputs.namespace.files['/example.ion'] }}"
```

Read more about the tasks below:
- [UploadFiles](/plugins/core/tasks/namespace/io.kestra.plugin.core.namespace.uploadfiles)
- [DownloadFiles](/plugins/core/tasks/namespace/io.kestra.plugin.core.namespace.downloadfiles)
- [DeleteFiles](/plugins/core/tasks/namespace/io.kestra.plugin.core.namespace.deletefiles)

## Include / Exclude Namespace Files

You can selectively include or exclude namespace files.
Expand Down

0 comments on commit 8e37e4a

Please sign in to comment.