diff --git a/content/docs/05.concepts/02.namespace-files.md b/content/docs/05.concepts/02.namespace-files.md index d0498b6e0a..d7897cf36b 100644 --- a/content/docs/05.concepts/02.namespace-files.md +++ b/content/docs/05.concepts/02.namespace-files.md @@ -25,8 +25,8 @@ namespace: company.team tasks: - id: for_each_row - type: io.kestra.plugin.core.flow.EachSequential - value: "{{ trigger.rows }}" + type: io.kestra.plugin.core.flow.ForEach + values: "{{ trigger.rows }}" tasks: - id: return type: io.kestra.plugin.core.debug.Return @@ -37,8 +37,8 @@ triggers: type: io.kestra.plugin.jdbc.clickhouse.Trigger interval: "PT5M" url: jdbc:clickhouse://127.0.0.1:56982/ - username: ch_user - password: ch_passwd + username: "{{ secret('CLICKHOUSE_USERNAME') }}" + password: "{{ secret('CLICKHOUSE_PASSWORD') }}" sql: "{{ read('queries/my_query.sql') }}" # 🚀 The read() function reads the content of the file as a string! fetchType: FETCH ``` @@ -88,46 +88,6 @@ tasks: The `Execute` button allows you to run your flow directly from the Code Editor. Click on the `Execute` button to run your flow. You should then see the Execution being created in a new browser tab and once you navigate to the `Logs` tab, you should see a friendly message ``Hello from the Editor!`` in the logs. -To make the example more concrete, here is a simple `weather.py` script that reads a secret to talk to a Weather Data API: - -```python -import requests -api_key = '{{ secret("WEATHER_DATA_API_KEY") }}' -url = f"https://api.openweathermap.org/data/2.5/weather?q=Paris&APPID={api_key}" -weather_data = requests.get(url) -print(weather_data.json()) -``` - -And here is the flow that uses the script: -```yaml -id: weather_data -namespace: company.team - -tasks: - - id: get_weather - type: io.kestra.plugin.scripts.python.Commands - namespaceFiles: - enabled: true - include: - - scripts/weather.py - taskRunner: - type: io.kestra.plugin.scripts.runner.docker.Docker - containerImage: ghcr.io/kestra-io/pydata:latest - commands: - - python scripts/weather.py -``` - -#### `namespaceFiles` property - -The example above uses the `include` field to only allow the `scripts/weather.py` file to be accessible by the task. - -We can control what namespace files are available to our flow with the `namespaceFiles` property. - -`namespaceFiles` has 3 attributes: -- `enabled`: when set to true enables all files in that namespace to be visible to the task -- `include`: allows you to specify files you want to be accessible by the task -- `exclude`: allows you to specify files you don't want to be accessible by the task - ### GitHub Actions CI/CD You can leverage our official GitHub Action called [deploy-action](https://github.com/kestra-io/deploy-action) to synchronize your Git repository with a given namespace. This is useful if you want to orchestrate complex Python modules, dbt projects, Terraform or Ansible infrastructure, or any other project that contains code and configuration files with potentially multiple nested directories and files. @@ -231,19 +191,100 @@ Then, open the Kestra UI at `http://localhost:28080` and create a new flow with ## How to use Namespace Files in your flows -There are multiple ways to use Namespace Files in your flows. You can use the `read()` function to read the content of a file as a string, or you can point to the file path in the supported tasks. +There are multiple ways to use Namespace Files in your flows. You can use the `read()` function to read the content of a file as a string, point to the file path in the supported tasks or use a dedicated task to retrieve it as an output. Usually, pointing to a file location, rather than reading the file's content, is required when you want to use a file as an input to a CLI command, e.g. in a `Commands` task such as `io.kestra.plugin.scripts.python.Commands` or `io.kestra.plugin.scripts.node.Commands`. In all other cases, the `read()` function can be used to read the content of a file as a string e.g. in `Query` or `Script` tasks. You can also use the `io.kestra.plugin.core.flow.WorkingDirectory` task to read namespace files there and then use them in child tasks that require reading the file path in CLI commands e.g. `python scipts/hello.py`. - ### The `read()` function Note how the script in the first section used the `read()` function to read the content of the `scripts/hello.py` file as a string using the expression `"{{ read('scripts/hello.py') }}"`. It'a important to remeber that this function reads **the content of the file as a string**. Therefore, you should use that function only in tasks that expect a string as an input, e.g., `io.kestra.plugin.scripts.python.Script` or `io.kestra.plugin.scripts.node.Script`, rather than `io.kestra.plugin.scripts.python.Commands` or `io.kestra.plugin.scripts.node.Commands`. The `read()` function allows you to read the content of a Namespace File stored in the Kestra's internal storage backend. The `read()` function takes a single argument, which is the absolute path to the file you want to read. The path must point to a file stored in the **same namespace** as the flow you are executing. +In this example, we have a namespace file called `example.txt` that contains the text `Hello, World!`. We can print the content to the logs by using `{{ read('example.txt') }}`: + +```yaml +id: files +namespace: company.team + +tasks: + - id: log + type: io.kestra.plugin.core.log.Log + message: "{{ read('example.txt') }}" +``` + +### `namespaceFiles.enabled` on supported tasks + +With supported tasks, such as the `io.kestra.plugin.scripts` group, we can access files using their path and enabling the task to read namespace files. + +Here is a simple `weather.py` script that reads a secret to talk to a Weather Data API: + +```python +import requests +api_key = '{{ secret("WEATHER_DATA_API_KEY") }}' +url = f"https://api.openweathermap.org/data/2.5/weather?q=Paris&APPID={api_key}" +weather_data = requests.get(url) +print(weather_data.json()) +``` + +And here is the flow that uses the script: +```yaml +id: weather_data +namespace: company.team + +tasks: + - id: get_weather + type: io.kestra.plugin.scripts.python.Commands + namespaceFiles: + enabled: true + include: + - scripts/weather.py + taskRunner: + type: io.kestra.plugin.scripts.runner.docker.Docker + containerImage: ghcr.io/kestra-io/pydata:latest + commands: + - python scripts/weather.py +``` + +#### `namespaceFiles` property + +The example above uses the `include` field to only allow the `scripts/weather.py` file to be accessible by the task. + +We can control what namespace files are available to our flow with the `namespaceFiles` property. + +`namespaceFiles` has 3 attributes: +- `enabled`: when set to true enables all files in that namespace to be visible to the task +- `include`: allows you to specify files you want to be accessible by the task +- `exclude`: allows you to specify files you don't want to be accessible by the task + +### Namespace Tasks + +You can use the Namespace Tasks to upload, download and delete tasks in Kestra. + +In the example below, we have a namespace file called `example.ion` that we want to convert to a csv file. We can use the `DownloadFiles` task to generate an output that contains the file so we can easily pass it dynamically to the `IonToCsv` task. + +```yaml +id: files +namespace: company.team +tasks: + - id: namespace + type: io.kestra.plugin.core.namespace.DownloadFiles + namespace: company.team + files: + - example.ion + + - id: ion_to_csv + type: io.kestra.plugin.serdes.csv.IonToCsv + from: "{{ outputs.namespace.files['/example.ion'] }}" +``` + +Read more about the tasks below: +- [UploadFiles](/plugins/core/tasks/namespace/io.kestra.plugin.core.namespace.uploadfiles) +- [DownloadFiles](/plugins/core/tasks/namespace/io.kestra.plugin.core.namespace.downloadfiles) +- [DeleteFiles](/plugins/core/tasks/namespace/io.kestra.plugin.core.namespace.deletefiles) + ## Include / Exclude Namespace Files You can selectively include or exclude namespace files.