Skip to content

Commit

Permalink
(docs): reorganize EE Information Architecture (#2175)
Browse files Browse the repository at this point in the history
* (docs): reorganize ee ia

* add index placeholders

* update indexes & move cli

* update sidebar & faq

* update getting started pages

* update namespace pages

* add worker group use cases

* update links

* small change to trigger ci

* update authentication title

* update auth-user links

* update blog links

* update logshipper links

* update index pages

* update titles
  • Loading branch information
aj-emerich authored Feb 10, 2025
1 parent bfeb0b5 commit be2e1b1
Show file tree
Hide file tree
Showing 85 changed files with 205 additions and 134 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -281,11 +281,11 @@ Apart from new features and integrations, we've also made several improvements t

This release also adds several new features to the Enterprise Edition, including:

🔹 [Azure Secrets Manager](https://kestra.io/docs/administrator-guide/configuration/enterprise-edition/secrets/azure-key-manager) allows you to store your Kestra secrets in Azure Key Vault.
🔹 [Azure Key Vault](../docs/configuration/index.md#azure-key-vault) allows you to store your Kestra secrets in Azure Key Vault.

🔹 [AWS Secrets Manager](https://kestra.io/docs/administrator-guide/configuration/enterprise-edition/secrets/aws-secrets-manager) allows you to store your Kestra secrets in AWS Secrets Manager.
🔹 [AWS Secrets Manager](../docs/configuration/index.md#aws-secret-manager) allows you to store your Kestra secrets in AWS Secrets Manager.

🔹 [Google Secrets Manager](https://kestra.io/docs/administrator-guide/configuration/enterprise-edition/secrets/google-secrets-manager) allows you to store your Kestra secrets in Google Secrets Manager.
🔹 [Google Secrets Manager](../docs/configuration/index.md#google-secret-manager) allows you to store your Kestra secrets in Google Secrets Manager.

Apart from those changes, the [task runs page](https://github.com/kestra-io/kestra-ee/issues/407) now also displays execution labels and attempt numbers.

Expand Down
2 changes: 1 addition & 1 deletion content/blogs/2024-01-22-release-0-14.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ triggers:

The website has now a **new improved search** across the docs, plugins and blog posts, as well as [blueprints](https://kestra.io/blueprints) which are now available not only in the UI but also on the website. The plugins got their own (beautiful!) [homepage](https://kestra.io/plugins) with a dedicated search to make it easier to find the plugin you need.

We've also added new documentation pages for [Azure DevOps CI/CD](https://kestra.io/docs/developer-guide/cicd/05-azure-devops), and [Role-Based Access Control (RBAC)](https://kestra.io/docs/enterprise/rbac).
We've also added new documentation pages for [Azure DevOps CI/CD](../docs/version-control-cicd/cicd/05-azure-devops.md), and [Role-Based Access Control (RBAC)](../docs/06.enterprise/03.auth-users/rbac.md).

## Bug fixes and reliability improvements

Expand Down
4 changes: 2 additions & 2 deletions content/blogs/2024-02-29-release-0-15.md
Original file line number Diff line number Diff line change
Expand Up @@ -285,15 +285,15 @@ We've also added a new generic Singer plugin that simultaneously integrates with

## Enterprise Edition Updates

For Enterprise Edition users, we've added a new [Setup page](/docs/enterprise/setup) in the UI, showing the most important configuration options and streamlining the setup process for a new Kestra instance. That setup wizard will guide you through the initial configuration of your instance, making it easier to get started.
For Enterprise Edition users, we've added a new [Setup page](../docs/06.enterprise/01.getting-started-ee/02.setup.md) in the UI, showing the most important configuration options and streamlining the setup process for a new Kestra instance. That setup wizard will guide you through the initial configuration of your instance, making it easier to get started.

![ee_setup_page](/blogs/2024-02-29-release-0-15/ee_setup_page.png)

We've also revamped the RBAC system, with improved handling of Superadmin access. There is also a dedicated UI page called Access allowing you to configure tenant-level RBAC.

![access_page](/blogs/2024-02-29-release-0-15/access_page.png)

The new **Service Accounts UI page** allows you to create and manage [service accounts](/docs/enterprise/service-accounts), and we've introduced API tokens valid for a specific period of time, allowing you to grant programmatic access to Kestra for Users and Service Accounts. This feature is particularly useful for CI/CD with GitHub Actions and Terraform, as well as for using the API token in API calls.
The new **Service Accounts UI page** allows you to create and manage [service accounts](../docs/06.enterprise/03.auth-users/service-accounts.md), and we've introduced API tokens valid for a specific period of time, allowing you to grant programmatic access to Kestra for Users and Service Accounts. This feature is particularly useful for CI/CD with GitHub Actions and Terraform, as well as for using the API token in API calls.

![service_account_create](/docs/user-interface-guide/service_account_create.png)

Expand Down
4 changes: 2 additions & 2 deletions content/blogs/2024-04-22-liveness-heartbeat.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ This approach was successful in most deployment scenarios. However, in more comp

One of the first disadvantages was that the heartbeat configuration had to be the same for all workers. This configuration was managed globally by the Executor service, which was responsible for detecting unhealthy workers by applying the same rule to all. However, all workers don't necessarily have the same load, the same type of processing or being deployed in the same network. As a result, some workers may be more prone to resource saturation, leading to thread starvation or even network disconnection due to reduced bandwidth.

As an example, Kestra Edition Enterprise provides the [Worker Group](https://kestra.io/docs/enterprise/worker-group) feature, which allows you to create logical groups of Workers. Those groups can then be targeted for specific task executions.  Worker groups come in handy when you need a task to be executed on a worker having specific hardware configurations (GPUs with preconfigured CUDA drivers), in a specific network availability zone, or when you want to isolate long-running and resource-intensive workloads. In such a context, you can relax the heartbeat mechanism and tolerate more missing heartbeats to avoid considering a worker dead when it is not.
As an example, Kestra Edition Enterprise provides the [Worker Group](../docs/06.enterprise/04.scalability-productivity/worker-group.md) feature, which allows you to create logical groups of Workers. Those groups can then be targeted for specific task executions.  Worker groups come in handy when you need a task to be executed on a worker having specific hardware configurations (GPUs with preconfigured CUDA drivers), in a specific network availability zone, or when you want to isolate long-running and resource-intensive workloads. In such a context, you can relax the heartbeat mechanism and tolerate more missing heartbeats to avoid considering a worker dead when it is not.

### Zombies may lead to duplicates

Expand Down Expand Up @@ -94,7 +94,7 @@ Now that we have a better understanding of the lifecycle of services and how the

### Configuring liveness and heartbeat

Starting from Kestra 0.16.0, the liveness and heartbeat mechanism can be configured individually for each service through the properties under `kestra.server.liveness`. This means you can now adapt your configuration depending on the service type, the service load, or even your [Worker Group](https://kestra.io/docs/enterprise/worker-group).
Starting from Kestra 0.16.0, the liveness and heartbeat mechanism can be configured individually for each service through the properties under `kestra.server.liveness`. This means you can now adapt your configuration depending on the service type, the service load, or even your [Worker Group](../docs/06.enterprise/04.scalability-productivity/worker-group.md).

Without going into too much detail, here is the default and recommended configuration for a Kestra JDBC deployment.

Expand Down
2 changes: 1 addition & 1 deletion content/blogs/2024-06-04-release-0-17.md
Original file line number Diff line number Diff line change
Expand Up @@ -494,7 +494,7 @@ When you navigate to any Namespace and go to the `Edit` tab, you can explicitly
However, you can restrict that access if you want only specific namespaces (or no namespace at all) to trigger its corresponding resources.

::alert{type="info"}
Check the [Allowed Namespaces](../docs/06.enterprise/allowed-namespaces.md) documentation for more details.
Check the [Allowed Namespaces](../docs/06.enterprise/02.governance/07.namespace-management.md#allowed-namespaces) documentation for more details.
::

## Improved Execution page
Expand Down
2 changes: 1 addition & 1 deletion content/blogs/2024-06-13-quadis.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ Once the installation and setup of the [secret configuration](https://kestra.io/
## What’s Next

Moving from a legacy system made of more than a hundred pipelines is always a complex task. At Kestra we’re impressed by the involvement of Quadis teams and the speed of their development with Kestra.
After onboarding the first users, they expect to have more than 30 users using Kestra in the coming months. Even more: some of them will be less technical people. Thanks to the Kestra user interface and [role-based access control](https://kestra.io/docs/enterprise/rbac) they could execute flows without the fear of messing up with other works.
After onboarding the first users, they expect to have more than 30 users using Kestra in the coming months. Even more: some of them will be less technical people. Thanks to the Kestra user interface and [role-based access control](../docs/06.enterprise/03.auth-users/rbac.md) they could execute flows without the fear of messing up with other works.

As part of a broader modernization of Quadis operation, Kestra would also be the support for the new data lake architecture.

Expand Down
2 changes: 1 addition & 1 deletion content/blogs/2024-10-17-cd-cd-kestra-comparison.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ Kestra, instead, uses a declarative YAML-based syntax that is much easier to rea
#### Scalability Challenges
Jenkins requires a lot of manual setup for distributed builds or scaling across [multiple nodes](https://www.jenkins.io/doc/book/managing/nodes/), and this can lead to bottlenecks when the infrastructure needs to grow.

In contrast, Kestra leverages [Worker Groups](../docs/06.enterprise/worker-group.md), which are collections of workers that can be targeted for executing tasks based on specific requirements, allowing for efficient workload distribution across different nodes. Additionally, [Task Runners](../docs/06.enterprise/task-runners.md) enable the dynamic allocation of tasks in various cloud environments, facilitating the execution of compute-intensive jobs without the need for permanent infrastructure.
In contrast, Kestra leverages [Worker Groups](../docs/06.enterprise/04.scalability-productivity/worker-group.md), which are collections of workers that can be targeted for executing tasks based on specific requirements, allowing for efficient workload distribution across different nodes. Additionally, [Task Runners](../docs/06.enterprise/04.scalability-productivity/task-runners.md) enable the dynamic allocation of tasks in various cloud environments, facilitating the execution of compute-intensive jobs without the need for permanent infrastructure.

These features provide streamlined and scalable approach to managing complex workflows, reducing the operational overhead associated with scaling Jenkins.

Expand Down
2 changes: 1 addition & 1 deletion content/blogs/release-0-19.md
Original file line number Diff line number Diff line change
Expand Up @@ -493,7 +493,7 @@ Using an invalid worker group key in a task leads to task runs being stuck in a

With the new Worker Groups UI page, worker groups are now treated as API-first objects — they must be created first from the UI, API, CLI, or Terraform before being used in flows. This ensures that worker group keys are valid and exist before they are referenced in tasks.

Check the [Worker Group](https://kestra.io/docs/enterprise/worker-group) documentation to learn how to create and manage worker groups.
Check the [Worker Group](../docs/06.enterprise/04.scalability-productivity/worker-group.md) documentation to learn how to create and manage worker groups.

In short, this new feature improves the way worker groups are managed, reducing the risk of misconfigured flows and providing better visibility into workers' health.

Expand Down
2 changes: 1 addition & 1 deletion content/blogs/release-0-21.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Let's dive into these highlights and other enhancements in more detail.

### Log Shipper

The new [Log Shipper feature](../docs/06.enterprise/logshipper.md) streamlines how you manage and distribute logs across your entire infrastructure. This synchronization automatically batches logs into optimized chunks and manages offset keys. It provides reliable, consistent log delivery without overloading your systems or losing critical data.
The new [Log Shipper feature](../docs/06.enterprise/02.governance/logshipper.md) streamlines how you manage and distribute logs across your entire infrastructure. This synchronization automatically batches logs into optimized chunks and manages offset keys. It provides reliable, consistent log delivery without overloading your systems or losing critical data.

Built on plugin architecture, the Log Shipper can forward logs to Elasticsearch, Datadog, New Relic, Azure Monitor, Google Operational Suite, AWS CloudWatch, and OpenTelemetry.

Expand Down
2 changes: 1 addition & 1 deletion content/blogs/serverless-data-pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Now that we covered what the project does and how it's structured, let's highlig

Serverless is often associated with a tangled mess of functions and services that are hard to manage and debug. But it doesn't have to be that way. With Kestra, you can create structured, modular workflows that are easy to understand, maintain, and scale.

Using [labels](https://kestra.io/docs/workflow-components/labels), [subflows](https://kestra.io/docs/workflow-components/subflows), [flow triggers](https://kestra.io/docs/workflow-components/triggers/flow-trigger), [tenants](https://kestra.io/docs/enterprise/tenants) and [namespaces](https://kestra.io/docs/workflow-components/namespace) you can bring order, structure and governance to serverless workflows.
Using [labels](https://kestra.io/docs/workflow-components/labels), [subflows](https://kestra.io/docs/workflow-components/subflows), [flow triggers](https://kestra.io/docs/workflow-components/triggers/flow-trigger), [tenants](../docs/06.enterprise/02.governance/tenants.md) and [namespaces](https://kestra.io/docs/workflow-components/namespace) you can bring order, structure and governance to serverless workflows.
- Each **dashboard** in Kestra can be filtered by namespaces or labels, so you can easily monitor your serverless data pipelines.
- **Subflows** let you encapsulate common tasks and reuse them across multiple flows.
- **Event triggers** allow you to start a workflow as soon as a new file arrives in a cloud storage bucket or a new message is received in your Pub/Sub topic.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ There are dedicated plugins for `Python`, `R`, `Julia`, `Ruby`, `Node.js`, `Pow

By default, these tasks run in individual Docker containers (taskRunner type: `io.kestra.plugin.scripts.runner.docker.Docker`). You can overwrite that default behavior if you prefer that your scripts run in a local process (taskRunner type: `io.kestra.plugin.core.runner.Process`) instead.

If you use a [commercial version of Kestra](../../../06.enterprise/index.md), you can also run your scripts on [dedicated remote workers](../../../06.enterprise/worker-group.md) by specifying a `workerGroup` property or using other [Task Runner types](../../../task-runners/04.types/index.md) for AWS, GCP, Azure, and Kubernetes.
If you use a [commercial version of Kestra](../../../06.enterprise/index.md), you can also run your scripts on [dedicated remote workers](../../../06.enterprise/04.scalability-productivity/worker-group.md) by specifying a `workerGroup` property or using other [Task Runner types](../../../task-runners/04.types/index.md) for AWS, GCP, Azure, and Kubernetes.

The following pages dive into details of each task runner, supported programming languages, and how to manage dependencies.

Expand Down
2 changes: 1 addition & 1 deletion content/docs/04.workflow-components/01.tasks/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ The table below lists the core task properties available to all tasks.
| `retry` | Task [retry](../../04.workflow-components/12.retries.md) behavior |
| `timeout` | Task [timeout](../../04.workflow-components/13.timeout.md) expressed in [ISO 8601 Durations](https://en.wikipedia.org/wiki/ISO_8601#Durations). |
| `disabled` | Set it to `true` to [disable](../../04.workflow-components/16.disabled.md) execution of the task. |
| `workerGroup.key` | To execute this task on a specific [Worker Group](../../06.enterprise/worker-group.md) (EE) |
| `workerGroup.key` | To execute this task on a specific [Worker Group](../../06.enterprise/04.scalability-productivity/worker-group.md) (EE) |
| `allowFailure` | Boolean flag allowing to [the execution to continue](../../04.workflow-components/11.errors.md) even if this task fails. |
| `logLevel` | Property to define the log level granularity for which logs will be inserted into the backend database. By default, all logs are stored. However, if you restrict that to e.g., the `INFO` level, all lower log levels such as `DEBUG` and TRACE won't be persisted. |
| `logToFile` | Boolean flag that lets you store logs as a file in internal storage. That file can be previewed and downloaded from the `Logs` and `Gantt` Execution tabs. When set to true, logs aren’t saved in the database, which is useful for tasks that produce a large amount of logs that would otherwise take up too much space. The same property can be set on triggers. |
Expand Down
4 changes: 2 additions & 2 deletions content/docs/04.workflow-components/02.namespace.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Namespaces are used to organize workflows and manage access to secrets, plugin d

You can think of a namespace as a **folder for your flows**. Similar to folders on your file system, namespaces can be used to organize flows into logical categories. Similar to filesystems, namespaces can be indefinitely nested.

If you're looking to completely isolate environments with their own resources on the same Kestra instance, you should consider [Tenants](../06.enterprise/03.tenants.md), part of the [Enterprise Edition](../06.enterprise/index.md).
If you're looking to completely isolate environments with their own resources on the same Kestra instance, you should consider [Tenants](../06.enterprise/02.governance/tenants.md), part of the [Enterprise Edition](../06.enterprise/index.md).

## Hierarchical structure when using nested namespaces

Expand Down Expand Up @@ -79,6 +79,6 @@ On the top of this page, you have different tabs:

4. **Dependencies:** shows all the flows and which ones are dependent on each other (for example through Subflows or Flow Triggers).

4. **KV Store:** manage the key-values pairs associated with this namespace. More details on KV Store can be found [here](../05.concepts/05.kv-store.md).
4. **KV Store:** manage the key-values pairs associated with this namespace. See [KV Store](../05.concepts/05.kv-store.md) for more details.

The other tabs: Edit, Variables, Plugin Defaults, Secrets and Audit Logs are only available for Kestra EE. More details about them can be found in our [Enterprise Edition documentation](../06.enterprise/index.md).
Loading

0 comments on commit be2e1b1

Please sign in to comment.