[DOCS-11274] Restructure and simplify OP docs

content/en/observability_pipelines/_index.md

@@ -67,72 +67,25 @@ The Observability Pipelines Worker is the software that runs in your infrastruct
 
 The Observability Pipelines UI provides a control plane to manage your Observability Pipelines Workers. You build and edit pipelines and deploy pipeline changes to your Workers from there. You can also enable out-of-the-box monitors for your pipelines so that you can evaluate their health.
 
-## Get started
-
-To set up a pipeline:
-
-1. Navigate to [Observability Pipelines][1].
-1. Select a template:
-    - [Log volume control][2]
-    - [Dual ship logs][3]
-    - [Split logs][4]
-    - [Archive logs to Datadog Archives][5]
-    - [Sensitive data redaction][6]
-    - [Log Enrichment][7]
-    - [Generate Metrics][8]
-1. Select and set up your [source][9].
-1. Select and set up your [destinations][10].
-1. Set up your [processors][11].
-1. [Install the Observability Pipelines Worker][14].
-1. Enable monitors for your pipeline.
-
-See [Set Up Pipelines][12] for more information.
-
-See [Advanced Configurations][13] for bootstrapping options and for details on setting up the Worker with Kubernetes.
-
 ## Explore Observability Pipelines
 
 ### Build pipelines with out-of-the-box templates
 
 {{< img src="observability_pipelines/templates_20241003.png" alt="The Observability Pipelines UI showing the six templates" style="width:100%;" >}}
 
-The templates are built for the following use cases:
-
-#### Log Volume Control
-
-Raw logs are noisy, and only some logs are useful for further search and analysis during investigations. Use the Log Volume Control template to determine which logs to send to your indexed solution, such as a SIEM or log management solution. This helps you to increase the value of your indexed logs and also remain within your planned budget.
-
-#### Dual Ship Logs
-
-As your organization grows, your observability needs for different use cases, such as security, archiving, and log management, also change. This could mean having to trial different archiving, SIEM, and log management solutions. However, managing log pipelines to different solutions can be complicated. Use the Dual Ship Logs template to centrally aggregate, process, and send copies of your logs to different destinations.
-
-#### Archive Logs
-
-Use the Archive Logs template to store logs in a cloud storage solution (Amazon S3, Google Cloud Storage, or Azure Storage). The archived logs are stored in a Datadog-rehydratable format, so that they can be rehydrated in Datadog as needed. This is useful when:
-
-- You have a high volume of noisy logs, but might need to index them in Datadog Log Management ad hoc for an investigation.
-- You are migrating to Datadog Log Management and want to have historical logs after completing the migration.
-- You have a retention policy to fulfill compliance requirements but don't necessarily need to index those logs.
+Build pipelines with out-of-the-box templates for the following [use cases][6]:
 
-#### Split Logs
-
-When you have logs from different services and applications, you might need to send them to different downstream services for querying, analysis, and alerting. For example, you might want to send security logs to a SIEM solution and DevOps logs to Datadog. Use the Split Logs template to preprocess your logs separately for each destination before sending them downstream.
-
-#### Sensitive Data Redaction
-
-Use the Sensitive Data Redaction template to detect and redact sensitive information on premises. The Observability Pipelines sensitive data scanner processor provides 70 out-of-the-box scanning rules, but you can also create your own custom scanning rules using regular expressions. The OOTB rules recognize standard patterns such as credit card numbers, email addresses, IP addresses, API and SSH keys, and access tokens.
-
-#### Log Enrichment
-
-Your organization's different services, systems, and applications all generate logs containing layers of information and in different formats. This can make it difficult to extract the data you need when searching and analyzing the data for an investigation. Use the Log Enrichment template to standardize your logs and enrich them with information, such as data from a reference table.
-
-#### Generate Metrics
-
-Some log sources, such as firewalls and network appliances, generate a large volume of log events that contain log data that don't need to be stored. Often, you just want to see a summary of the logs and compare it to historical data. Log-based metrics are also a cost-efficient way to summarize log data from your entire ingest stream. Use the Generate Metrics template to generate a count metric of logs that match a query or a distribution metric of a numeric value contained in the logs, such as a request duration.
+- [Archive Logs][7]
+- [Dual Ship Logs][8]
+- [Generate Metrics][9]
+- [Log Enrichment][10]
+- [Log Volume Control][11]
+- [Sensitive Data Redaction][12]
+- [Split Logs][13]
 
 ### Build pipelines in the Observability Pipelines UI
 
-{{% observability_pipelines/use_case_images/generate_metrics %}}
+{{< img src="observability_pipelines/dual_ship_pipeline.png" alt="Pipeline with one source connect to two processor groups and two destinations" style="width:100%;" >}}
 
 Build your pipelines in the Observability Pipelines UI. After you select one of the out-the-box templates, the onboarding workflow walks you through setting up your source, processors, and destinations. The installation page provides instructions on how to install the Worker in your environment (Docker, Kubernetes, Linux, or CloudFormation).
 
@@ -144,21 +97,28 @@ After you create your pipeline, enable out-of-the box monitors to get alerted wh
 - The Observability Pipelines Worker has high CPU usage or memory usage.
 - There are spikes in data dropped by a component.
 
+## Get started
+
+You must enable [Remote Configuration][1] to use Observability Pipelines.
+
+See [Set Up Pipelines][2] to set up a pipeline in the UI. You can also set up pipelines using the [Observability Pipelines API][3] or [Terraform][4].
+
+See [Advanced Configurations][5] for bootstrapping options and for details on setting up the Worker with Kubernetes.
+
 ## Further Reading
 
 {{< partial name="whats-next/whats-next.html" >}}
 
-[1]: https://app.datadoghq.com/observability-pipelines
-[2]: /observability_pipelines/log_volume_control/
-[3]: /observability_pipelines/dual_ship_logs/
-[4]: /observability_pipelines/split_logs/
-[5]: /observability_pipelines/archive_logs/
-[6]: /observability_pipelines/sensitive_data_redaction/
-[7]: /observability_pipelines/log_enrichment/
-[8]: /observability_pipelines/set_up_pipelines/generate_metrics/
-[9]: /observability_pipelines/sources/
-[10]: /observability_pipelines/destinations/
-[11]: /observability_pipelines/processors/
-[12]: /observability_pipelines/set_up_pipelines/
-[13]: /observability_pipelines/advanced_configurations/
-[14]: /observability_pipelines/install_the_worker/
+[1]: /agent/remote_config/#setup
+[2]: /observability_pipelines/set_up_pipelines/
+[3]: /observability_pipelines/set_up_pipelines/?tab=api#set-up-a-pipeline
+[4]: /observability_pipelines/set_up_pipelines/?tab=terraform#set-up-a-pipeline
+[5]: /observability_pipelines/advanced_configurations/
+[6]: /observability_pipelines/use_cases/
+[7]: /observability_pipelines/use_cases/#archive-logs
+[8]: /observability_pipelines/use_cases/#dual-ship-logs
+[9]: /observability_pipelines/use_cases/#generate-metrics
+[10]: /observability_pipelines/use_cases/#log-enrichment
+[11]: /observability_pipelines/use_cases/#log-volume-control
+[12]: /observability_pipelines/use_cases/#sensitive-data-redaction
+[13]: /observability_pipelines/use_cases/#split-logs

content/en/observability_pipelines/destinations/_index.md

@@ -9,16 +9,9 @@ further_reading:
 
 ## Overview
 
-Use the Observability Pipelines Worker to send your processed logs to different destinations.
+Use the Observability Pipelines Worker to send your processed logs to different destinations. Most Observability Pipelines destinations send events in batches to the downstream integration. See [Event batching](#event-batching) for more information. Some Observability Pipelines' destinations also have fields that support template syntax so you can set those fields based on specific log fields. See [Template syntax](#template-syntax) for more information.
 
-Select and set up your destinations when you [set up a pipeline][1]. This is step 4 in the pipeline setup process:
-
-1. Navigate to [Observability Pipelines][2].
-1. Select a template.
-1. Select and set up your source.
-1. Select and set up your destinations.
-1. Set up your processors.
-1. Install the Observability Pipelines Worker.
+Select a destination in the left navigation menu to see more information about it.
 
 ## Template syntax
 

content/en/observability_pipelines/install_the_worker/_index.md

@@ -12,13 +12,7 @@ further_reading:
 
 ## Overview
 
-The Observability Pipelines Worker is software that runs in your environment to centrally aggregate, process, and route your logs. You install and configure the Worker as part of the pipeline setup process. These are the general steps if you are setting up a pipeline in the UI:
-
-1. Select a log [source][2].
-1. Select [destinations][3] to which you want to send your logs.
-1. Select and configure [processors][4] to transform your logs.
-1. [Install the Worker](#install-the-worker).
-1. Deploy the pipeline.
+The Observability Pipelines Worker is software that runs in your environment to centrally aggregate, process, and route your logs.
 
 **Note**: If you are using a proxy, see the `proxy` option in [Bootstrap options][1].
 

content/en/observability_pipelines/processors/_index.md

@@ -11,17 +11,11 @@ further_reading:
 
 <div class="alert alert-info">The processors outlined in this documentation are specific to on-premises logging environments. To parse, structure, and enrich cloud-based logs, see the <a href="https://docs.datadoghq.com/logs/log_configuration/logs_to_metrics">Log Management</a> documentation.</div>
 
-Use Observability Pipelines' processors to parse, structure, and enrich your logs. All processors are available for all templates. Set up your processors in the Observability Pipelines UI after you have selected a template, source, and destinations. This is step 5 in the pipeline setup process:
+Use Observability Pipelines' processors to parse, structure, and enrich your logs. When you create a pipeline in the UI, pre-selected processors are added to your processor group based on the selected template. You can add additional processors and delete any existing ones based on your processing needs.
 
-1. Navigate to [Observability Pipelines][1].
-1. Select a template.
-1. Select and set up your source.
-1. Select and set up your destinations.
-1. Set up your processors.
-1. Install the Observability Pipelines Worker.
-1. Enable monitors for your pipeline.
+Processor groups are executed from top to bottom. The order of the processors is important because logs are checked by each processor, but only logs that match the processor's filters are processed. To modify the order of the processors, use the drag handle on the top left corner of the processor you want to move.
 
-{{% observability_pipelines/processors/intro %}}
+Select a processor in the left navigation menu to see more information about it.
 
 {{% observability_pipelines/processors/filter_syntax %}}
 

content/en/observability_pipelines/set_up_pipelines/_index.md

@@ -20,56 +20,75 @@ further_reading:
 
 <div class="alert alert-info">The pipelines and processors outlined in this documentation are specific to on-premises logging environments. To aggregate, process, and route cloud-based logs, see <a href="https://docs.datadoghq.com/logs/log_configuration/pipelines/?tab=source">Log Management Pipelines</a>.</div>
 
-In Observability Pipelines, a pipeline is a sequential path with three types of components: source, processors, and destinations. The Observability Pipeline [source][1] receives logs from your log source (for example, the Datadog Agent). The [processors][2] enrich and transform your data, and the [destination][3] is where your processed logs are sent. For some templates, your logs are sent to more than one destination. For example, if you use the Archive Logs template, your logs are sent to a cloud storage provider and another specified destination.
+In Observability Pipelines, a pipeline is a sequential path with three types of components: source, processors, and destinations. The Observability Pipeline [source][1] receives logs from your log source (for example, the Datadog Agent). The [processors][2] enrich and transform your data, and the [destination][3] is where your processed logs are sent.
+
+{{< img src="observability_pipelines/archive_log_pipeline.png" alt="Pipeline with one source connect to two processor groups and two destinations" style="width:100%;" >}}
 
 ## Set up a pipeline
 
 {{< tabs >}}
 {{% tab "Pipeline UI" %}}
 
-Set up your pipelines and its [sources][1], [processors][2], and [destinations][3] in the Observability Pipelines UI. The general setup steps are:
+Set up your pipelines and its sources, processors, and destinations in the Observability Pipelines UI.
 
 1. Navigate to [Observability Pipelines][13].
 1. Select a template.
-1. Select and set up your source.
-1. Select and set up your destinations.
-1. Set up your processors.
-1. [Install the Observability Pipelines Worker][12].
-1. Enable monitors for your pipeline.
-
-For detailed setup instructions, select a template-specific documentation and then select your source from that page:
-  - [Log volume control][4]
-  - [Dual ship logs][5]
-  - [Split logs][6]
-  - [Archive logs to Datadog Archives][7]
-  - [Sensitive data redaction][8]
-  - [Log Enrichment][9]
-  - [Generate Metrics][10]
+    - [Archive Logs][4]
+    - [Dual Ship Logs][5]
+    - [Generate Metrics][6]
+    - [Log Enrichment][7]
+    - [Log Volume Control][8]
+    - [Sensitive Data Redaction][9]
+    - [Split Logs][10]
+1. Select and set up your [source][1].
+1. Select and set up your [destinations][2].
+1. Set up your [processors][3].
+1. If you want to add another set of processors and destinations, click the plus sign (**+**) to the left of the processor group to add another set of processors and destinations to the source.
+    - To delete a processor group, you need to delete all destinations linked to that processor group. When the last destination is deleted, the processor group is removed with it.
+1. If you want to add an additional destination to a processor group, click the plus sign (**+**) to the right of the processor group.
+    - To delete a destination, click on the pencil icon to the top right of the destination, and select **Delete destination**. If you delete a destination from a processor group that has multiple destinations, only the deleted destination is removed. If you delete a destination from a processor group that only has one destination, both the destination and the processor group are removed.
+    - **Notes**:
+      - A pipeline must have at least one destination. If a processor group only has one destination, that destination cannot be deleted.
+      - You can add a total of three destinations for a pipeline.
+      - A specific destination can only be added once. For example, you cannot add multiple Splunk HEC destinations.
+1. Click **Next: Install**.
+1. Select the platform on which you want to install the Worker.
+1. Enter the [environment variables][15] for your sources and destinations, if applicable.
+1. Follow the instructions on installing the Worker for your platform. The command provided in the UI to install the Worker has the relevant environment variables populated. See [Install the Observability Pipelines Worker][12] for more information.
+    - **Note**: If you are using a proxy, see the `proxy` option in [Bootstrap options][16].
+1. Enable out-of-the-box monitors for your pipeline.
+    1. Navigate to the [Pipelines][1] page and find your pipelines.
+    1. Click **Enable monitors** in the **Monitors** column for your pipeline.
+    1. Click **Start** to set up a monitor for one of the suggested use cases.<br>
+      The new metric monitor page is configured based on the use case you selected. You can update the configuration to further customize it. See the [Metric monitor documentation][14] for more information.
 
 After you have set up your pipeline, see [Update Existing Pipelines][11] if you want to make any changes to it.
 
 [1]: /observability_pipelines/sources/
 [2]: /observability_pipelines/processors/
 [3]: /observability_pipelines/destinations/
-[4]: /observability_pipelines/set_up_pipelines/log_volume_control/
-[5]: /observability_pipelines/set_up_pipelines/dual_ship_logs/
-[6]: /observability_pipelines/set_up_pipelines/split_logs/
-[7]: /observability_pipelines/set_up_pipelines/archive_logs/
-[8]: /observability_pipelines/set_up_pipelines/sensitive_data_redaction/
-[9]: /observability_pipelines/set_up_pipelines/log_enrichment/
-[10]: /observability_pipelines/set_up_pipelines/generate_metrics/
+[4]: /observability_pipelines/use_cases/#archive-logs
+[5]: /observability_pipelines/use_cases/#dual-ship-logs
+[6]: /observability_pipelines/use_cases/#generate-metrics
+[7]: /observability_pipelines/use_cases/#log-enrichment
+[8]: /observability_pipelines/use_cases/#log-volume-control
+[9]: /observability_pipelines/use_cases/#sensitive-data-redaction
+[10]: /observability_pipelines/use_cases/#split-logs
 [11]: /observability_pipelines/update_existing_pipelines/
 [12]: /observability_pipelines/install_the_worker/
 [13]: https://app.datadoghq.com/observability-pipelines
+[14]: /monitors/types/metric/
+[15]: /observability_pipelines/environment_variables/
+[16]: /observability_pipelines/advanced_configurations/#bootstrap-options
 
 {{% /tab %}}
 {{% tab "API" %}}
 
 <div class="alert alert-warning">Creating pipelines using the Datadog API is in Preview. Fill out the <a href="https://www.datadoghq.com/product-preview/observability-pipelines-api-and-terraform-support/"> form</a> to request access.</div>
 
-You can use Datadog API to [create a pipeline][1]. After the pipeline has been created, [install the Worker][2] to start sending logs through the pipeline.
+You can use Observability Pipelines API to [create a pipeline][1]. After the pipeline has been created, [install the Worker][2] to start sending logs through the pipeline.
 
-Pipelines created using the API are read-only in the UI. Use the [update a pipeline][3] endpoint to make any changes to an existing pipeline.
+**Note**: Pipelines created using the API are read-only in the UI. Use the [update a pipeline][3] endpoint to make any changes to an existing pipeline.
 
 [1]: /api/latest/observability-pipelines/#create-a-new-pipeline
 [2]: /observability_pipelines/install_the_worker/

content/en/observability_pipelines/set_up_pipelines/archive_logs/_index.md

@@ -3,6 +3,10 @@ title: Archive Logs to Datadog Archives
 disable_toc: false
 aliases:
     - /observability_pipelines/archive_logs/
+further_reading:
+- link: "/observability_pipelines/set_up_pipelines/"
+  tag: "Documentation"
+  text: "Set up a pipeline"
 ---
 
 ## Overview
@@ -15,22 +19,9 @@ Use Observability Pipelines to route ingested logs to a cloud storage solution (
 
 {{% observability_pipelines/use_case_images/archive_logs %}}
 
-Select a source to get started:
+## Further reading
 
-- [Amazon Data Firehose][12]
-- [Amazon S3][11]
-- [Datadog Agent][1]
-- [Fluentd or Fluent Bit][2]
-- [Google Pub/Sub][3]
-- [HTTP Client][4]
-- [HTTP Server][5]
-- [Kafka][13]
-- [Logstash][6]
-- [Splunk HTTP Event Collector (HEC)][7]
-- [Splunk Heavy or Universal Forwarders (TCP)][8]
-- [Socket (TCP or UDP)][14]
-- [Sumo Logic Hosted Collector][9]
-- [rsylsog or syslog-ng][10]
+{{< partial name="whats-next/whats-next.html" >}}
 
 [1]: /observability_pipelines/archive_logs/datadog_agent
 [2]: /observability_pipelines/archive_logs/fluent