move stuff around

Signed-off-by: Alexa Kreizinger <[email protected]>

Author: alexakreizinger

.gitbook.yaml

@@ -82,3 +82,6 @@ redirects:
 
     # Processors
     processor/metrics_selector:   ./pipeline/processors/metrics_selector.md
+
+    # Other
+    concepts/buffering: ./pipeline/buffering.md

SUMMARY.md

@@ -13,7 +13,6 @@
 ## Concepts
 
 * [Key concepts](concepts/key-concepts.md)
-* [Buffering](concepts/buffering.md)
 * [Data pipeline](concepts/data-pipeline.md)
 
 ## Installation
@@ -83,7 +82,6 @@
 
 ## Data pipeline
 
-* [Pipeline monitoring](pipeline/pipeline-monitoring.md)
 * [Inputs](pipeline/inputs.md)
   * [Collectd](pipeline/inputs/collectd.md)
   * [CPU metrics](pipeline/inputs/cpu-metrics.md)
@@ -168,7 +166,8 @@
   * [Throttle](pipeline/filters/throttle.md)
   * [Type converter](pipeline/filters/type-converter.md)
   * [Wasm](pipeline/filters/wasm.md)
-* [Router](pipeline/router.md)
+* [Buffering](pipeline/buffering.md)
+* [Routing](pipeline/router.md)
 * [Outputs](pipeline/outputs.md)
   * [Amazon CloudWatch](pipeline/outputs/cloudwatch.md)
   * [Amazon Kinesis Data Firehose](pipeline/outputs/firehose.md)
@@ -217,6 +216,7 @@
   * [Treasure Data](pipeline/outputs/treasure-data.md)
   * [Vivo Exporter](pipeline/outputs/vivo-exporter.md)
   * [WebSocket](pipeline/outputs/websocket.md)
+* [Pipeline monitoring](pipeline/pipeline-monitoring.md)
 
 ## Stream processing
 

administration/backpressure.md

@@ -6,13 +6,13 @@ It's possible for logs or data to be ingested or created faster than the ability
 
 To avoid backpressure, Fluent Bit implements a mechanism in the engine that restricts the amount of data an input plugin can ingest. Restriction is done through the configuration parameters `Mem_Buf_Limit` and `storage.Max_Chunks_Up`.
 
-As described in the [Buffering](../concepts/buffering.md) concepts section, Fluent Bit offers two modes for data handling: in-memory only (default) and in-memory and filesystem (optional).
+As described in [Buffering and storage](../administration/buffering-and-storage.md) , Fluent Bit offers two modes for data handling: in-memory only (default) and in-memory and filesystem (optional).
 
 The default `storage.type memory` buffer can be restricted with `Mem_Buf_Limit`. If memory reaches this limit and you reach a backpressure scenario, you won't be able to ingest more data until the data chunks that are in memory can be flushed. The input pauses and Fluent Bit [emits](https://github.com/fluent/fluent-bit/blob/v2.0.0/src/flb_input_chunk.c#L1334) a `[warn] [input] {input name or alias} paused (mem buf overlimit)` log message.
 
 Depending on the input plugin in use, this might cause incoming data to be discarded (for example, TCP input plugin). The tail plugin can handle pauses without data loss, storing its current file offset and resuming reading later. When buffer memory is available, the input resumes accepting logs. Fluent Bit [emits](https://github.com/fluent/fluent-bit/blob/v2.0.0/src/flb_input_chunk.c#L1277) a `[info] [input] {input name or alias} resume (mem buf overlimit)` message.
 
-Mitigate the risk of data loss by configuring secondary storage on the filesystem using the `storage.type` of `filesystem` (as described in [Buffering and Storage](buffering-and-storage.md)). Initially, logs will be buffered to both memory and the filesystem. When the `storage.max_chunks_up` limit is reached, all new data will be stored in the filesystem. Fluent Bit stops queueing new data in memory and buffers only to the filesystem. When `storage.type filesystem` is set, the `Mem_Buf_Limit` setting no longer has any effect. Instead, the `[SERVICE]` level `storage.max_chunks_up` setting controls the size of the memory buffer.
+Mitigate the risk of data loss by configuring secondary storage on the filesystem using the `storage.type` of `filesystem` (as described in [Buffering and storage](../administration/buffering-and-storage.md)). Initially, logs will be buffered to both memory and the filesystem. When the `storage.max_chunks_up` limit is reached, all new data will be stored in the filesystem. Fluent Bit stops queueing new data in memory and buffers only to the filesystem. When `storage.type filesystem` is set, the `Mem_Buf_Limit` setting no longer has any effect. Instead, the `[SERVICE]` level `storage.max_chunks_up` setting controls the size of the memory buffer.
 
 ## `Mem_Buf_Limit`
 
@@ -70,4 +70,4 @@ With `storage.type filesystem` and `storage.max_chunks_up`, the following log me
 ```text
 [input] {input name or alias} paused (storage buf overlimit)
 [input] {input name or alias} resume (storage buf overlimit)
-```
+```

concepts/data-pipeline.md

@@ -2,26 +2,39 @@
 
 The Fluent Bit data pipeline incorporates several specific concepts. Data processing flows through the pipeline following these concepts in order.
 
-## Filters
+```mermaid
+graph LR
+    accTitle: Fluent Bit data pipeline
+    accDescr: A diagram of the Fluent Bit data pipeline, which includes input, a parser, a filter, a buffer, routing, and various outputs.
+    A[Input] --> B[Parser]
+    B --> C[Filter]
+    C --> D[Buffer]
+    D --> E((Routing))
+    E --> F[Output 1]
+    E --> G[Output 2]
+    E --> H[Output 3]
+```
 
-[Filters](../pipeline/filters.md) let you alter the collected data before delivering it to a destination. In production environments you need full control of the data you're collecting. Using filters lets you control data before processing.
+## Inputs
 
-## Buffer
+[Input plugins](../pipeline/inputs.md) gather information from different sources. Some plugins collect data from log files, and others gather metrics information from the operating system. There are many plugins to suit different needs.
 
-The [`buffer`](./buffering.md) phase in the pipeline aims to provide a unified and persistent mechanism to store your data, using the primary in-memory model or the file system-based mode.
+## Parser
 
-## Inputs
-
-Fluent Bit provides [input plugins](../pipeline/inputs.md) to gather information from different sources. Some plugins collect data from log files, and others gather metrics information from the operating system. There are many plugins to suit different needs.
+[Parsers](../pipeline/parsers.md) convert unstructured data to structured data. Use a parser to set a structure to the incoming data by using input plugins as data is collected.
 
-## Outputs
+## Filter
 
-[Output plugins](../pipeline/outputs.md) let you define destinations for your data. Common destinations are remote services, local file systems, or other standard interfaces.
+[Filters](../pipeline/filters.md) let you alter the collected data before delivering it to a destination. In production environments you need full control of the data you're collecting. Using filters lets you control data before processing.
 
-## Parsers
+## Buffer
 
-[Parsers](../pipeline/parsers.md) convert unstructured data to structured data. Use a parser to set a structure to the incoming data by using input plugins as data is collected.
+The [buffering](./buffering.md) phase in the pipeline aims to provide a unified and persistent mechanism to store your data, using the primary in-memory model or the file system-based mode.
 
-## Route
+## Routing
 
 [Routing](../pipeline/router.md) is a core feature that lets you route your data through filters, and then to one or multiple destinations. The router relies on the concept of [tags](./key-concepts.md#tag) and [matching](./key-concepts.md#match) rules.
+
+## Output
+
+[Output plugins](../pipeline/outputs.md) let you define destinations for your data. Common destinations are remote services, local file systems, or other standard interfaces.

concepts/buffering.md renamed to pipeline/buffering.md

@@ -4,9 +4,9 @@ description: Performance and data safety
 
 # Buffering
 
-When [Fluent Bit](https://fluentbit.io) processes data, it uses the system memory (heap) as a primary and temporary place to store the record logs before they get delivered. The records are processed in this private memory area.
+When Fluent Bit processes data, it uses the system memory (heap) as a primary and temporary place to store the record logs before they get delivered. The records are processed in this private memory area.
 
-Buffering is the ability to store the records, and continue storing incoming data while previous data is processed and delivered. Buffering in memory is the fastest mechanism, but there are scenarios requiring special strategies to deal with [backpressure](../administration/backpressure.md), data safety, or to reduce memory consumption by the service in constrained environments.
+Buffering is the ability to temporarily store incoming data before that data is processed and delivered. Buffering in memory is the fastest mechanism, but there are scenarios requiring special strategies to deal with [backpressure](../administration/backpressure.md), data safety, or to reduce memory consumption by the service in constrained environments.
 
 ```mermaid
 graph LR

pipeline/router.md

@@ -2,7 +2,7 @@
 description: Create flexible routing rules
 ---
 
-# Router
+# Routing
 
 Routing is a core feature that lets you route your data through filters and then to one or multiple destinations. The router relies on the concept of [Tags](../concepts/key-concepts.md) and [Matching](../concepts/key-concepts.md) rules.