Merge branch 'master' into out-es-custom-http-headers

Signed-off-by: esmerel <[email protected]>

.gitignore

@@ -2,4 +2,4 @@ _book/*
 node_modules/*
 .gitignore
 old/*
-
+.DS_Store

.vale.ini

@@ -0,0 +1,17 @@
+StylesPath = vale-styles
+MinAlertLevel = suggestion # suggestion, warning or error
+
+[formats]
+mdx = md
+
+[*.{md,markdown,txt,htm,html}]
+BasedOnStyles = FluentBit
+IgnoredScopes = tt, code
+
+FluentBit.Terms = YES
+FluentBit.Titles = YES
+FluentBit.FutureTense = NO
+FluentBit.Headings = NO
+FluentBit.Passive = NO
+FluentBit.Subjunctive = NO
+FluentBit.Colons = NO

README.md

@@ -2,39 +2,55 @@
 description: High Performance Telemetry Agent for Logs, Metrics and Traces
 ---
 
-# Fluent Bit v3.0 Documentation
+# Fluent Bit v3.2 Documentation
 
-<figure><img src=".gitbook/assets/3.0.png" alt=""><figcaption></figcaption></figure>
+<figure><img src=".gitbook/assets/3.1.png" alt=""><figcaption></figcaption></figure>
 
-[Fluent Bit](http://fluentbit.io) is a Fast and Lightweight **Telemetry Agent** for Logs, Metrics, and Traces for Linux, macOS, Windows, and BSD family operating systems. It has been made with a strong focus on performance to allow the collection and processing of telemetry data from different sources without complexity.![](https://static.scarf.sh/a.png?x-pxid=71f0e011-761f-4c6f-9a89-38817887faae)
+[Fluent Bit](http://fluentbit.io) is a fast and lightweight telemetry agent
+for logs, metrics, and traces for Linux, macOS, Windows, and BSD family
+operating systems. Fluent Bit has been made with a strong focus on performance to allow
+the collection and processing of telemetry data from different sources without
+complexity.![](https://static.scarf.sh/a.png?x-pxid=71f0e011-761f-4c6f-9a89-38817887faae)
 
 ## Features
 
-* High Performance: High throughput with low resources consumption
-* Data Parsing
-  * Convert your unstructured messages using our parsers: [JSON](pipeline/parsers/json.md), [Regex](pipeline/parsers/regular-expression.md), [LTSV](pipeline/parsers/ltsv.md) and [Logfmt](pipeline/parsers/logfmt.md)
-* Metrics Support: Prometheus and OpenTelemetry compatible
-* Reliability and Data Integrity
-  * [Backpressure](administration/backpressure.md) Handling
-  * [Data Buffering](administration/buffering-and-storage.md) in memory and file system
-* Networking
-  * Security: built-in TLS/SSL support
-  * Asynchronous I/O
-* Pluggable Architecture and [Extensibility](development/library_api.md): Inputs, Filters and Outputs
-  * Connect nearly any source to nearly any destination using preexisting plugins
-  * Extensibility
-    * Write any input, filter or output plugin in C language
-    * WASM: [WASM Filter Plugins](development/wasm-filter-plugins.md) or [WASM Input Plugins](development/wasm-input-plugins.md)
-    * Bonus: write [Filters in Lua](pipeline/filters/lua.md) or [Output plugins in Golang](development/golang-output-plugins.md)
-* [Monitoring](administration/monitoring.md): expose internal metrics over HTTP in JSON and [Prometheus](https://prometheus.io/) format
-* [Stream Processing](stream-processing/introduction.md): Perform data selection and transformation using simple SQL queries
-  * Create new streams of data using query results
-  * Aggregation Windows
-  * Data analysis and prediction: Timeseries forecasting
-* Portable: runs on Linux, macOS, Windows and BSD systems
+- High performance: High throughput with low resources consumption
+- Data parsing
+  - Convert your unstructured messages using our parsers:
+    [JSON](pipeline/parsers/json.md),
+    [Regex](pipeline/parsers/regular-expression.md),
+    [LTSV](pipeline/parsers/ltsv.md) and [Logfmt](pipeline/parsers/logfmt.md)
+- Metrics support: Prometheus and OpenTelemetry compatible
+- Reliability and data integrity
+  - [Backpressure](administration/backpressure.md) handling
+  - [Data buffering](administration/buffering-and-storage.md) in memory and file system
+- Networking
+  - Security: Built-in TLS/SSL support
+  - Asynchronous I/O
+- Pluggable architecture and [extensibility](development/library_api.md): Inputs,
+  Filters and Outputs:
+  - Connect nearly any source to nearly any destination using preexisting plugins
+  - Extensibility:
+    - Write input, filter, or output plugins in the C language
+    - WASM: [WASM Filter Plugins](development/wasm-filter-plugins.md) or
+      [WASM Input Plugins](development/wasm-input-plugins.md)
+    - Write [Filters in Lua](pipeline/filters/lua.md) or
+      [Output plugins in Golang](development/golang-output-plugins.md)
+- [Monitoring](administration/monitoring.md): Expose internal metrics over HTTP
+  in JSON and [Prometheus](https://prometheus.io/) format
+- [Stream Processing](stream-processing/introduction.md): Perform data selection
+  and transformation using simple SQL queries
+  - Create new streams of data using query results
+  - Aggregation windows
+  - Data analysis and prediction: Timeseries forecasting
+- Portable: Runs on Linux, macOS, Windows and BSD systems
 
 ## Fluent Bit, Fluentd and CNCF
 
-[Fluent Bit](http://fluentbit.io) is a [CNCF](https://cncf.io) **graduated** sub-project under the umbrella of [Fluentd](http://fluentd.org). Fluent Bit is licensed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).
+[Fluent Bit](http://fluentbit.io) is a [CNCF](https://cncf.io) graduated sub-project
+under the umbrella of [Fluentd](http://fluentd.org). Fluent Bit is licensed under
+the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).
 
-Fluent Bit was originally created by [Eduardo Silva](https://www.linkedin.com/in/edsiper/). As a CNCF-hosted project, it is a fully **vendor-neutral** and community-driven project.
+Fluent Bit was originally created by [Eduardo Silva](https://www.linkedin.com/in/edsiper/)
+and is now sponsored by [Chronosphere](https://chronosphere.io/). As a
+CNCF-hosted project, it is a fully vendor-neutral and community-driven project.

SUMMARY.md

@@ -1,6 +1,6 @@
 # Table of contents
 
-* [Fluent Bit v3.0 Documentation](README.md)
+* [Fluent Bit v3.2 Documentation](README.md)
 
 ## About
 
@@ -67,6 +67,7 @@
 * [Networking](administration/networking.md)
 * [Memory Management](administration/memory-management.md)
 * [Monitoring](administration/monitoring.md)
+* [Multithreading](administration/multithreading.md)
 * [HTTP Proxy](administration/http-proxy.md)
 * [Hot Reload](administration/hot-reload.md)
 * [Troubleshooting](administration/troubleshooting.md)
@@ -131,7 +132,9 @@
   * [Decoders](pipeline/parsers/decoders.md)
 * [Processors](pipeline/processors/README.md)
   * [Content Modifier](pipeline/processors/content-modifier.md)
+  * [Labels](pipeline/processors/labels.md)
   * [Metrics Selector](pipeline/processors/metrics-selector.md)
+  * [OpenTelemetry Envelope](pipeline/processors/opentelemetry-envelope.md)
   * [SQL](pipeline/processors/sql.md)
 * [Filters](pipeline/filters/README.md)
   * [AWS Metadata](pipeline/filters/aws-metadata.md)
@@ -184,6 +187,7 @@
   * [NATS](pipeline/outputs/nats.md)
   * [New Relic](pipeline/outputs/new-relic.md)
   * [NULL](pipeline/outputs/null.md)
+  * [OpenObserve](pipeline/inputs/openobserve.md)
   * [Observe](pipeline/outputs/observe.md)
   * [Oracle Log Analytics](pipeline/outputs/oci-logging-analytics.md)
   * [OpenSearch](pipeline/outputs/opensearch.md)

about/fluentd-and-fluent-bit.md

@@ -1,33 +1,45 @@
 ---
-description: The Production Grade Telemetry Ecosystem
+description: The production grade telemetry ecosystem
 ---
 
-# Fluentd & Fluent Bit
+# Fluentd and Fluent Bit
 
-Telemetry data processing in general can be complex, and at scale a bit more, that's why [Fluentd](https://www.fluentd.org) was born. Fluentd has become more than a simple tool, it has grown into a fullscale ecosystem that contains SDKs for different languages and sub-projects like [Fluent Bit](https://fluentbit.io).
+Telemetry data processing can be complex, especially at scale. That's why
+[Fluentd](https://www.fluentd.org) was created. Fluentd is more than a simple tool,
+it's grown into a fullscale ecosystem that contains SDKs for different languages
+and subprojects like [Fluent Bit](https://fluentbit.io).
 
-On this page, we will describe the relationship between the [Fluentd](http://fluentd.org) and [Fluent Bit](http://fluentbit.io) open source projects, as a summary we can say both are:
+Here, we describe the relationship between the [Fluentd](http://fluentd.org)
+and [Fluent Bit](http://fluentbit.io) open source projects.
 
-* Licensed under the terms of Apache License v2.0
-* **Graduated** Hosted projects by the [Cloud Native Computing Foundation (CNCF)](https://cncf.io)
-* Production Grade solutions: deployed **million** of times every single day.
-* **Vendor neutral** and community driven projects
-* Widely Adopted by the Industry: trusted by all major companies like AWS, Microsoft, Google Cloud and hundreds of others.
+Both projects are:
 
-Both projects share a lot of similarities, [Fluent Bit](https://fluentbit.io) is fully designed and built on top of the best ideas of [Fluentd](https://www.fluentd.org) architecture and general design. Choosing which one to use depends on the end-user needs.
+- Licensed under the terms of Apache License v2.0.
+- Graduated hosted projects by the [Cloud Native Computing Foundation (CNCF)](https://cncf.io).
+- Production grade solutions: Deployed millions of times every single day.
+- Vendor neutral and community driven.
+- Widely adopted by the industry: Trusted by major companies like AWS, Microsoft,
+  Google Cloud, and hundreds of others.
+
+The projects have many similarities: [Fluent Bit](https://fluentbit.io) is
+designed and built on top of the best ideas of [Fluentd](https://www.fluentd.org)
+architecture and general design. Which one you choose depends on your end-users' needs.
 
 The following table describes a comparison of different areas of the projects:
 
-|              | Fluentd                                                           | Fluent Bit                                                        |
-| ------------ | ----------------------------------------------------------------- | ----------------------------------------------------------------- |
-| Scope        | Containers / Servers                                              | Embedded Linux / Containers / Servers                             |
-| Language     | C & Ruby                                                          | C                                                                 |
-| Memory       |  > 60MB                                                           | \~1MB                                                             |
-| Performance  | Medium Performance                                                | High Performance                                                  |
-| Dependencies | Built as a Ruby Gem, it requires a certain number of gems.        | Zero dependencies, unless some special plugin requires them.      |
-| Plugins      | More than 1000 external plugins are available                     | More than 100 built-in plugins are available                      |
+|  Attribute   | Fluentd               | Fluent Bit            |
+| ------------ | --------------------- | --------------------- |
+| Scope        | Containers / Servers  | Embedded Linux / Containers / Servers |
+| Language     | C & Ruby              | C                                     |
+| Memory       | Greater than 60 MB | Approximately 1 MB |
+| Performance  | Medium Performance    | High Performance                      |
+| Dependencies | Built as a Ruby Gem, depends on other gems. | Zero dependencies, unless required by a plugin. |
+| Plugins      | Over 1,000 external plugins available. | Over 100 built-in plugins available. |
 | License      | [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0) | [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0) |
 
-Both [Fluentd](https://www.fluentd.org) and [Fluent Bit](https://fluentbit.io) can work as Aggregators or Forwarders, they both can complement each other or use them as standalone solutions.\
-\
-In the recent years, Cloud Providers switched from Fluentd to Fluent Bit for performance and compatibility reasons. Fluent Bit is now considered the **next generation** solution.
+Both [Fluentd](https://www.fluentd.org) and [Fluent Bit](https://fluentbit.io)
+can work as Aggregators or Forwarders, and can complement each other or be used
+as standalone solutions.
+
+In the recent years, cloud providers have switched from Fluentd to Fluent Bit for
+performance and compatibility. Fluent Bit is now considered the next-generation solution.

about/history.md

@@ -1,10 +1,20 @@
 ---
+title: A brief history of Fluent Bit
 description: Every project has a story
 ---
 
-# A Brief History of Fluent Bit
+# A brief history of Fluent Bit
 
-On 2014, the [Fluentd](https://www.fluentd.org/) team at [Treasure Data](https://www.treasuredata.com/) was forecasting the need for a lightweight log processor for constraint environments like Embedded Linux and Gateways, the project aimed to be part of the Fluentd Ecosystem; at that moment, Eduardo created [Fluent Bit](https://fluentbit.io/), a new open source solution written from scratch available under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).\
+In 2014, the [Fluentd](https://www.fluentd.org/) team at
+[Treasure Data](https://www.treasuredata.com/) was forecasting the need for a
+lightweight log processor for constraint environments like embedded Linux and
+gateways. The project aimed to be part of the Fluentd ecosystem. At that moment,
+Eduardo Silva created [Fluent Bit](https://fluentbit.io/), a new open source solution,
+written from scratch and available under the terms of the
+[Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).
 
-
-After the project was around for some time, it got more traction for normal Linux systems, also with the new containerized world, the Cloud Native community asked to extend the project scope to support more sources, filters, and destinations. Not so long after, Fluent Bit became one of the preferred solutions to solve the logging challenges in Cloud environments.
+After the project matured, it gained traction for normal Linux systems. With the
+new containerized world, the Cloud Native community asked to extend the 
+project scope to support more sources, filters, and destinations. Not long after,
+Fluent Bit became one of the preferred solutions to solve the logging challenges
+in Cloud environments.

about/license.md

@@ -1,10 +1,13 @@
 ---
-description: Strong Commitment to the Openness and Collaboration
+title: License
+description: Fluent Bit license description
 ---
 
 # License
 
-[Fluent Bit](http://fluentbit.io), including its core, plugins and tools are distributed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0):
+[Fluent Bit](http://fluentbit.io), including its core, plugins, and tools are
+distributed under the terms of the
+[Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0):
 
 ```text
                                  Apache License
@@ -184,4 +187,3 @@ description: Strong Commitment to the Openness and Collaboration
 
    END OF TERMS AND CONDITIONS
 ```
-

about/what-is-fluent-bit.md

@@ -4,14 +4,24 @@ description: Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
 
 # What is Fluent Bit?
 
+[Fluent Bit](https://fluentbit.io) is an open source telemetry agent specifically
+designed to efficiently handle the challenges of collecting and processing telemetry
+data across a wide range of environments, from constrained systems to complex cloud
+infrastructures. Managing telemetry data from various sources and formats can be a
+constant challenge, particularly when performance is a critical factor.
+
+Rather than serving as a drop-in replacement, Fluent Bit enhances the observability
+strategy for your infrastructure by adapting and optimizing your existing logging
+layer, and adding metrics and traces processing. Fluent Bit supports a
+vendor-neutral approach, seamlessly integrating with other ecosystems such as
+Prometheus and OpenTelemetry. Trusted by major cloud providers, banks, and companies
+in need of a ready-to-use telemetry agent solution, Fluent Bit effectively manages
+diverse data sources and formats while maintaining optimal performance and keeping
+resource consumption low.
+
+Fluent Bit can be deployed as an edge agent for localized telemetry data handling or
+utilized as a central aggregator/collector for managing telemetry data across
+multiple sources and environments.
 
 
-[Fluent Bit](https://fluentbit.io) is an open-source telemetry agent specifically designed to efficiently handle the challenges of collecting and processing telemetry data across a wide range of environments, from constrained systems to complex cloud infrastructures. Managing telemetry data from various sources and formats can be a constant challenge, particularly when performance is a critical factor.
-
-Rather than serving as a drop-in replacement, Fluent Bit enhances the observability strategy for your infrastructure by adapting and optimizing your existing logging layer, as well as metrics and traces processing. Furthermore, Fluent Bit supports a vendor-neutral approach, seamlessly integrating with other ecosystems such as Prometheus and OpenTelemetry. Trusted by major cloud providers, banks, and companies in need of a ready-to-use telemetry agent solution, Fluent Bit effectively manages diverse data sources and formats while maintaining optimal performance.
-
-Fluent Bit can be deployed as an edge agent for localized telemetry data handling or utilized as a central aggregator/collector for managing telemetry data across multiple sources and environments.
-
-[Fluent Bit](https://fluentbit.io) has been designed with performance and low resource consumption in mind.
-
 {% embed url="https://www.youtube.com/watch?v=3ELc1helke4" %}

administration/backpressure.md

@@ -1,5 +1,7 @@
 # Backpressure
 
+<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=63e37cfe-9ce3-4a18-933a-76b9198958c1" />
+
 Under certain scenarios it is possible for logs or data to be ingested or created faster than the ability to flush it to some destinations. One such common scenario is when reading from big log files, especially with a large backlog, and dispatching the logs to a backend over the network, which takes time to respond. This generates backpressure leading to high memory consumption in the service.
 
 In order to avoid backpressure, Fluent Bit implements a mechanism in the engine that restricts the amount of data that an input plugin can ingest, this is done through the configuration parameters **Mem\_Buf\_Limit** and **storage.Max\_Chunks\_Up**.

administration/buffering-and-storage.md

@@ -1,5 +1,7 @@
 # Buffering & Storage
 
+<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=cde12327-09ed-409c-ac02-7c0afa5eff51" />
+
 The end-goal of [Fluent Bit](https://fluentbit.io) is to collect, parse, filter and ship logs to a central place. In this workflow there are many phases and one of the critical pieces is the ability to do _buffering_ : a mechanism to place processed data into a temporary location until is ready to be shipped.
 
 By default when Fluent Bit processes data, it uses Memory as a primary and temporary place to store the records, but there are certain scenarios where it would be ideal to have a persistent buffering mechanism based in the filesystem to provide aggregation and data safety capabilities.

administration/configuring-fluent-bit/README.md

@@ -1,13 +1,14 @@
 # Configuring Fluent Bit
 
-Currently, Fluent Bit supports two configuration formats:
+Fluent Bit supports these configuration formats:
 
-* [Classic mode](classic-mode/README.md).
-* [Yaml](yaml/README.md). (YAML configuration is production ready since Fluent Bit 2.0.)
+- [Classic mode](classic-mode/README.md)
+- [YAML](yaml/README.md) (Fluent Bit 2.0 or greater)
 
 ## CLI flags
 
-Fluent Bit also supports a CLI interface with various flags matching up to the configuration options available.
+Fluent Bit also supports a CLI with various flags for the available configuration
+options.
 
 ```shell
 $ docker run --rm -it fluent/fluent-bit --help