+
Under certain scenarios it is possible for logs or data to be ingested or created faster than the ability to flush it to some destinations. One such common scenario is when reading from big log files, especially with a large backlog, and dispatching the logs to a backend over the network, which takes time to respond. This generates backpressure leading to high memory consumption in the service.
In order to avoid backpressure, Fluent Bit implements a mechanism in the engine that restricts the amount of data that an input plugin can ingest, this is done through the configuration parameters **Mem\_Buf\_Limit** and **storage.Max\_Chunks\_Up**.
diff --git a/administration/buffering-and-storage.md b/administration/buffering-and-storage.md
index 1b373bf64..a022208d7 100644
--- a/administration/buffering-and-storage.md
+++ b/administration/buffering-and-storage.md
@@ -1,5 +1,7 @@
# Buffering & Storage
+
+
The end-goal of [Fluent Bit](https://fluentbit.io) is to collect, parse, filter and ship logs to a central place. In this workflow there are many phases and one of the critical pieces is the ability to do _buffering_ : a mechanism to place processed data into a temporary location until is ready to be shipped.
By default when Fluent Bit processes data, it uses Memory as a primary and temporary place to store the records, but there are certain scenarios where it would be ideal to have a persistent buffering mechanism based in the filesystem to provide aggregation and data safety capabilities.
diff --git a/administration/configuring-fluent-bit/README.md b/administration/configuring-fluent-bit/README.md
index e1a73a507..794d41950 100644
--- a/administration/configuring-fluent-bit/README.md
+++ b/administration/configuring-fluent-bit/README.md
@@ -1,13 +1,14 @@
# Configuring Fluent Bit
-Currently, Fluent Bit supports two configuration formats:
+Fluent Bit supports these configuration formats:
-* [Classic mode](classic-mode/README.md).
-* [Yaml](yaml/README.md). (YAML configuration is production ready since Fluent Bit 2.0.)
+- [Classic mode](classic-mode/README.md)
+- [YAML](yaml/README.md) (Fluent Bit 2.0 or greater)
## CLI flags
-Fluent Bit also supports a CLI interface with various flags matching up to the configuration options available.
+Fluent Bit also supports a CLI with various flags for the available configuration
+options.
```shell
$ docker run --rm -it fluent/fluent-bit --help
diff --git a/administration/configuring-fluent-bit/classic-mode/configuration-file.md b/administration/configuring-fluent-bit/classic-mode/configuration-file.md
index ce423fa65..34c8cbfc4 100644
--- a/administration/configuring-fluent-bit/classic-mode/configuration-file.md
+++ b/administration/configuring-fluent-bit/classic-mode/configuration-file.md
@@ -4,6 +4,8 @@ description: This page describes the main configuration file used by Fluent Bit
# Configuration File
+
+
One of the ways to configure Fluent Bit is using a main configuration file. Fluent Bit allows to use one configuration file which works at a global scope and uses the [Format and Schema](format-schema.md) defined previously.
The main configuration file supports four types of sections:
@@ -137,12 +139,6 @@ The following configuration file example demonstrates how to collect CPU metrics
Match my*cpu
```
-## Visualize
-
-You can also visualize Fluent Bit INPUT, FILTER, and OUTPUT configuration via [Calyptia](https://calyptia.com/free-trial)
-
-
-
## Include File
To avoid complicated long configuration files is better to split specific parts in different files and call them (include) from one main file.
@@ -167,5 +163,5 @@ Wildcard character (\*) is supported to include multiple files, e.g:
@INCLUDE input_*.conf
```
-Note files matching the wildcard character are included unsorted.
+Note files matching the wildcard character are included unsorted.
If plugins ordering between files need to be preserved, the files should be included explicitly.
diff --git a/administration/configuring-fluent-bit/classic-mode/variables.md b/administration/configuring-fluent-bit/classic-mode/variables.md
index 11e32a56b..1344e4f4c 100644
--- a/administration/configuring-fluent-bit/classic-mode/variables.md
+++ b/administration/configuring-fluent-bit/classic-mode/variables.md
@@ -1,5 +1,7 @@
# Variables
+
+
Fluent Bit supports the usage of environment variables in any value associated to a key when using a configuration file.
The variables are case sensitive and can be used in the following format:
diff --git a/administration/configuring-fluent-bit/multiline-parsing.md b/administration/configuring-fluent-bit/multiline-parsing.md
index f6fa1e379..d7965c9ee 100644
--- a/administration/configuring-fluent-bit/multiline-parsing.md
+++ b/administration/configuring-fluent-bit/multiline-parsing.md
@@ -2,6 +2,8 @@
In an ideal world, applications might log their messages within a single line, but in reality applications generate multiple log messages that sometimes belong to the same context. But when is time to process such information it gets really complex. Consider application stack traces which always have multiple log lines.
+
+
Starting from Fluent Bit v1.8, we have implemented a unified Multiline core functionality to solve all the user corner cases. In this section, you will learn about the features and configuration options available.
## Concepts
diff --git a/administration/configuring-fluent-bit/yaml/configuration-file.md b/administration/configuring-fluent-bit/yaml/configuration-file.md
index 0205ccc5f..87e062799 100644
--- a/administration/configuring-fluent-bit/yaml/configuration-file.md
+++ b/administration/configuring-fluent-bit/yaml/configuration-file.md
@@ -2,6 +2,8 @@
description: This page describes the yaml configuration file used by Fluent Bit
---
+
+
# YAML Configuration File
One of the ways to configure Fluent Bit is using a YAML configuration file that works at a global scope.
diff --git a/administration/memory-management.md b/administration/memory-management.md
index dd8d33f4c..afd6e1c71 100644
--- a/administration/memory-management.md
+++ b/administration/memory-management.md
@@ -1,5 +1,7 @@
# Memory Management
+
+
In certain scenarios it would be ideal to estimate how much memory Fluent Bit could be using, this is very useful for containerized environments where memory limits are a must.
In order to that we will assume that the input plugins have set the **Mem\_Buf\_Limit** option \(you can learn more about it in the [Backpressure](backpressure.md) section\).
diff --git a/administration/monitoring.md b/administration/monitoring.md
index 0d6fae6dc..e515190a4 100644
--- a/administration/monitoring.md
+++ b/administration/monitoring.md
@@ -1,27 +1,35 @@
---
+title: Monitor data pipelines
description: Learn how to monitor your Fluent Bit data pipelines
---
-# Monitoring
+
-Fluent Bit comes with built-it features to allow you to monitor the internals of your pipeline, connect to Prometheus and Grafana, Health checks and also connectors to use external services for such purposes:
+# Monitor data pipelines
-* [HTTP Server: JSON and Prometheus Exporter-style metrics](monitoring.md#http-server)
-* [Grafana Dashboards and Alerts](monitoring.md#grafana-dashboard-and-alerts)
-* [Health Checks](monitoring.md#health-check-for-fluent-bit)
-* [Calyptia Cloud: hosted service to monitor and visualize your pipelines](monitoring.md#calyptia-cloud)
+Fluent Bit includes features for monitoring the internals of your pipeline, in
+addition to connecting to Prometheus and Grafana, Health checks, and connectors to
+use external services:
-## HTTP Server
+- [HTTP Server: JSON and Prometheus Exporter-style metrics](monitoring.md#http-server)
+- [Grafana Dashboards and Alerts](monitoring.md#grafana-dashboard-and-alerts)
+- [Health Checks](monitoring.md#health-check-for-fluent-bit)
+- [Calyptia Cloud: hosted service to monitor and visualize your pipelines](monitoring.md#calyptia-cloud)
-Fluent Bit comes with a built-in HTTP Server that can be used to query internal information and monitor metrics of each running plugin.
+## HTTP server
-The monitoring interface can be easily integrated with Prometheus since we support it native format.
+Fluent Bit includes an HTTP server for querying internal information and monitoring
+metrics of each running plugin.
-### Getting Started
+You can integrate the monitoring interface with Prometheus.
-To get started, the first step is to enable the HTTP Server from the configuration file:
+### Getting started
-```
+To get started, enable the HTTP server from the configuration file. The following
+configuration instructs Fluent Bit to start an HTTP server on TCP port `2020` and
+listen on all network interfaces:
+
+```yaml
[SERVICE]
HTTP_Server On
HTTP_Listen 0.0.0.0
@@ -35,10 +43,15 @@ To get started, the first step is to enable the HTTP Server from the configurati
Match *
```
-the above configuration snippet will instruct Fluent Bit to start it HTTP Server on TCP Port 2020 and listening on all network interfaces:
+Apply the configuration file:
+```shell
+bin/fluent-bit -c fluent-bit.conf
```
-$ bin/fluent-bit -c fluent-bit.conf
+
+Fluent Bit starts and generates output in your terminal:
+
+```shell
Fluent Bit v1.4.0
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
@@ -49,10 +62,12 @@ Fluent Bit v1.4.0
[2020/03/10 19:08:24] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
```
-now with a simple **curl** command is enough to gather some information:
+Use `curl` to gather information about the HTTP server. The following command sends
+the command output to the `jq` program, which outputs human-readable JSON data to the
+terminal.
-```
-$ curl -s http://127.0.0.1:2020 | jq
+```curl
+curl -s http://127.0.0.1:2020 | jq
{
"fluent-bit": {
"version": "0.13.0",
@@ -78,132 +93,154 @@ $ curl -s http://127.0.0.1:2020 | jq
}
```
-Note that we are sending the _curl_ command output to the _jq_ program which helps to make the JSON data easy to read from the terminal. Fluent Bit doesn't aim to do JSON pretty-printing.
-
-### REST API Interface
-
-Fluent Bit aims to expose useful interfaces for monitoring, as of Fluent Bit v0.14 the following end points are available:
-
-| URI | Description | Data Format |
-| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------- |
-| / | Fluent Bit build information | JSON |
-| /api/v1/uptime | Get uptime information in seconds and human readable format | JSON |
-| /api/v1/metrics | Internal metrics per loaded plugin | JSON |
-| /api/v1/metrics/prometheus | Internal metrics per loaded plugin ready to be consumed by a Prometheus Server | Prometheus Text 0.0.4 |
-| /api/v1/storage | Get internal metrics of the storage layer / buffered data. This option is enabled only if in the `SERVICE` section the property `storage.metrics` has been enabled | JSON |
-| /api/v1/health | Fluent Bit health check result | String |
-| /api/v2/metrics | Internal metrics per loaded plugin | [cmetrics text format](https://github.com/fluent/cmetrics) |
-| /api/v2/metrics/prometheus | Internal metrics per loaded plugin ready to be consumed by a Prometheus Server | Prometheus Text 0.0.4 |
-| /api/v2/reload | Execute hot reloading or get the status of hot reloading. For more details, please refer to the [hot-reloading documentation](hot-reload.md). | JSON |
-
-### Metric Descriptions
-
-#### For v1 metrics
-
-The following are detailed descriptions for the metrics outputted in prometheus format by `/api/v1/metrics/prometheus`.
-
-The following definitions are key to understand:
-* record: a single message collected from a source, such as a single long line in a file.
-* chunk: Fluent Bit input plugin instances ingest log records and store them in chunks. A batch of records in a chunk are tracked together as a single unit; the Fluent Bit engine attempts to fit records into chunks of at most 2 MB, but the size can vary at runtime. Chunks are then sent to an output. An output plugin instance can either successfully send the full chunk to the destination and mark it as successful, or it can fail the chunk entirely if an unrecoverable error is encountered, or it can ask for the chunk to be retried.
-
-| Metric Name | Labels | Description | Type | Unit |
-|----------------------------------------|-------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|---------|
-| fluentbit_input_bytes_total | name: the name or alias for the input instance | The number of bytes of log records that this input instance has successfully ingested | counter | bytes |
-| fluentbit_input_records_total | name: the name or alias for the input instance | The number of log records this input has successfully ingested | counter | records |
-| fluentbit_output_dropped_records_total | name: the name or alias for the output instance | The number of log records that have been dropped by the output. This means they met an unrecoverable error or retries expired for their chunk. | counter | records |
-| fluentbit_output_errors_total | name: the name or alias for the output instance | The number of chunks that have faced an error (either unrecoverable or retriable). This is the number of times a chunk has failed, and does not correspond with the number of error messages you see in the Fluent Bit log output. | counter | chunks |
-| fluentbit_output_proc_bytes_total | name: the name or alias for the output instance | The number of bytes of log records that this output instance has *successfully* sent. This is the total byte size of all unique chunks sent by this output. If a record is not sent due to some error, then it will not count towards this metric. | counter | bytes |
-| fluentbit_output_proc_records_total | name: the name or alias for the output instance | The number of log records that this output instance has *successfully* sent. This is the total record count of all unique chunks sent by this output. If a record is not successfully sent, it does not count towards this metric. | counter | records |
-| fluentbit_output_retried_records_total | name: the name or alias for the output instance | The number of log records that experienced a retry. Note that this is calculated at the chunk level, the count increased when an entire chunk is marked for retry. An output plugin may or may not perform multiple actions that generate many error messages when uploading a single chunk. | counter | records |
-| fluentbit_output_retries_failed_total | name: the name or alias for the output instance | The number of times that retries expired for a chunk. Each plugin configures a Retry_Limit which applies to chunks. Once the Retry_Limit has been reached for a chunk it is discarded and this metric is incremented. | counter | chunks |
-| fluentbit_output_retries_total | name: the name or alias for the output instance | The number of times this output instance requested a retry for a chunk. | counter | chunks |
-| fluentbit_uptime | | The number of seconds that Fluent Bit has been running. | counter | seconds |
-| process_start_time_seconds | | The Unix Epoch time stamp for when Fluent Bit started. | gauge | seconds |
-
-
-The following are detailed descriptions for the metrics outputted in JSON format by `/api/v1/storage`.
-
-
-| Metric Key | Description | Unit |
-|---------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
-| chunks.total_chunks | The total number of chunks of records that Fluent Bit is currently buffering | chunks |
-| chunks.mem_chunks | The total number of chunks that are buffered in memory at this time. Note that chunks can be both in memory and on the file system at the same time. | chunks |
-| chunks.fs_chunks | The total number of chunks saved to the filesystem. | chunks |
-| chunks.fs_chunks_up | A chunk is "up" if it is in memory. So this is the count of chunks that are both in filesystem and in memory. | chunks |
-| chunks.fs_chunks_down | The count of chunks that are "down" and thus are only in the filesystem. | chunks |
-| | | |
-| input_chunks.{plugin name}.status.overlimit | Is this input instance over its configured Mem_Buf_Limit? | boolean |
-| input_chunks.{plugin name}.status.mem_size | The size of memory that this input is consuming to buffer logs in chunks. | bytes |
-| input_chunks.{plugin name}.status.mem_limit | The buffer memory limit (Mem_Buf_Limit) that applies to this input plugin. | bytes |
-| | | |
-| input_chunks.{plugin name}.chunks.total | The current total number of chunks owned by this input instance. | chunks |
-| input_chunks.{plugin name}.chunks.up | The current number of chunks that are "up" in memory for this input. Chunks that are "up" will also be in the filesystem layer as well if filesystem storage is enabled. | chunks |
-| input_chunks.{plugin name}.chunks.down | The current number of chunks that are "down" in the filesystem for this input. | chunks |
-| input_chunks.{plugin name}.chunks.busy | "Busy" chunks are chunks that are being processed/sent by outputs and are not eligible to have new data appended. | chunks |
-| input_chunks.{plugin name}.chunks.busy_size | The sum of the byte size of each chunk which is currently marked as busy. | bytes |
-
-#### For v2 metrics
-
-The following are detailed descriptions for the metrics outputted in prometheus format by `/api/v2/metrics/prometheus` or `/api/v2/metrics`.
-
-The following definitions are key to understand:
-* record: a single message collected from a source, such as a single long line in a file.
-* chunk: Fluent Bit input plugin instances ingest log records and store them in chunks. A batch of records in a chunk are tracked together as a single unit; the Fluent Bit engine attempts to fit records into chunks of at most 2 MB, but the size can vary at runtime. Chunks are then sent to an output. An output plugin instance can either successfully send the full chunk to the destination and mark it as successful, or it can fail the chunk entirely if an unrecoverable error is encountered, or it can ask for the chunk to be retried.
-
-| Metric Name | Labels | Description | Type | Unit |
-|--------------------------------------------|-------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|---------|
-| fluentbit\_input\_bytes\_total | name: the name or alias for the input instance | The number of bytes of log records that this input instance has successfully ingested | counter | bytes |
-| fluentbit\_input\_records\_total | name: the name or alias for the input instance | The number of log records this input has successfully ingested | counter | records |
-| fluentbit\_filter\_bytes\_total | name: the name or alias for the filter instance | The number of bytes of log records that this filter instance has successfully ingested | counter | bytes |
-| fluentbit\_filter\_records\_total | name: the name or alias for the filter instance | The number of log records this filter has successfully ingested | counter | records |
-| fluentbit\_filter\_added\_records\_total | name: the name or alias for the filter instance | The number of log records that have been added by the filter. This means they added into the data pipeline. | counter | records |
-| fluentbit\_filter\_dropped\_records\_total | name: the name or alias for the filter instance | The number of log records that have been dropped by the filter. This means they removed from the data pipeline. | counter | records |
-| fluentbit\_output\_dropped\_records\_total | name: the name or alias for the output instance | The number of log records that have been dropped by the output. This means they met an unrecoverable error or retries expired for their chunk. | counter | records |
-| fluentbit\_output\_errors\_total | name: the name or alias for the output instance | The number of chunks that have faced an error (either unrecoverable or retriable). This is the number of times a chunk has failed, and does not correspond with the number of error messages you see in the Fluent Bit log output. | counter | chunks |
-| fluentbit\_output\_proc\_bytes\_total | name: the name or alias for the output instance | The number of bytes of log records that this output instance has *successfully* sent. This is the total byte size of all unique chunks sent by this output. If a record is not sent due to some error, then it will not count towards this metric. | counter | bytes |
-| fluentbit\_output\_proc\_records\_total | name: the name or alias for the output instance | The number of log records that this output instance has *successfully* sent. This is the total record count of all unique chunks sent by this output. If a record is not successfully sent, it does not count towards this metric. | counter | records |
-| fluentbit\_output\_retried\_records\_total | name: the name or alias for the output instance | The number of log records that experienced a retry. Note that this is calculated at the chunk level, the count increased when an entire chunk is marked for retry. An output plugin may or may not perform multiple actions that generate many error messages when uploading a single chunk. | counter | records |
-| fluentbit\_output\_retries\_failed\_total | name: the name or alias for the output instance | The number of times that retries expired for a chunk. Each plugin configures a Retry\_Limit which applies to chunks. Once the Retry\_Limit has been reached for a chunk it is discarded and this metric is incremented. | counter | chunks |
-| fluentbit\_output\_retries\_total | name: the name or alias for the output instance | The number of times this output instance requested a retry for a chunk. | counter | chunks |
-| fluentbit\_uptime | hostname: the hostname on running fluent-bit | The number of seconds that Fluent Bit has been running. | counter | seconds |
-| fluentbit\_process\_start\_time\_seconds | hostname: the hostname on running fluent-bit | The Unix Epoch time stamp for when Fluent Bit started. | gauge | seconds |
-| fluentbit\_build\_info | hostname: the hostname, version: the version of fluent-bit, os: OS type | Build version information. The returned value is originated from initializing the Unix Epoch time stamp of config context. | gauge | seconds |
-| fluentbit\_hot\_reloaded\_times | hostname: the hostname on running fluent-bit | Collect the count of hot reloaded times. | gauge | seconds |
-
-The following are detailed descriptions for the metrics which is collected by storage layer.
-
-
-| Metric Name | Labels | Description | Type | Unit |
-|-------------------------------------------------|-------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|---------|
-| fluentbit\_input\_chunks.storage\_chunks | None | The total number of chunks of records that Fluent Bit is currently buffering | gauge | chunks |
-| fluentbit\_storage\_mem\_chunk | None | The total number of chunks that are buffered in memory at this time. Note that chunks can be both in memory and on the file system at the same time. | gauge | chunks |
-| fluentbit\_storage\_fs\_chunks | None | The total number of chunks saved to the filesystem. | gauge | chunks |
-| fluentbit\_storage\_fs\_chunks\_up | None | A chunk is "up" if it is in memory. So this is the count of chunks that are both in filesystem and in memory. | gauge | chunks |
-| fluentbit\_storage\_fs\_chunks\_down | None | The count of chunks that are "down" and thus are only in the filesystem. | gauge | chunks |
-| fluentbit\_storage\_fs\_chunks\_busy | None | The total number of chunks are in a busy state. | gauge | chunks |
-| fluentbit\_storage\_fs\_chunks\_busy\_bytes | None | The total bytes of chunks are in a busy state. | gauge | bytes |
-| | | | | |
-| fluentbit\_input\_storage\_overlimit | name: the name or alias for the input instance | Is this input instance over its configured Mem\_Buf\_Limit? | gauge | boolean |
-| fluentbit\_input\_storage\_memory\_bytes | name: the name or alias for the input instance | The size of memory that this input is consuming to buffer logs in chunks. | gauge | bytes |
-| | | | | |
-| fluentbit\_input\_storage\_chunks | name: the name or alias for the input instance | The current total number of chunks owned by this input instance. | gauge | chunks |
-| fluentbit\_input\_storage\_chunks\_up | name: the name or alias for the input instance | The current number of chunks that are "up" in memory for this input. Chunks that are "up" will also be in the filesystem layer as well if filesystem storage is enabled. | gauge | chunks |
-| fluentbit\_input\_storage\_chunks\_down | name: the name or alias for the input instance | The current number of chunks that are "down" in the filesystem for this input. | gauge | chunks |
-| fluentbit\_input\_storage\_chunks\_busy | name: the name or alias for the input instance | "Busy" chunks are chunks that are being processed/sent by outputs and are not eligible to have new data appended. | gauge | chunks |
-| fluentbit\_input\_storage\_chunks\_busy\_bytes | name: the name or alias for the input instance | The sum of the byte size of each chunk which is currently marked as busy. | gauge | bytes |
-| | | | | |
-| fluentbit\_output\_upstream\_total\_connections | name: the name or alias for the output instance | The sum of the connection count of each output plugins. | gauge | bytes |
-| fluentbit\_output\_upstream\_busy\_connections | name: the name or alias for the output instance | The sum of the connection count in a busy state of each output plugins. | gauge | bytes |
-
-### Uptime Example
+### REST API interface
+
+Fluent Bit exposes the following endpoints for monitoring.
+
+| URI | Description | Data format |
+| -------------------------- | ------------- | --------------------- |
+| / | Fluent Bit build information. | JSON |
+| /api/v1/uptime | Return uptime information in seconds. | JSON |
+| /api/v1/metrics | Display internal metrics per loaded plugin. | JSON |
+| /api/v1/metrics/prometheus | Display internal metrics per loaded plugin in Prometheus Server format. | Prometheus Text 0.0.4 |
+| /api/v1/storage | Get internal metrics of the storage layer / buffered data. This option is enabled only if in the `SERVICE` section of the property `storage.metrics` is enabled. | JSON |
+| /api/v1/health | Display the Fluent Bit health check result. | String |
+| /api/v2/metrics | Display internal metrics per loaded plugin. | [cmetrics text format](https://github.com/fluent/cmetrics) |
+| /api/v2/metrics/prometheus | Display internal metrics per loaded plugin ready in Prometheus Server format. | Prometheus Text 0.0.4 |
+| /api/v2/reload | Execute hot reloading or get the status of hot reloading. See the [hot-reloading documentation](hot-reload.md). | JSON |
+
+### v1 metrics
+
+The following descriptions apply to v1 metric endpoints.
+
+#### `/api/v1/metrics/prometheus` endpoint
+
+The following descriptions apply to metrics outputted in Prometheus format by the
+`/api/v1/metrics/prometheus` endpoint.
+
+The following terms are key to understanding how Fluent Bit processes metrics:
+
+- **Record**: a single message collected from a source, such as a single long line in
+ a file.
+- **Chunk**: log records ingested and stored by Fluent Bit input plugin instances. A
+ batch of records in a chunk are tracked together as a single unit.
+
+ The Fluent Bit engine attempts to fit records into chunks of at most `2 MB`, but
+ the size can vary at runtime. Chunks are then sent to an output. An output plugin
+ instance can either successfully send the full chunk to the destination and mark it
+ as successful, or it can fail the chunk entirely if an unrecoverable error is
+ encountered, or it can ask for the chunk to be retried.
+
+| Metric name | Labels | Description | Type | Unit |
+|----------------------------------------|-------------------------------------------------|-------------|---------|---------|
+| `fluentbit_input_bytes_total` | name: the name or alias for the input instance | The number of bytes of log records that this input instance has ingested successfully. | counter | bytes |
+| `fluentbit_input_records_total` | name: the name or alias for the input instance | The number of log records this input ingested successfully. | counter | records |
+| `fluentbit_output_dropped_records_total` | name: the name or alias for the output instance | The number of log records dropped by the output. These records hit an unrecoverable error or retries expired for their chunk. | counter | records |
+| `fluentbit_output_errors_total` | name: the name or alias for the output instance | The number of chunks with an error that's either unrecoverable or unable to retry. This metric represents the number of times a chunk failed, and doesn't correspond with the number of error messages visible in the Fluent Bit log output. | counter | chunks |
+| `fluentbit_output_proc_bytes_total` | name: the name or alias for the output instance | The number of bytes of log records that this output instance sent successfully. This metric represents the total byte size of all unique chunks sent by this output. If a record is not sent due to some error, it doesn't count towards this metric. | counter | bytes |
+| `fluentbit_output_proc_records_total` | name: the name or alias for the output instance | The number of log records that this output instance sent successfully. This metric represents the total record count of all unique chunks sent by this output. If a record is not sent successfully, it doesn't count towards this metric. | counter | records |
+| `fluentbit_output_retried_records_total` | name: the name or alias for the output instance | The number of log records that experienced a retry. This metric is calculated at the chunk level, the count increased when an entire chunk is marked for retry. An output plugin might perform multiple actions that generate many error messages when uploading a single chunk. | counter | records |
+| `fluentbit_output_retries_failed_total` | name: the name or alias for the output instance | The number of times that retries expired for a chunk. Each plugin configures a `Retry_Limit`, which applies to chunks. When the `Retry_Limit` is exceeded, the chunk is discarded and this metric is incremented. | counter | chunks |
+| `fluentbit_output_retries_total` | name: the name or alias for the output instance | The number of times this output instance requested a retry for a chunk. | counter | chunks |
+| `fluentbit_uptime` | | The number of seconds that Fluent Bit has been running. | counter | seconds |
+| `process_start_time_seconds` | | The Unix Epoch timestamp for when Fluent Bit started. | gauge | seconds |
+
+#### `/api/v1/storage` endpoint
+
+The following descriptions apply to metrics outputted in JSON format by the
+`/api/v1/storage` endpoint.
+
+| Metric Key | Description | Unit |
+|-----------------------------------------------|---------------|---------|
+| `chunks.total_chunks` | The total number of chunks of records that Fluent Bit is currently buffering. | chunks |
+| `chunks.mem_chunks` | The total number of chunks that are currently buffered in memory. Chunks can be both in memory and on the file system at the same time. | chunks |
+| `chunks.fs_chunks` | The total number of chunks saved to the filesystem. | chunks |
+| `chunks.fs_chunks_up` | The count of chunks that are both in file system and in memory. | chunks |
+| `chunks.fs_chunks_down` | The count of chunks that are only in the file system. | chunks |
+| `input_chunks.{plugin name}.status.overlimit` | Indicates whether the input instance exceeded its configured `Mem_Buf_Limit.` | boolean |
+| `input_chunks.{plugin name}.status.mem_size` | The size of memory that this input is consuming to buffer logs in chunks. | bytes |
+| `input_chunks.{plugin name}.status.mem_limit` | The buffer memory limit (`Mem_Buf_Limit`) that applies to this input plugin. | bytes |
+| `input_chunks.{plugin name}.chunks.total` | The current total number of chunks owned by this input instance. | chunks |
+| `input_chunks.{plugin name}.chunks.up` | The current number of chunks that are in memory for this input. If file system storage is enabled, chunks that are "up" are also stored in the filesystem layer. | chunks |
+| `input_chunks.{plugin name}.chunks.down` | The current number of chunks that are "down" in the filesystem for this input. | chunks |
+| `input_chunks.{plugin name}.chunks.busy` | Chunks are that are being processed or sent by outputs and are not eligible to have new data appended. | chunks |
+| `input_chunks.{plugin name}.chunks.busy_size` | The sum of the byte size of each chunk which is currently marked as busy. | bytes |
+
+### v2 metrics
+
+The following descriptions apply to v2 metric endpoints.
+
+#### `/api/v2/metrics/prometheus` or `/api/v2/metrics` endpoint
+
+The following descriptions apply to metrics outputted in Prometheus format by the
+`/api/v2/metrics/prometheus` or `/api/v2/metrics` endpoints.
+
+The following terms are key to understanding how Fluent Bit processes metrics:
+
+- **Record**: a single message collected from a source, such as a single long line in
+ a file.
+- **Chunk**: log records ingested and stored by Fluent Bit input plugin instances. A
+ batch of records in a chunk are tracked together as a single unit.
+
+ The Fluent Bit engine attempts to fit records into chunks of at most `2 MB`, but
+ the size can vary at runtime. Chunks are then sent to an output. An output plugin
+ instance can either successfully send the full chunk to the destination and mark it
+ as successful, or it can fail the chunk entirely if an unrecoverable error is
+ encountered, or it can ask for the chunk to be retried.
+
+| Metric Name | Labels | Description | Type | Unit |
+|--------------------------------------------|-------------------------------------------------------------------------|-------------|---------|---------|
+| `fluentbit_input_bytes_total` | name: the name or alias for the input instance | The number of bytes of log records that this input instance has ingested successfully. | counter | bytes |
+| `fluentbit_input_records_total` | name: the name or alias for the input instance | The number of log records this input ingested successfully. | counter | records |
+| `fluentbit_filter_bytes_total` | name: the name or alias for the filter instance | The number of bytes of log records that this filter instance has ingested successfully. | counter | bytes |
+| `fluentbit_filter_records_total` | name: the name or alias for the filter instance | The number of log records this filter has ingested successfully. | counter | records |
+| `fluentbit_filter_added_records_total` | name: the name or alias for the filter instance | The number of log records added by the filter into the data pipeline. | counter | records |
+| `fluentbit_filter_drop_records_total` | name: the name or alias for the filter instance | The number of log records dropped by the filter and removed from the data pipeline. | counter | records |
+| `fluentbit_output_dropped_records_total` | name: the name or alias for the output instance | The number of log records dropped by the output. These records hit an unrecoverable error or retries expired for their chunk. | counter | records |
+| `fluentbit_output_errors_total` | name: the name or alias for the output instance | The number of chunks with an error that's either unrecoverable or unable to retry. This metric represents the number of times a chunk failed, and doesn't correspond with the number of error messages visible in the Fluent Bit log output. | counter | chunks |
+| `fluentbit_output_proc_bytes_total` | name: the name or alias for the output instance | The number of bytes of log records that this output instance sent successfully. This metric represents the total byte size of all unique chunks sent by this output. If a record is not sent due to some error, it doesn't count towards this metric. | counter | bytes |
+| `fluentbit_output_proc_records_total` | name: the name or alias for the output instance | The number of log records that this output instance sent successfully. This metric represents the total record count of all unique chunks sent by this output. If a record is not sent successfully, it doesn't count towards this metric. | counter | records |
+| `fluentbit_output_retried_records_total` | name: the name or alias for the output instance | The number of log records that experienced a retry. This metric is calculated at the chunk level, the count increased when an entire chunk is marked for retry. An output plugin might perform multiple actions that generate many error messages when uploading a single chunk. | counter | records |
+| `fluentbit_output_retries_failed_total` | name: the name or alias for the output instance | The number of times that retries expired for a chunk. Each plugin configures a `Retry_Limit`, which applies to chunks. When the `Retry_Limit` is exceeded, the chunk is discarded and this metric is incremented. | counter | chunks |
+| `fluentbit_output_retries_total` | name: the name or alias for the output instance | The number of times this output instance requested a retry for a chunk. | counter | chunks |
+| `fluentbit_uptime` | hostname: the hostname on running Fluent Bit | The number of seconds that Fluent Bit has been running. | counter | seconds |
+| `fluentbit_process_start_time_seconds` | hostname: the hostname on running Fluent Bit | The Unix Epoch time stamp for when Fluent Bit started. | gauge | seconds |
+| `fluentbit_build_info` | hostname: the hostname, version: the version of Fluent Bit, os: OS type | Build version information. The returned value is originated from initializing the Unix Epoch time stamp of configuration context. | gauge | seconds |
+| `fluentbit_hot_reloaded_times` | hostname: the hostname on running Fluent Bit | Collect the count of hot reloaded times. | gauge | seconds |
+
+#### Storage layer
+
+The following are detailed descriptions for the metrics collected by the storage
+layer.
+
+| Metric Name | Labels | Description | Type | Unit |
+|---------------------------------------------|------------------------------|---------------|---------|---------|
+| `fluentbit_input_chunks.storage_chunks` | None | The total number of chunks of records that Fluent Bit is currently buffering. | gauge | chunks |
+| `fluentbit_storage_mem_chunk` | None | The total number of chunks that are currently buffered in memory. Chunks can be both in memory and on the file system at the same time. | gauge | chunks |
+| `fluentbit_storage_fs_chunks` | None | The total number of chunks saved to the file system. | gauge | chunks |
+| `fluentbit_storage_fs_chunks_up` | None | The count of chunks that are both in file system and in memory. | gauge | chunks |
+| `fluentbit_storage_fs_chunks_down` | None | The count of chunks that are only in the file system. | gauge | chunks |
+| `fluentbit_storage_fs_chunks_busy` | None | The total number of chunks are in a busy state. | gauge | chunks |
+| `fluentbit_storage_fs_chunks_busy_bytes` | None | The total bytes of chunks are in a busy state. | gauge | bytes |
+| `fluentbit_input_storage_overlimit` | name: the name or alias for the input instance | Indicates whether the input instance exceeded its configured `Mem_Buf_Limit.` | gauge | boolean |
+| `fluentbit_input_storage_memory_bytes` | name: the name or alias for the input instance | The size of memory that this input is consuming to buffer logs in chunks. | gauge | bytes |
+| `fluentbit_input_storage_chunks` | name: the name or alias for the input instance | The current total number of chunks owned by this input instance. | gauge | chunks |
+| `fluentbit_input_storage_chunks_up` | name: the name or alias for the input instance | The current number of chunks that are in memory for this input. If file system storage is enabled, chunks that are "up" are also stored in the filesystem layer. | gauge | chunks |
+| `fluentbit_input_storage_chunks_down` | name: the name or alias for the input instance | The current number of chunks that are "down" in the filesystem for this input. | gauge | chunks |
+| `fluentbit_input_storage_chunks_busy` | name: the name or alias for the input instance | Chunks are that are being processed or sent by outputs and are not eligible to have new data appended. | gauge | chunks |
+| `fluentbit_input_storage_chunks_busy_bytes` | name: the name or alias for the input instance | The sum of the byte size of each chunk which is currently marked as busy. | gauge | bytes |
+| `fluentbit_output_upstream_total_connections` | name: the name or alias for the output instance | The sum of the connection count of each output plugins. | gauge | bytes |
+| `fluentbit_output_upstream_busy_connections` | name: the name or alias for the output instance | The sum of the connection count in a busy state of each output plugins. | gauge | bytes |
+
+### Uptime example
Query the service uptime with the following command:
-```
+```curl
$ curl -s http://127.0.0.1:2020/api/v1/uptime | jq
```
-it should print a similar output like this:
+The command prints a similar output like this:
```javascript
{
@@ -212,7 +249,7 @@ it should print a similar output like this:
}
```
-### Metrics Examples
+### Metrics example
Query internal metrics in JSON format with the following command:
@@ -220,7 +257,7 @@ Query internal metrics in JSON format with the following command:
$ curl -s http://127.0.0.1:2020/api/v1/metrics | jq
```
-it should print a similar output like this:
+The command prints a similar output like this:
```javascript
{
@@ -242,7 +279,7 @@ it should print a similar output like this:
}
```
-### Metrics in Prometheus format
+### Query metrics in Prometheus format
Query internal metrics in Prometheus Text 0.0.4 format:
@@ -250,9 +287,9 @@ Query internal metrics in Prometheus Text 0.0.4 format:
$ curl -s http://127.0.0.1:2020/api/v1/metrics/prometheus
```
-this time the same metrics will be in Prometheus format instead of JSON:
+This command returns the same metrics in Prometheus format instead of JSON:
-```
+```text
fluentbit_input_records_total{name="cpu.0"} 57 1509150350542
fluentbit_input_bytes_total{name="cpu.0"} 18069 1509150350542
fluentbit_output_proc_records_total{name="stdout.0"} 54 1509150350542
@@ -262,13 +299,17 @@ fluentbit_output_retries_total{name="stdout.0"} 0 1509150350542
fluentbit_output_retries_failed_total{name="stdout.0"} 0 1509150350542
```
-### Configuring Aliases
+### Configure aliases
-By default configured plugins on runtime get an internal name in the format _plugin_name.ID_. For monitoring purposes, this can be confusing if many plugins of the same type were configured. To make a distinction each configured input or output section can get an _alias_ that will be used as the parent name for the metric.
+By default, configured plugins on runtime get an internal name in the format
+`_plugin_name.ID_`. For monitoring purposes, this can be confusing if many plugins of
+the same type were configured. To make a distinction each configured input or output
+section can get an _alias_ that will be used as the parent name for the metric.
-The following example set an alias to the INPUT section which is using the [CPU](../pipeline/inputs/cpu-metrics.md) input plugin:
+The following example sets an alias to the `INPUT` section of the configuration file,
+which is using the [CPU](../pipeline/inputs/cpu-metrics.md) input plugin:
-```
+```yaml
[SERVICE]
HTTP_Server On
HTTP_Listen 0.0.0.0
@@ -284,7 +325,8 @@ The following example set an alias to the INPUT section which is using the [CPU]
Match *
```
-Now when querying the metrics we get the aliases in place instead of the plugin name:
+When querying the related metrics, the aliases are returned instead of the plugin
+name:
```javascript
{
@@ -306,13 +348,22 @@ Now when querying the metrics we get the aliases in place instead of the plugin
}
```
-## Grafana Dashboard and Alerts
+## Grafana dashboard and alerts
+
+
-Fluent Bit's exposed [prometheus style metrics](https://docs.fluentbit.io/manual/administration/monitoring) can be leveraged to create dashboards and alerts.
+You can create Grafana dashboards and alerts using Fluent Bit's exposed Prometheus
+style metrics.
-The provided [example dashboard](https://github.com/fluent/fluent-bit-docs/tree/8172a24d278539a1420036a9434e9f56d987a040/monitoring/dashboard.json) is heavily inspired by [Banzai Cloud](https://banzaicloud.com)'s [logging operator dashboard](https://grafana.com/grafana/dashboards/7752) but with a few key differences such as the use of the `instance` label (see [why here](https://www.robustperception.io/controlling-the-instance-label)), stacked graphs and a focus on Fluent Bit metrics.
+The provided [example dashboard](https://github.com/fluent/fluent-bit-docs/tree/8172a24d278539a1420036a9434e9f56d987a040/monitoring/dashboard.json)
+is heavily inspired by [Banzai Cloud](https://banzaicloud.com)'s
+[logging operator dashboard](https://grafana.com/grafana/dashboards/7752) with a few
+key differences, such as the use of the `instance` label, stacked graphs, and a focus
+on Fluent Bit metrics. See
+[this blog post](https://www.robustperception.io/controlling-the-instance-label)
+for more information.
-
+
### Alerts
@@ -322,34 +373,43 @@ Sample alerts are available [here](https://github.com/fluent/fluent-bit-docs/tre
Fluent bit now supports four new configs to set up the health check.
-| Config Name | Description | Default Value |
-| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ------------- |
-| Health_Check | enable Health check feature | Off |
-| HC_Errors_Count | the error count to meet the unhealthy requirement, this is a sum for all output plugins in a defined HC_Period, example for output error: ` [2022/02/16 10:44:10] [ warn] [engine] failed to flush chunk '1-1645008245.491540684.flb', retry in 7 seconds: task_id=0, input=forward.1 > output=cloudwatch_logs.3 (out_id=3)` | 5 |
-| HC_Retry_Failure_Count | the retry failure count to meet the unhealthy requirement, this is a sum for all output plugins in a defined HC_Period, example for retry failure: `[2022/02/16 20:11:36] [ warn] [engine] chunk '1-1645042288.260516436.flb' cannot be retried: task_id=0, input=tcp.3 > output=cloudwatch_logs.1 ` | 5 |
-| HC_Period | The time period by second to count the error and retry failure data point | 60 |
+| Configuration name | Description | Default |
+| ---------------------- | ------------| ------------- |
+| `Health_Check` | enable Health check feature | Off |
+| `HC_Errors_Count` | the error count to meet the unhealthy requirement, this is a sum for all output plugins in a defined HC_Period, example for output error: `[2022/02/16 10:44:10] [ warn] [engine] failed to flush chunk '1-1645008245.491540684.flb', retry in 7 seconds: task_id=0, input=forward.1 > output=cloudwatch_logs.3 (out_id=3)` | 5 |
+| `HC_Retry_Failure_Count` | the retry failure count to meet the unhealthy requirement, this is a sum for all output plugins in a defined HC_Period, example for retry failure: `[2022/02/16 20:11:36] [ warn] [engine] chunk '1-1645042288.260516436.flb' cannot be retried: task_id=0, input=tcp.3 > output=cloudwatch_logs.1` | 5 |
+| `HC_Period` | The time period by second to count the error and retry failure data point | 60 |
-*Note: Not every error log means an error nor be counted, the errors retry failures count only on specific errors which is the example in config table description*
+Not every error log means an error to be counted. The error retry failures count only
+on specific errors, which is the example in configuration table description.
-So the feature works as: Based on the HC_Period customer setup, if the real error number is over `HC_Errors_Count` or retry failure is over `HC_Retry_Failure_Count`, fluent bit will be considered as unhealthy. The health endpoint will return HTTP status 500 and String `error`. Otherwise it's healthy, will return HTTP status 200 and string `ok`
+Based on the `HC_Period` setting, if the real error number is over `HC_Errors_Count`,
+or retry failure is over `HC_Retry_Failure_Count`, Fluent Bit is considered
+unhealthy. The health endpoint returns an HTTP status `500` and an `error` message.
+Otherwise, the endpoint returns HTTP status `200` and an `ok` message.
-The equation is:
-```
-health status = (HC_Errors_Count > HC_Errors_Count config value) OR (HC_Retry_Failure_Count > HC_Retry_Failure_Count config value) IN the HC_Period interval
+The equation to calculate this behavior is:
+
+```text
+health status = (HC_Errors_Count > HC_Errors_Count config value) OR
+(HC_Retry_Failure_Count > HC_Retry_Failure_Count config value) IN
+the HC_Period interval
```
-*Note: the HC_Errors_Count and HC_Retry_Failure_Count only count for output plugins and count a sum for errors and retry failures from all output plugins which is running.*
-See the config example:
+The `HC_Errors_Count` and `HC_Retry_Failure_Count` only count for output plugins and
+count a sum for errors and retry failures from all running output plugins.
-```
+The following configuration file example shows how to define these settings:
+
+```yaml
[SERVICE]
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
- Health_Check On
- HC_Errors_Count 5
- HC_Retry_Failure_Count 5
- HC_Period 5
+ Health_Check On
+ HC_Errors_Count 5
+ HC_Retry_Failure_Count 5
+ HC_Period 5
[INPUT]
Name cpu
@@ -359,53 +419,23 @@ See the config example:
Match *
```
-The command to call health endpoint
+Use the following command to call the health endpoint:
```bash
-$ curl -s http://127.0.0.1:2020/api/v1/health
+curl -s http://127.0.0.1:2020/api/v1/health
```
-Based on the fluent bit status, the result will be:
-
-* HTTP status 200 and "ok" in response to healthy status
-* HTTP status 500 and "error" in response for unhealthy status
+With the example config, the health status is determined by the following equation:
-With the example config, the health status is determined by following equation:
-```
+```text
Health status = (HC_Errors_Count > 5) OR (HC_Retry_Failure_Count > 5) IN 5 seconds
```
-If (HC_Errors_Count > 5) OR (HC_Retry_Failure_Count > 5) IN 5 seconds is TRUE, then it's unhealthy.
-
-If (HC_Errors_Count > 5) OR (HC_Retry_Failure_Count > 5) IN 5 seconds is FALSE, then it's healthy.
-
-
-## Calyptia
-
-[Calyptia](https://calyptia.com/free-trial) is a hosted service that allows you to monitor your Fluent Bit agents including data flow, metrics and configurations.
-
-
-
-### Get Started with Calyptia Cloud
-
-Register your Fluent Bit agent will take **less than one minute**, steps:
-
-* Go to the calyptia core console and sign-in
-* On the left menu click on settings and generate/copy your API key
-
-In your Fluent Bit configuration file, append the following configuration section:
-
-```
-[CUSTOM]
- name calyptia
- api_key 
+
[Fluent Bit](https://fluentbit.io) has an Engine that helps to coordinate the data ingestion from input plugins and calls the _Scheduler_ to decide when it is time to flush the data through one or multiple output plugins. The Scheduler flushes new data at a fixed time of seconds and the _Scheduler_ retries when asked.
Once an output plugin gets called to flush some data, after processing that data it can notify the Engine three possible return statuses:
diff --git a/administration/transport-security.md b/administration/transport-security.md
index f2a644c8a..4443cfd70 100644
--- a/administration/transport-security.md
+++ b/administration/transport-security.md
@@ -9,6 +9,7 @@ Both input and output plugins that perform Network I/O can optionally enable TLS
| :--- | :--- | :--- |
| tls | enable or disable TLS support | Off |
| tls.verify | force certificate validation | On |
+| tls.verify\_hostname | force TLS verification of hostnames | Off |
| tls.debug | Set TLS debug verbosity level. It accept the following values: 0 \(No debug\), 1 \(Error\), 2 \(State change\), 3 \(Informational\) and 4 Verbose | 1 |
| tls.ca\_file | absolute path to CA certificate file | |
| tls.ca\_path | absolute path to scan for certificate files | |
@@ -171,3 +172,42 @@ Fluent Bit supports [TLS server name indication](https://en.wikipedia.org/wiki/S
tls.ca_file /etc/certs/fluent.crt
tls.vhost fluent.example.com
```
+
+### Verify subjectAltName
+
+By default, TLS verification of hostnames is not done automatically.
+As an example, we can extract the X509v3 Subject Alternative Name from a certificate:
+
+```
+X509v3 Subject Alternative Name:
+ DNS:my.fluent-aggregator.net
+```
+
+As you can see, this certificate covers only `my.fluent-aggregator.net` so if we use a different hostname it should fail.
+
+To fully verify the alternative name and demonstrate the failure we enable `tls.verify_hostname`:
+
+
+```text
+[INPUT]
+ Name cpu
+ Tag cpu
+
+[OUTPUT]
+ Name forward
+ Match *
+ Host other.fluent-aggregator.net
+ Port 24224
+ tls On
+ tls.verify On
+ tls.verify_hostname on
+ tls.ca_file /path/to/fluent-x509v3-alt-name.crt
+```
+
+This outgoing connect will be failed and disconnected:
+
+```
+[2024/06/17 16:51:31] [error] [tls] error: unexpected EOF with reason: certificate verify failed
+[2024/06/17 16:51:31] [debug] [upstream] connection #50 failed to other.fluent-aggregator.net:24224
+[2024/06/17 16:51:31] [error] [output:forward:forward.0] no upstream connections available
+```
diff --git a/administration/troubleshooting.md b/administration/troubleshooting.md
index cae5ad2ce..033ad23d1 100644
--- a/administration/troubleshooting.md
+++ b/administration/troubleshooting.md
@@ -1,5 +1,7 @@
# Troubleshooting
+
+
* [Tap Functionality: generate events or records](troubleshooting.md#tap-functionality)
* [Dump Internals Signal](troubleshooting#dump-internals-signal)
diff --git a/concepts/data-pipeline/buffer.md b/concepts/data-pipeline/buffer.md
index c13f904e4..928a3b7f6 100644
--- a/concepts/data-pipeline/buffer.md
+++ b/concepts/data-pipeline/buffer.md
@@ -6,9 +6,21 @@ description: Data processing with reliability
Previously defined in the [Buffering](../buffering.md) concept section, the `buffer` phase in the pipeline aims to provide a unified and persistent mechanism to store your data, either using the primary in-memory model or using the filesystem based mode.
-The `buffer` phase already contains the data in an immutable state, meaning, no other filter can be applied.
+The `buffer` phase already contains the data in an immutable state, meaning that no other filter can be applied.
- (1) (2) (2) (2) (2) (2) (2) (2) (1).png>)
+```mermaid
+graph LR
+ accTitle: Fluent Bit data pipeline
+ accDescr: A diagram of the Fluent Bit data pipeline, which includes input, a parser, a filter, a buffer, routing, and various outputs.
+ A[Input] --> B[Parser]
+ B --> C[Filter]
+ C --> D[Buffer]
+ D --> E((Routing))
+ E --> F[Output 1]
+ E --> G[Output 2]
+ E --> H[Output 3]
+ style D stroke:darkred,stroke-width:2px;
+```
{% hint style="info" %}
Note that buffered data is not raw text, it's in Fluent Bit's internal binary representation.
diff --git a/concepts/data-pipeline/filter.md b/concepts/data-pipeline/filter.md
index 2323d165e..c903b7c0a 100644
--- a/concepts/data-pipeline/filter.md
+++ b/concepts/data-pipeline/filter.md
@@ -6,7 +6,19 @@ description: Modify, Enrich or Drop your records
In production environments we want to have full control of the data we are collecting, filtering is an important feature that allows us to **alter** the data before delivering it to some destination.
- (2) (2) (2) (2) (2) (2) (1).png>)
+```mermaid
+graph LR
+ accTitle: Fluent Bit data pipeline
+ accDescr: A diagram of the Fluent Bit data pipeline, which includes input, a parser, a filter, a buffer, routing, and various outputs.
+ A[Input] --> B[Parser]
+ B --> C[Filter]
+ C --> D[Buffer]
+ D --> E((Routing))
+ E --> F[Output 1]
+ E --> G[Output 2]
+ E --> H[Output 3]
+ style C stroke:darkred,stroke-width:2px;
+```
Filtering is implemented through plugins, so each filter available could be used to match, exclude or enrich your logs with some specific metadata.
diff --git a/concepts/data-pipeline/input.md b/concepts/data-pipeline/input.md
index ca8500b0b..0779c0985 100644
--- a/concepts/data-pipeline/input.md
+++ b/concepts/data-pipeline/input.md
@@ -6,7 +6,19 @@ description: The way to gather data from your sources
[Fluent Bit](http://fluentbit.io) provides different _Input Plugins_ to gather information from different sources, some of them just collect data from log files while others can gather metrics information from the operating system. There are many plugins for different needs.
- (2) (2) (2) (2) (2) (2) (2) (1).png>)
+```mermaid
+graph LR
+ accTitle: Fluent Bit data pipeline
+ accDescr: A diagram of the Fluent Bit data pipeline, which includes input, a parser, a filter, a buffer, routing, and various outputs.
+ A[Input] --> B[Parser]
+ B --> C[Filter]
+ C --> D[Buffer]
+ D --> E((Routing))
+ E --> F[Output 1]
+ E --> G[Output 2]
+ E --> H[Output 3]
+ style A stroke:darkred,stroke-width:2px;
+```
When an input plugin is loaded, an internal _instance_ is created. Every instance has its own and independent configuration. Configuration keys are often called **properties**.
diff --git a/concepts/data-pipeline/output.md b/concepts/data-pipeline/output.md
index 5a96f7ee6..2fad550bd 100644
--- a/concepts/data-pipeline/output.md
+++ b/concepts/data-pipeline/output.md
@@ -6,7 +6,21 @@ description: 'Destinations for your data: databases, cloud services and more!'
The output interface allows us to define destinations for the data. Common destinations are remote services, local file system or standard interface with others. Outputs are implemented as plugins and there are many available.
- (1).png>)
+```mermaid
+graph LR
+ accTitle: Fluent Bit data pipeline
+ accDescr: A diagram of the Fluent Bit data pipeline, which includes input, a parser, a filter, a buffer, routing, and various outputs.
+ A[Input] --> B[Parser]
+ B --> C[Filter]
+ C --> D[Buffer]
+ D --> E((Routing))
+ E --> F[Output 1]
+ E --> G[Output 2]
+ E --> H[Output 3]
+ style F stroke:darkred,stroke-width:2px;
+ style G stroke:darkred,stroke-width:2px;
+ style H stroke:darkred,stroke-width:2px;
+```
When an output plugin is loaded, an internal _instance_ is created. Every instance has its own independent configuration. Configuration keys are often called **properties**.
diff --git a/concepts/data-pipeline/parser.md b/concepts/data-pipeline/parser.md
index 034376606..b3d6e05b5 100644
--- a/concepts/data-pipeline/parser.md
+++ b/concepts/data-pipeline/parser.md
@@ -6,8 +6,19 @@ description: Convert Unstructured to Structured messages
Dealing with raw strings or unstructured messages is a constant pain; having a structure is highly desired. Ideally we want to set a structure to the incoming data by the Input Plugins as soon as they are collected:
- (1) (1) (1) (2) (2) (2) (3) (3) (3) (3) (3) (1).png>)
-
+```mermaid
+graph LR
+ accTitle: Fluent Bit data pipeline
+ accDescr: A diagram of the Fluent Bit data pipeline, which includes input, a parser, a filter, a buffer, routing, and various outputs.
+ A[Input] --> B[Parser]
+ B --> C[Filter]
+ C --> D[Buffer]
+ D --> E((Routing))
+ E --> F[Output 1]
+ E --> G[Output 2]
+ E --> H[Output 3]
+ style B stroke:darkred,stroke-width:2px;
+```
The Parser allows you to convert from unstructured to structured data. As a demonstrative example consider the following Apache (HTTP Server) log entry:
```
diff --git a/concepts/data-pipeline/router.md b/concepts/data-pipeline/router.md
index 0041c992e..d267f3607 100644
--- a/concepts/data-pipeline/router.md
+++ b/concepts/data-pipeline/router.md
@@ -6,7 +6,19 @@ description: Create flexible routing rules
Routing is a core feature that allows to **route** your data through Filters and finally to one or multiple destinations. The router relies on the concept of [Tags](../key-concepts.md) and [Matching](../key-concepts.md) rules
- (1) (2) (2) (2) (2) (2) (2) (2) (1) (1).png>)
+```mermaid
+graph LR
+ accTitle: Fluent Bit data pipeline
+ accDescr: A diagram of the Fluent Bit data pipeline, which includes input, a parser, a filter, a buffer, routing, and various outputs.
+ A[Input] --> B[Parser]
+ B --> C[Filter]
+ C --> D[Buffer]
+ D --> E((Routing))
+ E --> F[Output 1]
+ E --> G[Output 2]
+ E --> H[Output 3]
+ style E stroke:darkred,stroke-width:2px;
+```
There are two important concepts in Routing:
@@ -77,7 +89,7 @@ The following example demonstrates how to route data from sources based on a reg
[OUTPUT]
Name stdout
- Match_regex .*_sensor_[AB]
+ Match_regex .*_sensor_[AB]
```
In this configuration, the **Match_regex** rule is set to `.*_sensor_[AB]`. This regular expression will match any Tag that ends with "_sensor_A" or "_sensor_B", regardless of what precedes it.
diff --git a/imgs/processor_opentelemetry_envelope.png b/imgs/processor_opentelemetry_envelope.png
new file mode 100644
index 000000000..d44920a14
Binary files /dev/null and b/imgs/processor_opentelemetry_envelope.png differ
diff --git a/installation/docker.md b/installation/docker.md
index 55d011c76..c4a468876 100644
--- a/installation/docker.md
+++ b/installation/docker.md
@@ -17,6 +17,28 @@ The following table describes the Linux container tags that are available on Doc
| Tag(s) | Manifest Architectures | Description |
| ------------ | ------------------------- | -------------------------------------------------------------- |
+| 3.1.9-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
+| 3.1.9 | x86_64, arm64v8, arm32v7, s390x | Release [v3.1.9](https://fluentbit.io/announcements/v3.1.9/) |
+| 3.1.8-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
+| 3.1.8 | x86_64, arm64v8, arm32v7, s390x | Release [v3.1.8](https://fluentbit.io/announcements/v3.1.8/) |
+| 3.1.7-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
+| 3.1.7 | x86_64, arm64v8, arm32v7, s390x | Release [v3.1.7](https://fluentbit.io/announcements/v3.1.7/) |
+| 3.1.6-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
+| 3.1.6 | x86_64, arm64v8, arm32v7, s390x | Release [v3.1.6](https://fluentbit.io/announcements/v3.1.6/) |
+| 3.1.5-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
+| 3.1.5 | x86_64, arm64v8, arm32v7, s390x | Release [v3.1.5](https://fluentbit.io/announcements/v3.1.5/) |
+| 3.1.4-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
+| 3.1.4 | x86_64, arm64v8, arm32v7, s390x | Release [v3.1.4](https://fluentbit.io/announcements/v3.1.4/) |
+| 3.1.3-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
+| 3.1.3 | x86_64, arm64v8, arm32v7, s390x | Release [v3.1.3](https://fluentbit.io/announcements/v3.1.3/) |
+| 3.1.2-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
+| 3.1.2 | x86_64, arm64v8, arm32v7, s390x | Release [v3.1.2](https://fluentbit.io/announcements/v3.1.2/) |
+| 3.1.1-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
+| 3.1.1 | x86_64, arm64v8, arm32v7, s390x | Release [v3.1.1](https://fluentbit.io/announcements/v3.1.1/) |
+| 3.1.0-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
+| 3.1.0 | x86_64, arm64v8, arm32v7, s390x | Release [v3.1.0](https://fluentbit.io/announcements/v3.1.0/) |
+| 3.0.7-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
+| 3.0.7 | x86_64, arm64v8, arm32v7, s390x | Release [v3.0.7](https://fluentbit.io/announcements/v3.0.7/) |
| 3.0.6-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
| 3.0.6 | x86_64, arm64v8, arm32v7, s390x | Release [v3.0.6](https://fluentbit.io/announcements/v3.0.6/) |
| 3.0.5-debug | x86_64, arm64v8, arm32v7, s390x | Debug images |
diff --git a/installation/getting-started-with-fluent-bit.md b/installation/getting-started-with-fluent-bit.md
index 04354dcff..5b1e4223c 100644
--- a/installation/getting-started-with-fluent-bit.md
+++ b/installation/getting-started-with-fluent-bit.md
@@ -2,6 +2,8 @@
description: The following serves as a guide on how to install/deploy/upgrade Fluent Bit
---
+
+
# Getting Started with Fluent Bit
## Container Deployment
diff --git a/installation/sources/build-and-install.md b/installation/sources/build-and-install.md
index 568130cb4..99674abf8 100644
--- a/installation/sources/build-and-install.md
+++ b/installation/sources/build-and-install.md
@@ -220,3 +220,4 @@ The following table describes the processors available on this version:
| option | description | default |
| :--- | :--- | :--- |
| [FLB\_PROCESSOR\_METRICS\_SELECTOR](../../pipeline/processors/metrics-selector.md) | Enable metrics selector processor | On |
+| [FLB\_PROCESSOR\_LABELS](../../pipeline/processors/labels.md) | Enable metrics label manipulation processor | On |
diff --git a/installation/windows.md b/installation/windows.md
index 1dd7d0a88..0b42b3dfa 100644
--- a/installation/windows.md
+++ b/installation/windows.md
@@ -79,30 +79,30 @@ From version 1.9, `td-agent-bit` is a deprecated package and was removed after 1
## Installation Packages
-The latest stable version is 3.0.6.
+The latest stable version is 3.1.9.
Each version is available via the following download URLs.
| INSTALLERS | SHA256 CHECKSUMS |
| ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------- |
-| [fluent-bit-3.0.6-win32.exe](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-win32.exe) | [9ea39f5efda5b013e9135a5b67b60fa94d27dcacf5c43094dfffb693ada11f2b](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-win32.exe.sha256) |
-| [fluent-bit-3.0.6-win32.zip](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-win32.zip) | [97b6465ed324dd9060dcf10b28341fcfeb3d30ec3abc02583e45902a80dece5e](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-win32.zip.sha256) |
-| [fluent-bit-3.0.6-win64.exe](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-win64.exe) | [4f4e02bca5adac697a8d9c0477b9060c038c9ca94f4b9c31f5ca65c746a4ddfc](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-win64.exe.sha256) |
-| [fluent-bit-3.0.6-win64.zip](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-win64.zip) | [00701ab3260a7f9347dcaeeb55d5183dd6ba1cd51cde92625c05b0e643f2b5c1](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-win64.zip.sha256) |
-| [fluent-bit-3.0.6-winarm64.exe](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-winarm64.exe) | [fc90682c155e1288af81742c50740d3be0bff2ab211b8785c7c7e85196f121cb](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-winarm64.exe.sha256) |
-| [fluent-bit-3.0.6-winarm64.zip](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-winarm64.zip) | [fda4788899de83e68d9abf79f19ea5ced3e324568551fc1543891747ba241832](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-winarm64.zip.sha256) |
+| [fluent-bit-3.1.9-win32.exe](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-win32.exe) | [a42458b6275cd08bbede45ffc9f3d7abd3145e7a82bd806226b86e5cf67793bb](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-win32.exe.sha256) |
+| [fluent-bit-3.1.9-win32.zip](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-win32.zip) | [3eb8bcdbb394bed326b19386ba95c932819aaa6ea0418adbce6675dd98656b41](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-win32.zip.sha256) |
+| [fluent-bit-3.1.9-win64.exe](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-win64.exe) | [ccc12e5c01e9e87b88d431ecae34c014d2b227d2d72282359795605e1efeea3d](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-win64.exe.sha256) |
+| [fluent-bit-3.1.9-win64.zip](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-win64.zip) | [e83b9f1d8c91ebdbaec1b95426c54d1df55f09c84b5446ca1476662ea72f3a36](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-win64.zip.sha256) |
+| [fluent-bit-3.1.9-winarm64.exe](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-winarm64.exe) | [f87aaa56ed3e0dfca39947f0d46e2a7721d89c08efd58fd3f7bf0968522ffc13](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-winarm64.exe.sha256) |
+| [fluent-bit-3.1.9-winarm64.zip](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-winarm64.zip) | [4be74c9696836fd802d79a00e65e53e13f901a2e3adb6c7786e74ffde89b072b](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-winarm64.zip.sha256) |
**Note these are now using the Github Actions built versions, the legacy AppVeyor builds are still available (AMD 32/64 only) at releases.fluentbit.io but are deprecated.**
MSI installers are also available:
-- [fluent-bit-3.0.6-win32.msi](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-win32.msi)
-- [fluent-bit-3.0.6-win64.msi](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-win64.msi)
-- [fluent-bit-3.0.6-winarm64.msi](https://packages.fluentbit.io/windows/fluent-bit-3.0.6-winarm64.msi)
+- [fluent-bit-3.1.9-win32.msi](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-win32.msi)
+- [fluent-bit-3.1.9-win64.msi](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-win64.msi)
+- [fluent-bit-3.1.9-winarm64.msi](https://packages.fluentbit.io/windows/fluent-bit-3.1.9-winarm64.msi)
To check the integrity, use `Get-FileHash` cmdlet on PowerShell.
```powershell
-PS> Get-FileHash fluent-bit-3.0.6-win32.exe
+PS> Get-FileHash fluent-bit-3.1.9-win32.exe
```
## Installing from ZIP archive
@@ -112,7 +112,7 @@ Download a ZIP archive from above. There are installers for 32-bit and 64-bit en
Then you need to expand the ZIP archive. You can do this by clicking "Extract All" on Explorer, or if you're using PowerShell, you can use `Expand-Archive` cmdlet.
```powershell
-PS> Expand-Archive fluent-bit-3.0.6-win64.zip
+PS> Expand-Archive fluent-bit-3.1.9-win64.zip
```
The ZIP package contains the following set of files.
diff --git a/local-testing/validating-your-data-and-structure.md b/local-testing/validating-your-data-and-structure.md
index 81e3d2e63..aa72c6af7 100644
--- a/local-testing/validating-your-data-and-structure.md
+++ b/local-testing/validating-your-data-and-structure.md
@@ -1,47 +1,80 @@
-# Validating your Data and Structure
+# Validating your data and structure
-Fluent Bit is a powerful log processing tool that can deal with different sources and formats, in addition it provides several filters that can be used to perform custom modifications. This flexibility is really good but while your pipeline grows, it's strongly recommended to validate your data and structure.
+Fluent Bit is a powerful log processing tool that supports mulitple sources and
+formats. In addition, it provides filters that can be used to perform custom
+modifications. As your pipeline grows, it's important to validate your data and
+structure.
-> We encourage Fluent Bit users to integrate data validation in their CI systems
+Fluent Bit users are encouraged to integrate data validation in their contininuous
+integration (CI) systems.
-A simplified view of our data processing pipeline is as follows:
+In a normal production environment, inputs, filters, and outputs are defined in the
+configuration. Fluent Bit provides the [Expect](../pipeline/filters/expect.md) filter,
+which can be used to validate `keys` and `values` from your records and take action
+when an exception is found.
-
+A simplified view of the data processing pipeline is as follows:
-In a normal production environment, many Inputs, Filters, and Outputs are defined in the configuration, so integrating a continuous validation of your configuration against expected results is a must. For this requirement, Fluent Bit provides a specific Filter called **Expect** which can be used to validate expected Keys and Values from your records and takes some action when an exception is found.
+```mermaid
+flowchart LR
+IS[Inputs / Sources]
+Fil[Filters]
+OD[Outputs/ Destination]
+IS --> Fil --> OD
+```
-## How it Works
+## Understand structure and configuration
-As an example, consider the following pipeline where your source of data is a normal file with JSON content on it and then two filters: [grep](../pipeline/filters/grep.md) to exclude certain records and [record\_modifier](../pipeline/filters/record-modifier.md) to alter the record content adding and removing specific keys.
+Consider the following pipeline, where your source of data is a file with JSON
+content and two filters:
-
+- [grep](../pipeline/filters/grep.md) to exclude certain records
+- [record_modifier](../pipeline/filters/record-modifier.md) to alter the record
+ content by adding and removing specific keys.
-Ideally you want to add checkpoints of validation of your data between each step so you can know if your data structure is correct, we do this by using **expect** filter.
+```mermaid
+flowchart LR
+tail["tail (input)"]
+grep["grep (filter)"]
+record["record_modifier (filter)"]
+stdout["stdout (output)"]
-
+tail --> grep
+grep --> record
+record --> stdout
+```
-Expect filter sets rules that aims to validate certain criteria like:
+Add data validation between each step to ensure your data structure is correct.
+
+This example uses the `expect` filter.
+
+```mermaid
+flowchart LR
+tail["tail (input)"]
+grep["grep (filter)"]
+record["record_modifier (filter)"]
+stdout["stdout (output)"]
+E1["expect (filter)"]
+E2["expect (filter)"]
+E3["expect (filter)"]
+tail --> E1 --> grep
+grep --> E2 --> record --> E3 --> stdout
+```
-* does the record contain a key A ?
-* does the record not contains key A?
-* does the record key A value equals NULL ?
-* does the record key A value a different value than NULL ?
-* does the record key A value equals B ?
+`Expect` filters set rules aiming to validate criteria like:
-Every expect filter configuration can expose specific rules to validate the content of your records, it supports the following configuration properties:
+- Does the record contain a key `A`?
+- Does the record not contain key `A`?
+- Does the record key `A` value equal `NULL`?
+- Is the record key `A` value not `NULL`?
+- Does the record key `A` value equal `B`?
-| Property | Description |
-| :--- | :--- |
-| key\_exists | Check if a key with a given name exists in the record. |
-| key\_not\_exists | Check if a key does not exist in the record. |
-| key\_val\_is\_null | check that the value of the key is NULL. |
-| key\_val\_is\_not\_null | check that the value of the key is NOT NULL. |
-| key\_val\_eq | check that the value of the key equals the given value in the configuration. |
-| action | action to take when a rule does not match. The available options are `warn` or `exit`. On `warn`, a warning message is sent to the logging layer when a mismatch of the rules above is found; using `exit` makes Fluent Bit abort with status code `255`. |
+Every `expect` filter configuration exposes rules to validate the content of your
+records using [configuration properties](../pipeline/filters/expect.md#configuration-parameters).
-## Start Testing
+## Test the configuration
-Consider the following JSON file called `data.log` with the following content:
+Consider a JSON file `data.log` with the following content:
```javascript
{"color": "blue", "label": {"name": null}}
@@ -49,7 +82,9 @@ Consider the following JSON file called `data.log` with the following content:
{"color": "green", "label": {"name": "abc"}, "meta": null}
```
-The following Fluent Bit configuration file will configure a pipeline to consume the log above apply an expect filter to validate that keys `color` and `label` exists:
+The following Fluent Bit configuration file configures a pipeline to consume the
+log, while applying an `expect` filter to validate that the keys `color` and `label`
+exist:
```python
[SERVICE]
@@ -76,9 +111,12 @@ The following Fluent Bit configuration file will configure a pipeline to consume
match *
```
-note that if for some reason the JSON parser failed or is missing in the `tail` input \(line 9\), the `expect` filter will trigger the `exit` action. As a test, go ahead and comment out or remove line 9.
+If the JSON parser fails or is missing in the `tail` input
+(`parser json`), the `expect` filter triggers the `exit` action.
-As a second step, we will extend our pipeline and we will add a grep filter to match records that map `label` contains a key called `name` with value `abc`, then an expect filter to re-validate that condition:
+To extend the pipeline, add a grep filter to match records that map `label`
+containing a key called `name` with value the `abc`, and add an `expect` filter
+to re-validate that condition:
```python
[SERVICE]
@@ -131,7 +169,8 @@ As a second step, we will extend our pipeline and we will add a grep filter to m
match *
```
-## Deploying in Production
-
-When deploying your configuration in production, you might want to remove the expect filters from your configuration since it's an unnecessary _extra work_ unless you want to have a 100% coverage of checks at runtime.
+## Production deployment
+When deploying in production, consider removing the `expect` filters from your
+configuration. These filters are unneccesary unless you need 100% coverage of
+checks at runtime.
diff --git a/pipeline/filters/grep.md b/pipeline/filters/grep.md
index c3aeb609d..019b007c5 100644
--- a/pipeline/filters/grep.md
+++ b/pipeline/filters/grep.md
@@ -1,28 +1,33 @@
---
-description: Select or exclude records per patterns
+description: Select or exclude records using patterns
---
# Grep
-The _Grep Filter_ plugin allows you to match or exclude specific records based on regular expression patterns for values or nested values.
+The _Grep Filter_ plugin lets you match or exclude specific records based on
+regular expression patterns for values or nested values.
-## Configuration Parameters
+## Configuration parameters
The plugin supports the following configuration parameters:
-| Key | Value Format | Description |
-| :--- | :--- | :--- |
-| Regex | KEY REGEX | Keep records in which the content of KEY matches the regular expression. |
-| Exclude | KEY REGEX | Exclude records in which the content of KEY matches the regular expression. |
-| Logical_Op| Operation | Specify which logical operator to use. `AND` , `OR` and `legacy` are allowed as an Operation. Default is `legacy` for backward compatibility. In `legacy` mode the behaviour is either AND or OR depending whether the `grep` is including (uses AND) or excluding (uses OR). Only available from 2.1+. |
+| Key | Value Format | Description |
+| ------------ | ------------ | ----------- |
+| `Regex` | KEY REGEX | Keep records where the content of KEY matches the regular expression. |
+| `Exclude` | KEY REGEX | Exclude records where the content of KEY matches the regular expression. |
+| `Logical_Op` | Operation | Specify a logical operator: `AND`, `OR` or `legacy` (default). In `legacy` mode the behaviour is either `AND` or `OR` depending on whether the `grep` is including (uses AND) or excluding (uses OR). Available from 2.1 or higher. |
-#### Record Accessor Enabled
+### Record Accessor Enabled
-This plugin enables the [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) feature to specify the KEY. Using the _record accessor_ is suggested if you want to match values against nested values.
+Enable the [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md)
+feature to specify the KEY. Use the record accessor to match values against nested
+values.
-## Getting Started
+## Filter records
-In order to start filtering records, you can run the filter from the command line or through the configuration file. The following example assumes that you have a file called `lines.txt` with the following content:
+To start filtering records, run the filter from the command line or through the
+configuration file. The following example assumes that you have a file named
+`lines.txt` with the following content:
```text
{"log": "aaa"}
@@ -35,20 +40,25 @@ In order to start filtering records, you can run the filter from the command lin
{"log": "ggg"}
```
-### Command Line
+### Command line
-> Note: using the command line mode need special attention to quote the regular expressions properly. It's suggested to use a configuration file.
+When using the command line, pay close attention to quote the regular expressions.
+Using a configuration file might be easier.
-The following command will load the _tail_ plugin and read the content of `lines.txt` file. Then the _grep_ filter will apply a regular expression rule over the _log_ field \(created by tail plugin\) and only _pass_ the records which field value starts with _aa_:
+The following command loads the [tail](../../pipeline/inputs/tail) plugin and
+reads the content of `lines.txt`. Then the `grep` filter applies a regular
+expression rule over the `log` field created by the `tail` plugin and only passes
+records with a field value starting with `aa`:
```text
$ bin/fluent-bit -i tail -p 'path=lines.txt' -F grep -p 'regex=log aa' -m '*' -o stdout
```
-### Configuration File
+### Configuration file
{% tabs %}
{% tab title="fluent-bit.conf" %}
+
```python
[SERVICE]
parsers_file /path/to/parsers.conf
@@ -67,9 +77,11 @@ $ bin/fluent-bit -i tail -p 'path=lines.txt' -F grep -p 'regex=log aa' -m '*' -o
name stdout
match *
```
+
{% endtab %}
{% tab title="fluent-bit.yaml" %}
+
```yaml
service:
parsers_file: /path/to/parsers.conf
@@ -87,14 +99,21 @@ pipeline:
match: '*'
```
+
{% endtab %}
{% endtabs %}
-The filter allows to use multiple rules which are applied in order, you can have many _Regex_ and _Exclude_ entries as required.
+The filter lets you use multiple rules which are applied in order. You can
+have as many `Regex` and `Exclude` entries as required.
### Nested fields example
-If you want to match or exclude records based on nested values, you can use a [Record Accessor ](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md)format as the KEY name. Consider the following record example:
+To match or exclude records based on nested values, you can use
+[Record
+Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md)
+format as the `KEY` name.
+
+Consider the following record example:
```javascript
{
@@ -113,40 +132,45 @@ If you want to match or exclude records based on nested values, you can use a [R
}
```
-if you want to exclude records that match given nested field \(for example `kubernetes.labels.app`\), you can use the following rule:
+For example, to exclude records that match the nested field `kubernetes.labels.app`,
+use the following rule:
{% tabs %}
{% tab title="fluent-bit.conf" %}
+
```python
[FILTER]
Name grep
Match *
Exclude $kubernetes['labels']['app'] myapp
```
-{% endtab %}
+{% endtab %}
{% tab title="fluent-bit.yaml" %}
+
```yaml
filters:
- name: grep
match: '*'
exclude: $kubernetes['labels']['app'] myapp
```
+
{% endtab %}
{% endtabs %}
-### Excluding records missing/invalid fields
-
-It may be that in your processing pipeline you want to drop records that are missing certain keys.
+### Excluding records with missing or invalid fields
-A simple way to do this is just to `exclude` with a regex that matches anything, a missing key will fail this check.
+You might want to drop records that are missing certain keys.
-Here is an example that checks for a specific valid value for the key as well:
+One way to do this is to `exclude` with a regex that matches anything. A missing
+key fails this check.
+The followinfg example checks for a specific valid value for the key:
{% tabs %}
{% tab title="fluent-bit.conf" %}
-```
+
+```text
# Use Grep to verify the contents of the iot_timestamp value.
# If the iot_timestamp key does not exist, this will fail
# and exclude the row.
@@ -156,9 +180,10 @@ Here is an example that checks for a specific valid value for the key as well:
Match iots_thread.*
Regex iot_timestamp ^\d{4}-\d{2}-\d{2}
```
-{% endtab %}
+{% endtab %}
{% tab title="fluent-bit.yaml" %}
+
```yaml
filters:
- name: grep
@@ -166,20 +191,23 @@ Here is an example that checks for a specific valid value for the key as well:
match: iots_thread.*
regex: iot_timestamp ^\d{4}-\d{2}-\d{2}
```
+
{% endtab %}
{% endtabs %}
-The specified key `iot_timestamp` must match the expected expression - if it does not or is missing/empty then it will be excluded.
+The specified key `iot_timestamp` must match the expected expression. If it doesn't,
+or is missing or empty, then it will be excluded.
### Multiple conditions
-If you want to set multiple `Regex` or `Exclude`, you can use `Logical_Op` property to use logical conjuction or disjunction.
-
-Note: If `Logical_Op` is set, setting both 'Regex' and `Exclude` results in an error.
+If you want to set multiple `Regex` or `Exclude`, use the `Logical_Op` property
+to use a logical conjuction or disjunction.
+If `Logical_Op` is set, setting both `Regex` and `Exclude` results in an error.
{% tabs %}
{% tab title="fluent-bit.conf" %}
+
```python
[INPUT]
Name dummy
@@ -196,9 +224,11 @@ Note: If `Logical_Op` is set, setting both 'Regex' and `Exclude` results in an e
[OUTPUT]
Name stdout
```
+
{% endtab %}
{% tab title="fluent-bit.yaml" %}
+
```yaml
pipeline:
inputs:
@@ -215,11 +245,13 @@ pipeline:
outputs:
- name: stdout
```
+
{% endtab %}
{% endtabs %}
-Output will be
-```
+The output looks similar to:
+
+```text
Fluent Bit v2.0.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
@@ -236,4 +268,4 @@ Fluent Bit v2.0.9
[2023/01/22 09:46:49] [ info] [output:stdout:stdout.0] worker #0 started
[0] dummy: [1674348410.558341857, {"endpoint"=>"localhost", "value"=>"something"}]
[0] dummy: [1674348411.546425499, {"endpoint"=>"localhost", "value"=>"something"}]
-```
\ No newline at end of file
+```
diff --git a/pipeline/filters/kubernetes.md b/pipeline/filters/kubernetes.md
index 3a352eadb..492c86862 100644
--- a/pipeline/filters/kubernetes.md
+++ b/pipeline/filters/kubernetes.md
@@ -37,6 +37,7 @@ The plugin supports the following configuration parameters:
| Keep\_Log | When `Keep_Log` is disabled, the `log` field is removed from the incoming message once it has been successfully merged \(`Merge_Log` must be enabled as well\). | On |
| tls.debug | Debug level between 0 \(nothing\) and 4 \(every detail\). | -1 |
| tls.verify | When enabled, turns on certificate validation when connecting to the Kubernetes API server. | On |
+| tls.verify\_hostname | When enabled, turns on hostname validation for certificates | Off |
| Use\_Journal | When enabled, the filter reads logs coming in Journald format. | Off |
| Cache\_Use\_Docker\_Id | When enabled, metadata will be fetched from K8s when docker\_id is changed. | Off |
| Regex\_Parser | Set an alternative Parser to process record Tag and extract pod\_name, namespace\_name, container\_name and docker\_id. The parser must be registered in a [parsers file](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf) \(refer to parser _filter-kube-test_ as an example\). | |
@@ -270,7 +271,7 @@ There are some configuration setup needed for this feature.
Role Configuration for Fluent Bit DaemonSet Example:
-```text
+```yaml
---
apiVersion: v1
kind: ServiceAccount
@@ -313,34 +314,34 @@ The difference is that kubelet need a special permission for resource `nodes/pro
Fluent Bit Configuration Example:
```text
- [INPUT]
- Name tail
- Tag kube.*
- Path /var/log/containers/*.log
- DB /var/log/flb_kube.db
- Parser docker
- Docker_Mode On
- Mem_Buf_Limit 50MB
- Skip_Long_Lines On
- Refresh_Interval 10
-
- [FILTER]
- Name kubernetes
- Match kube.*
- Kube_URL https://kubernetes.default.svc.cluster.local:443
- Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
- Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
- Merge_Log On
- Buffer_Size 0
- Use_Kubelet true
- Kubelet_Port 10250
+[INPUT]
+ Name tail
+ Tag kube.*
+ Path /var/log/containers/*.log
+ DB /var/log/flb_kube.db
+ Parser docker
+ Docker_Mode On
+ Mem_Buf_Limit 50MB
+ Skip_Long_Lines On
+ Refresh_Interval 10
+
+[FILTER]
+ Name kubernetes
+ Match kube.*
+ Kube_URL https://kubernetes.default.svc.cluster.local:443
+ Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+ Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
+ Merge_Log On
+ Buffer_Size 0
+ Use_Kubelet true
+ Kubelet_Port 10250
```
So for fluent bit configuration, you need to set the `Use_Kubelet` to true to enable this feature.
DaemonSet config Example:
-```text
+```yaml
---
apiVersion: apps/v1
kind: DaemonSet
diff --git a/pipeline/filters/log_to_metrics.md b/pipeline/filters/log_to_metrics.md
index 5dbc017d7..6e1630fbf 100644
--- a/pipeline/filters/log_to_metrics.md
+++ b/pipeline/filters/log_to_metrics.md
@@ -2,6 +2,8 @@
description: Generate metrics from logs
---
+
+
# Log To Metrics
The _Log To Metrics Filter_ plugin allows you to generate log-derived metrics. It currently supports modes to count records, provide a gauge for field values or create a histogram. You can also match or exclude specific records based on regular expression patterns for values or nested values. This filter plugin does not actually act as a record filter and does not change or drop records. All records will pass this filter untouched and generated metrics will be emitted into a seperate metric pipeline.
diff --git a/pipeline/filters/lua.md b/pipeline/filters/lua.md
index 09d56808a..4f1bab092 100644
--- a/pipeline/filters/lua.md
+++ b/pipeline/filters/lua.md
@@ -2,6 +2,8 @@
The **Lua** filter allows you to modify the incoming records (even split one record into multiple records) using custom [Lua](https://www.lua.org/) scripts.
+
+
Due to the necessity to have a flexible filtering mechanism, it is now possible to extend Fluent Bit capabilities by writing custom filters using Lua programming language. A Lua-based filter takes two steps:
1. Configure the Filter in the main configuration
@@ -194,12 +196,12 @@ We want to extract the `sandboxbsh` name and add it to our record as a special k
{% tabs %}
{% tab title="fluent-bit.conf" %}
```
- [FILTER]
- Name lua
- Alias filter-iots-lua
- Match iots_thread.*
- Script filters.lua
- Call set_landscape_deployment
+[FILTER]
+ Name lua
+ Alias filter-iots-lua
+ Match iots_thread.*
+ Script filters.lua
+ Call set_landscape_deployment
```
{% endtab %}
@@ -356,23 +358,23 @@ Configuration to get istio logs and apply response code filter to them.
{% tabs %}
{% tab title="fluent-bit.conf" %}
```ini
- [INPUT]
- Name tail
- Path /var/log/containers/*_istio-proxy-*.log
- multiline.parser docker, cri
- Tag istio.*
- Mem_Buf_Limit 64MB
- Skip_Long_Lines Off
-
- [FILTER]
- Name lua
- Match istio.*
- Script response_code_filter.lua
- call cb_response_code_filter
-
- [Output]
- Name stdout
- Match *
+[INPUT]
+ Name tail
+ Path /var/log/containers/*_istio-proxy-*.log
+ multiline.parser docker, cri
+ Tag istio.*
+ Mem_Buf_Limit 64MB
+ Skip_Long_Lines Off
+
+[FILTER]
+ Name lua
+ Match istio.*
+ Script response_code_filter.lua
+ call cb_response_code_filter
+
+[Output]
+ Name stdout
+ Match *
```
{% endtab %}
@@ -436,3 +438,134 @@ pipeline:
#### Output
In the output only the messages with response code 0 or greater than 399 are shown.
+
+
+### Timeformat Conversion
+
+The following example converts a field's specific type of `datetime` format to
+`utc ISO 8601` format.
+
+#### Lua script
+
+Script `custom_datetime_format.lua`
+
+```lua
+function convert_to_utc(tag, timestamp, record)
+ local date_time = record["pub_date"]
+ local new_record = record
+ if date_time then
+ if string.find(date_time, ",") then
+ local pattern = "(%a+, %d+ %a+ %d+ %d+:%d+:%d+) ([+-]%d%d%d%d)"
+ local date_part, zone_part = date_time:match(pattern)
+
+ if date_part and zone_part then
+ local command = string.format("date -u -d '%s %s' +%%Y-%%m-%%dT%%H:%%M:%%SZ", date_part, zone_part)
+ local handle = io.popen(command)
+ local result = handle:read("*a")
+ handle:close()
+ new_record["pub_date"] = result:match("%S+")
+ end
+ end
+ end
+ return 1, timestamp, new_record
+end
+```
+
+#### Configuration
+
+Use this configuration to obtain a JSON key with `datetime`, and then convert it to
+another format.
+
+{% tabs %}
+{% tab title="fluent-bit.conf" %}
+```ini
+[INPUT]
+ Name dummy
+ Dummy {"event": "Restock", "pub_date": "Tue, 30 Jul 2024 18:01:06 +0000"}
+ Tag event_category_a
+
+[INPUT]
+ Name dummy
+ Dummy {"event": "Soldout", "pub_date": "Mon, 29 Jul 2024 10:15:00 +0600"}
+ Tag event_category_b
+
+
+[FILTER]
+ Name lua
+ Match *
+ Script custom_datetime_format.lua
+ call convert_to_utc
+
+[Output]
+ Name stdout
+ Match *
+```
+{% endtab %}
+
+{% tab title="fluent-bit.yaml" %}
+```yaml
+pipeline:
+ inputs:
+ - name: dummy
+ dummy: '{"event": "Restock", "pub_date": "Tue, 30 Jul 2024 18:01:06 +0000"}'
+ tag: event_category_a
+
+ - name: dummy
+ dummy: '{"event": "Soldout", "pub_date": "Mon, 29 Jul 2024 10:15:00 +0600"}'
+ tag: event_category_b
+
+ filters:
+ - name: lua
+ match: '*'
+ code: |
+ function convert_to_utc(tag, timestamp, record)
+ local date_time = record["pub_date"]
+ local new_record = record
+ if date_time then
+ if string.find(date_time, ",") then
+ local pattern = "(%a+, %d+ %a+ %d+ %d+:%d+:%d+) ([+-]%d%d%d%d)"
+ local date_part, zone_part = date_time:match(pattern)
+ if date_part and zone_part then
+ local command = string.format("date -u -d '%s %s' +%%Y-%%m-%%dT%%H:%%M:%%SZ", date_part, zone_part)
+ local handle = io.popen(command)
+ local result = handle:read("*a")
+ handle:close()
+ new_record["pub_date"] = result:match("%S+")
+ end
+ end
+ end
+ return 1, timestamp, new_record
+ end
+ call: convert_to_utc
+
+ outputs:
+ - name: stdout
+ match: '*'
+```
+{% endtab %}
+{% endtabs %}
+
+#### Input
+
+```json
+{"event": "Restock", "pub_date": "Tue, 30 Jul 2024 18:01:06 +0000"}
+```
+and
+
+```json
+{"event": "Soldout", "pub_date": "Mon, 29 Jul 2024 10:15:00 +0600"}
+```
+Which are handled by dummy in this example.
+
+#### Output
+
+The output of this process shows the conversion of the `datetime` of two timezones to
+`ISO 8601` format in `UTC`.
+
+```ini
+...
+[2024/08/01 00:56:25] [ info] [output:stdout:stdout.0] worker #0 started
+[0] event_category_a: [[1722452186.727104902, {}], {"event"=>"Restock", "pub_date"=>"2024-07-30T18:01:06Z"}]
+[0] event_category_b: [[1722452186.730255842, {}], {"event"=>"Soldout", "pub_date"=>"2024-07-29T04:15:00Z"}]
+...
+```
\ No newline at end of file
diff --git a/pipeline/filters/nest.md b/pipeline/filters/nest.md
index 96990ca81..b0262d324 100644
--- a/pipeline/filters/nest.md
+++ b/pipeline/filters/nest.md
@@ -1,15 +1,16 @@
# Nest
-The _Nest Filter_ plugin allows you to operate on or with nested data. Its modes of operation are
+The _Nest Filter_ plugin lets you operate on or with nested data. Its modes of operation are:
-* `nest` - Take a set of records and place them in a map
-* `lift` - Take a map by key and lift its records up
+- `nest` - Take a set of records and place them in a map.
+- `lift` - Take a map by key and lift its records up.
-## Example usage \(nest\)
+## Example usage for `nest`
-As an example using JSON notation, to nest keys matching the `Wildcard` value `Key*` under a new key `NestKey` the transformation becomes,
+As an example using JSON notation, to nest keys matching the `Wildcard` value `Key*`
+under a new key `NestKey` the transformation becomes:
-_Example \(input\)_
+Input:
```text
{
@@ -19,7 +20,7 @@ _Example \(input\)_
}
```
-_Example \(output\)_
+Output:
```text
{
@@ -31,11 +32,12 @@ _Example \(output\)_
}
```
-## Example usage \(lift\)
+## Example usage for `lift`
-As an example using JSON notation, to lift keys nested under the `Nested_under` value `NestKey*` the transformation becomes,
+As an example using JSON notation, to lift keys nested under the `Nested_under` value
+`NestKey*` the transformation becomes:
-_Example \(input\)_
+Input:
```text
{
@@ -47,7 +49,7 @@ _Example \(input\)_
}
```
-_Example \(output\)_
+Output:
```text
{
@@ -61,40 +63,47 @@ _Example \(output\)_
The plugin supports the following configuration parameters:
-| Key | Value Format | Operation | Description |
+| Key | Value format | Operation | Description |
| :--- | :--- | :--- | :--- |
-| Operation | ENUM \[`nest` or `lift`\] | | Select the operation `nest` or `lift` |
-| Wildcard | FIELD WILDCARD | `nest` | Nest records which field matches the wildcard |
-| Nest\_under | FIELD STRING | `nest` | Nest records matching the `Wildcard` under this key |
-| Nested\_under | FIELD STRING | `lift` | Lift records nested under the `Nested_under` key |
-| Add\_prefix | FIELD STRING | ANY | Prefix affected keys with this string |
-| Remove\_prefix | FIELD STRING | ANY | Remove prefix from affected keys if it matches this string |
+| `Operation` | ENUM [`nest` or `lift`] | | Select the operation `nest` or `lift` |
+| `Wildcard` | FIELD WILDCARD | `nest` | Nest records which field matches the wildcard |
+| `Nest_under` | FIELD STRING | `nest` | Nest records matching the `Wildcard` under this key |
+| `Nested_under` | FIELD STRING | `lift` | Lift records nested under the `Nested_under` key |
+| `Add_prefix` | FIELD STRING | ANY | Prefix affected keys with this string |
+| `Remove_prefix` | FIELD STRING | ANY | Remove prefix from affected keys if it matches this string |
## Getting Started
-In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the [Memory Usage Input Plugin](../inputs/memory-metrics.md), which outputs the following \(example\),
+To start filtering records, run the filter from the command line or through the
+configuration file. The following example invokes the
+[Memory Usage Input Plugin](../inputs/memory-metrics.md), which outputs the
+following:
```text
[0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
```
-## Example \#1 - nest
+## Example 1 - nest
### Command Line
-> Note: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.
+Using command line mode requires quotes to parse the wildcard properly. The use
+of a configuration file is recommended.
-The following command will load the _mem_ plugin. Then the _nest_ filter will match the wildcard rule to the keys and nest the keys matching `Mem.*` under the new key `NEST`.
+The following command loads the _mem_ plugin. Then the _nest_ filter matches the
+wildcard rule to the keys and nests the keys matching `Mem.*` under the new key
+`NEST`.
-```text
-$ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout
+```shell copy
+bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout
```
### Configuration File
{% tabs %}
{% tab title="fluent-bit.conf" %}
-```python
+
+```python copy
[INPUT]
Name mem
Tag mem.local
@@ -111,10 +120,12 @@ $ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildc
Nest_under Memstats
Remove_prefix Mem.
```
+
{% endtab %}
{% tab title="fluent-bit.yaml" %}
-```yaml
+
+```yaml copy
pipeline:
inputs:
- name: mem
@@ -130,6 +141,7 @@ pipeline:
- name: stdout
match: '*'
```
+
{% endtab %}
{% endtabs %}
@@ -142,15 +154,17 @@ The output of both the command line and configuration invocations should be iden
[0] mem.local: [1522978514.007359767, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Memstats"=>{"total"=>4050908, "used"=>714984, "free"=>3335924}}]
```
-## Example \#2 - nest and lift undo
+## Example 2 - nest and lift undo
-This example nests all `Mem.*` and `Swap,*` items under the `Stats` key and then reverses these actions with a `lift` operation. The output appears unchanged.
+This example nests all `Mem.*` and `Swap.*` items under the `Stats` key and then
+reverses these actions with a `lift` operation. The output appears unchanged.
-### Configuration File
+### Example 2 Configuration File
{% tabs %}
{% tab title="fluent-bit.conf" %}
-```python
+
+```python copy
[INPUT]
Name mem
Tag mem.local
@@ -175,10 +189,11 @@ This example nests all `Mem.*` and `Swap,*` items under the `Stats` key and then
Nested_under Stats
Remove_prefix NESTED
```
-{% endtab %}
+{% endtab %}
{% tab title="fluent-bit.yaml" %}
-```yaml
+
+```yaml copy
pipeline:
inputs:
- name: mem
@@ -201,6 +216,7 @@ pipeline:
- name: stdout
match: '*'
```
+
{% endtab %}
{% endtabs %}
@@ -211,15 +227,17 @@ pipeline:
[0] mem.local: [1529566958.000940636, {"Mem.total"=>8053656, "Mem.used"=>6940380, "Mem.free"=>1113276, "Swap.total"=>16532988, "Swap.used"=>1286772, "Swap.free"=>15246216}]
```
-## Example \#3 - nest 3 levels deep
+## Example 3 - nest 3 levels deep
-This example takes the keys starting with `Mem.*` and nests them under `LAYER1`, which itself is then nested under `LAYER2`, which is nested under `LAYER3`.
+This example takes the keys starting with `Mem.*` and nests them under `LAYER1`,
+which is then nested under `LAYER2`, which is nested under `LAYER3`.
-### Configuration File
+### Example 3 Configuration File
{% tabs %}
{% tab title="fluent-bit.conf" %}
-```python
+
+```python copy
[INPUT]
Name mem
Tag mem.local
@@ -249,10 +267,11 @@ This example takes the keys starting with `Mem.*` and nests them under `LAYER1`,
Wildcard LAYER2*
Nest_under LAYER3
```
-{% endtab %}
+{% endtab %}
{% tab title="fluent-bit.yaml" %}
-```yaml
+
+```yaml copy
pipeline:
inputs:
- name: mem
@@ -277,6 +296,7 @@ pipeline:
- name: stdout
match: '*'
```
+
{% endtab %}
{% endtabs %}
@@ -302,15 +322,19 @@ pipeline:
}
```
-## Example \#4 - multiple nest and lift filters with prefix
+## Example 4 - multiple nest and lift filters with prefix
-This example starts with the 3-level deep nesting of _Example 2_ and applies the `lift` filter three times to reverse the operations. The end result is that all records are at the top level, without nesting, again. One prefix is added for each level that is lifted.
+This example uses the 3-level deep nesting of _Example 2_ and applies the
+`lift` filter three times to reverse the operations. The end result is that all
+records are at the top level, without nesting, again. One prefix is added for each
+level that's lifted.
### Configuration file
{% tabs %}
{% tab title="fluent-bit.conf" %}
-```python
+
+```python copy
[INPUT]
Name mem
Tag mem.local
@@ -361,10 +385,12 @@ This example starts with the 3-level deep nesting of _Example 2_ and applies the
Nested_under Lifted3_Lifted2_LAYER1
Add_prefix Lifted3_Lifted2_Lifted1_
```
+
{% endtab %}
{% tab title="fluent-bit.yaml" %}
-```yaml
+
+```yaml copy
pipeline:
inputs:
- name: mem
@@ -404,23 +430,21 @@ pipeline:
- name: stdout
match: '*'
```
+
{% endtab %}
{% endtabs %}
-
### Result
```text
[0] mem.local: [1524862951.013414798, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996}]
-
{
- "Swap.total"=>1046524,
- "Swap.used"=>0,
- "Swap.free"=>1046524,
- "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908,
- "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912,
+ "Swap.total"=>1046524,
+ "Swap.used"=>0,
+ "Swap.free"=>1046524,
+ "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908,
+ "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912,
"Lifted3_Lifted2_Lifted1_Mem.free"=>2796996
}
```
-
diff --git a/pipeline/filters/record-modifier.md b/pipeline/filters/record-modifier.md
index 0a1f74572..dc64aea7c 100644
--- a/pipeline/filters/record-modifier.md
+++ b/pipeline/filters/record-modifier.md
@@ -1,24 +1,26 @@
# Record Modifier
-The _Record Modifier Filter_ plugin allows to append fields or to exclude specific fields.
+The _Record Modifier_ [filter](pipeline/filters.md) plugin lets you append
+fields to a record, or exclude specific fields.
-## Configuration Parameters
+## Configuration parameters
-The plugin supports the following configuration parameters: _Remove\_key_ and _Allowlist\_key_ are exclusive.
+The plugin supports the following configuration parameters:
| Key | Description |
| :--- | :--- |
-| Record | Append fields. This parameter needs key and value pair. |
-| Remove\_key | If the key is matched, that field is removed. |
-| Allowlist\_key | If the key is **not** matched, that field is removed. |
-| Whitelist\_key | An alias of `Allowlist_key` for backwards compatibility. |
-| Uuid\_key| If set, the plugin appends uuid to each record. The value assigned becomes the key in the map.|
+| `Record` | Append fields. This parameter needs a key/value pair. |
+| `Remove_key` | If the key is matched, that field is removed. You can this or `Allowlist_key`.|
+| `Allowlist_key` | If the key isn't matched, that field is removed. You can this or `Remove_key`. |
+| `Whitelist_key` | An alias of `Allowlist_key` for backwards compatibility. |
+| `Uuid_key` | If set, the plugin appends Uuid to each record. The value assigned becomes the key in the map. |
-## Getting Started
+## Get started
-In order to start filtering records, you can run the filter from the command line or through the configuration file.
+To start filtering records, run the filter from the command line or through a
+configuration file.
-This is a sample in\_mem record to filter.
+This is a sample `in_mem` record to filter.
```text
{"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724}
@@ -26,11 +28,13 @@ This is a sample in\_mem record to filter.
### Append fields
-The following configuration file is to append product name and hostname \(via environment variable\) to record.
+The following configuration file appends a product name and hostname to a record
+using an environment variable:
{% tabs %}
{% tab title="fluent-bit.conf" %}
-```python
+
+```python copy
[INPUT]
Name mem
Tag mem.local
@@ -45,10 +49,12 @@ The following configuration file is to append product name and hostname \(via en
Record hostname ${HOSTNAME}
Record product Awesome_Tool
```
+
{% endtab %}
{% tab title="fluent-bit.yaml" %}
-```yaml
+
+```yaml copy
pipeline:
inputs:
- name: mem
@@ -56,37 +62,37 @@ pipeline:
filters:
- name: record_modifier
match: '*'
- record:
+ record:
- hostname ${HOSTNAME}
- product Awesome_Tool
outputs:
- name: stdout
match: '*'
```
+
{% endtab %}
{% endtabs %}
+You can run the filter from command line:
-You can also run the filter from command line.
-
-```text
-$ fluent-bit -i mem -o stdout -F record_modifier -p 'Record=hostname ${HOSTNAME}' -p 'Record=product Awesome_Tool' -m '*'
+```shell copy
+fluent-bit -i mem -o stdout -F record_modifier -p 'Record=hostname ${HOSTNAME}' -p 'Record=product Awesome_Tool' -m '*'
```
-The output will be
+The output looks something like:
-```python
+```python copy
[0] mem.local: [1492436882.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724, "hostname"=>"localhost.localdomain", "product"=>"Awesome_Tool"}]
```
-### Remove fields with Remove\_key
-
-The following configuration file is to remove 'Swap.\*' fields.
+### Remove fields with `Remove_key`
+The following configuration file removes `Swap.*` fields:
{% tabs %}
{% tab title="fluent-bit.conf" %}
-```python
+
+```python copy
[INPUT]
Name mem
Tag mem.local
@@ -102,10 +108,12 @@ The following configuration file is to remove 'Swap.\*' fields.
Remove_key Swap.used
Remove_key Swap.free
```
+
{% endtab %}
{% tab title="fluent-bit.yaml" %}
-```yaml
+
+```yaml copy
pipeline:
inputs:
- name: mem
@@ -113,7 +121,7 @@ pipeline:
filters:
- name: record_modifier
match: '*'
- remove_key:
+ remove_key:
- Swap.total
- Swap.used
- Swap.free
@@ -121,28 +129,30 @@ pipeline:
- name: stdout
match: '*'
```
+
{% endtab %}
{% endtabs %}
You can also run the filter from command line.
-```text
-$ fluent-bit -i mem -o stdout -F record_modifier -p 'Remove_key=Swap.total' -p 'Remove_key=Swap.free' -p 'Remove_key=Swap.used' -m '*'
+```shell copy
+fluent-bit -i mem -o stdout -F record_modifier -p 'Remove_key=Swap.total' -p 'Remove_key=Swap.free' -p 'Remove_key=Swap.used' -m '*'
```
-The output will be
+The output looks something like:
```python
[0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]
```
-### Remove fields with Allowlist\_key
+### Retain fields with `Allowlist_key`
-The following configuration file is to remain 'Mem.\*' fields.
+The following configuration file retains `Mem.*` fields.
{% tabs %}
{% tab title="fluent-bit.conf" %}
-```python
+
+```python copy
[INPUT]
Name mem
Tag mem.local
@@ -158,10 +168,12 @@ The following configuration file is to remain 'Mem.\*' fields.
Allowlist_key Mem.used
Allowlist_key Mem.free
```
+
{% endtab %}
{% tab title="fluent-bit.yaml" %}
-```yaml
+
+```yaml copy
pipeline:
inputs:
- name: mem
@@ -169,7 +181,7 @@ pipeline:
filters:
- name: record_modifier
match: '*'
- Allowlist_key:
+ Allowlist_key:
- Mem.total
- Mem.used
- Mem.free
@@ -177,18 +189,18 @@ pipeline:
- name: stdout
match: '*'
```
+
{% endtab %}
{% endtabs %}
-You can also run the filter from command line.
+You can also run the filter from command line:
-```text
-$ fluent-bit -i mem -o stdout -F record_modifier -p 'Allowlist_key=Mem.total' -p 'Allowlist_key=Mem.free' -p 'Allowlist_key=Mem.used' -m '*'
+```shell copy
+fluent-bit -i mem -o stdout -F record_modifier -p 'Allowlist_key=Mem.total' -p 'Allowlist_key=Mem.free' -p 'Allowlist_key=Mem.used' -m '*'
```
-The output will be
+The output looks something like:
```python
[0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]
```
-
diff --git a/pipeline/filters/rewrite-tag.md b/pipeline/filters/rewrite-tag.md
index 17574ad2a..72a991c1b 100644
--- a/pipeline/filters/rewrite-tag.md
+++ b/pipeline/filters/rewrite-tag.md
@@ -59,7 +59,7 @@ If we wanted to match against the value of the key `name` we must use `$name`. T
* `$name` = "abc-123"
* `$ss['s1']['s2']` = "flb"
-Note that a key must point a value that contains a string, it's **not valid** for numbers, booleans, maps or arrays.
+Note that a key must point to a value that contains a string, it's **not valid** for numbers, booleans, maps or arrays.
### Regex
diff --git a/pipeline/filters/type-converter.md b/pipeline/filters/type-converter.md
index b185467f8..d55d5be40 100644
--- a/pipeline/filters/type-converter.md
+++ b/pipeline/filters/type-converter.md
@@ -2,6 +2,8 @@
The _Type Converter Filter_ plugin allows to convert data type and append new key value pair.
+
+
This plugin is useful in combination with plugins which expect incoming string value.
e.g. [filter_grep](grep.md), [filter_modify](modify.md)
diff --git a/pipeline/filters/wasm.md b/pipeline/filters/wasm.md
index c311447ba..0140bc28d 100644
--- a/pipeline/filters/wasm.md
+++ b/pipeline/filters/wasm.md
@@ -21,7 +21,9 @@ The plugin supports the following configuration parameters:
| Wasm\_Path | Path to the built Wasm program that will be used. This can be a relative path against the main configuration file. |
| Event\_Format | Define event format to interact with Wasm programs: msgpack or json. Default: json |
| Function\_Name | Wasm function name that will be triggered to do filtering. It's assumed that the function is built inside the Wasm program specified above. |
-| Accessible\_Paths | Specify the whilelist of paths to be able to access paths from WASM programs. |
+| Accessible\_Paths | Specify the whitelist of paths to be able to access paths from WASM programs. |
+| Wasm\_Heap\_Size | Size of the heap size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values. |
+| Wasm\_Stack\_Size | Size of the stack size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values. |
## Configuration Examples
diff --git a/pipeline/inputs/collectd.md b/pipeline/inputs/collectd.md
index acbf773c1..b10ea73fd 100644
--- a/pipeline/inputs/collectd.md
+++ b/pipeline/inputs/collectd.md
@@ -11,6 +11,7 @@ The plugin supports the following configuration parameters:
| Listen | Set the address to listen to | 0.0.0.0 |
| Port | Set the port to listen to | 25826 |
| TypesDB | Set the data specification file | /usr/share/collectd/types.db |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
## Configuration Examples
@@ -31,4 +32,3 @@ Here is a basic configuration example.
With this configuration, Fluent Bit listens to `0.0.0.0:25826`, and outputs incoming datagram packets to stdout.
You must set the same types.db files that your collectd server uses. Otherwise, Fluent Bit may not be able to interpret the payload properly.
-
diff --git a/pipeline/inputs/cpu-metrics.md b/pipeline/inputs/cpu-metrics.md
index 3c296f0cd..c54558cbf 100644
--- a/pipeline/inputs/cpu-metrics.md
+++ b/pipeline/inputs/cpu-metrics.md
@@ -4,13 +4,15 @@ The **cpu** input plugin, measures the CPU usage of a process or the whole syste
The following tables describes the information generated by the plugin. The keys below represent the data used by the overall system, all values associated to the keys are in a percentage unit \(0 to 100%\):
-The CPU metrics plugin creates metrics that are log-based \(I.e. JSON payload\). If you are looking for Prometheus-based metrics please see the Node Exporter Metrics input plugin.
+The CPU metrics plugin creates metrics that are log-based, such as JSON payload. For
+Prometheus-based metrics, see the Node Exporter Metrics input plugin.
| key | description |
| :--- | :--- |
| cpu\_p | CPU usage of the overall system, this value is the summation of time spent on user and kernel space. The result takes in consideration the numbers of CPU cores in the system. |
| user\_p | CPU usage in User mode, for short it means the CPU usage by user space programs. The result of this value takes in consideration the numbers of CPU cores in the system. |
| system\_p | CPU usage in Kernel mode, for short it means the CPU usage by the Kernel. The result of this value takes in consideration the numbers of CPU cores in the system. |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). Default: `false`. |
In addition to the keys reported in the above table, a similar content is created **per** CPU core. The cores are listed from _0_ to _N_ as the Kernel reports:
diff --git a/pipeline/inputs/disk-io-metrics.md b/pipeline/inputs/disk-io-metrics.md
index 024399314..c28cc4acf 100644
--- a/pipeline/inputs/disk-io-metrics.md
+++ b/pipeline/inputs/disk-io-metrics.md
@@ -2,7 +2,8 @@
The **disk** input plugin, gathers the information about the disk throughput of the running system every certain interval of time and reports them.
-The Disk I/O metrics plugin creates metrics that are log-based \(I.e. JSON payload\). If you are looking for Prometheus-based metrics please see the Node Exporter Metrics input plugin.
+The Disk I/O metrics plugin creates metrics that are log-based, such as JSON payload.
+For Prometheus-based metrics, see the Node Exporter Metrics input plugin.
## Configuration Parameters
@@ -13,6 +14,7 @@ The plugin supports the following configuration parameters:
| Interval\_Sec | Polling interval \(seconds\). | 1 |
| Interval\_NSec | Polling interval \(nanosecond\). | 0 |
| Dev\_Name | Device name to limit the target. \(e.g. sda\). If not set, _in\_disk_ gathers information from all of disks and partitions. | all disks |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
## Getting Started
@@ -72,4 +74,3 @@ pipeline:
Note: Total interval \(sec\) = Interval\_Sec + \(Interval\_Nsec / 1000000000\).
e.g. 1.5s = 1s + 500000000ns
-
diff --git a/pipeline/inputs/docker-events.md b/pipeline/inputs/docker-events.md
index 6e850c1ee..40d85ec5c 100644
--- a/pipeline/inputs/docker-events.md
+++ b/pipeline/inputs/docker-events.md
@@ -14,6 +14,7 @@ This plugin supports the following configuration parameters:
| Key | When a message is unstructured \(no parser applied\), it's appended as a string under the key name _message_. | message |
| Reconnect.Retry_limits| The maximum number of retries allowed. The plugin tries to reconnect with docker socket when EOF is detected. | 5 |
| Reconnect.Retry_interval| The retrying interval. Unit is second. | 1 |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
### Command Line
diff --git a/pipeline/inputs/docker-metrics.md b/pipeline/inputs/docker-metrics.md
index 1102e576f..ce044604e 100644
--- a/pipeline/inputs/docker-metrics.md
+++ b/pipeline/inputs/docker-metrics.md
@@ -6,12 +6,7 @@ description: >-
# Docker Metrics
-Content:
-
-* [Configuration Parameters](https://app.gitbook.com/s/-LKKSx-3LBTCtaHbg0gl-887967055/pipeline/inputs/docker.md#configuration-parameters)
-* [Configuration File](https://app.gitbook.com/s/-LKKSx-3LBTCtaHbg0gl-887967055/pipeline/inputs/docker.md#configuration-file)
-
-### Configuration Parameters
+## Configuration Parameters
The plugin supports the following configuration parameters:
@@ -20,10 +15,11 @@ The plugin supports the following configuration parameters:
| Interval_Sec | Polling interval in seconds | 1 |
| Include | A space-separated list of containers to include | |
| Exclude | A space-separated list of containers to exclude | |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
If you set neither `Include` nor `Exclude`, the plugin will try to get metrics from _all_ the running containers.
-### Configuration File
+## Configuration File
Here is an example configuration that collects metrics from two docker instances (`6bab19c3a0f9` and `14159be4ca2c`).
diff --git a/pipeline/inputs/dummy.md b/pipeline/inputs/dummy.md
index 745bacaa2..48177ac10 100644
--- a/pipeline/inputs/dummy.md
+++ b/pipeline/inputs/dummy.md
@@ -6,18 +6,19 @@ The **dummy** input plugin, generates dummy events. It is useful for testing, de
The plugin supports the following configuration parameters:
-| Key | Description |
-| :--- | :--- |
-| Dummy | Dummy JSON record. Default: `{"message":"dummy"}` |
-| Metadata | Dummy JSON metadata. Default: `{}` |
-| Start\_time\_sec | Dummy base timestamp in seconds. Default: 0 |
-| Start\_time\_nsec | Dummy base timestamp in nanoseconds. Default: 0 |
-| Rate | Rate at which messages are generated expressed in how many times per second. Default: 1 |
-| Interval\_sec | Set seconds of time interval at which every message is generated. If set, `Rate` configuration will be ignored. Default: 0 |
-| Interval\_nsec | Set nanoseconds of time interval at which every message is generated. If set, `Rate` configuration will be ignored. Default: 0 |
-| Samples | If set, the events number will be limited. e.g. If Samples=3, the plugin only generates three events and stops. |
-| Copies | Number of messages to generate each time they are generated. Defaults to 1. |
-| Flush\_on\_startup | If set to `true`, the first dummy event is generated at startup. Default: `false` |
+| Key | Description | Default |
+| :----------------- | :---------- | :------ |
+| Dummy | Dummy JSON record. | `{"message":"dummy"}` |
+| Metadata | Dummy JSON metadata. | `{}` |
+| Start\_time\_sec | Dummy base timestamp, in seconds. | `0` |
+| Start\_time\_nsec | Dummy base timestamp, in nanoseconds. | `0` |
+| Rate | Rate at which messages are generated expressed in how many times per second. | `1` |
+| Interval\_sec | Set time interval, in seconds, at which every message is generated. If set, `Rate` configuration is ignored. | `0` |
+| Interval\_nsec | Set time interval, in nanoseconds, at which every message is generated. If set, `Rate` configuration is ignored. | `0` |
+| Samples | If set, the events number will be limited. For example, if Samples=3, the plugin generates only three events and stops. | _none_ |
+| Copies | Number of messages to generate each time they are generated. | `1` |
+| Flush\_on\_startup | If set to `true`, the first dummy event is generated at startup. | `false` |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
## Getting Started
diff --git a/pipeline/inputs/elasticsearch.md b/pipeline/inputs/elasticsearch.md
index 27f659095..08a67d795 100644
--- a/pipeline/inputs/elasticsearch.md
+++ b/pipeline/inputs/elasticsearch.md
@@ -14,6 +14,7 @@ The plugin supports the following configuration parameters:
| meta\_key | Specify a key name for meta information. | "@meta" |
| hostname | Specify hostname or FQDN. This parameter can be used for "sniffing" (auto-discovery of) cluster node information. | "localhost" |
| version | Specify Elasticsearch server version. This parameter is effective for checking a version of Elasticsearch/OpenSearch server version. | "8.0.0" |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
**Note:** The Elasticsearch cluster uses "sniffing" to optimize the connections between its cluster and clients.
Elasticsearch can build its cluster and dynamically generate a connection list which is called "sniffing".
diff --git a/pipeline/inputs/exec-wasi.md b/pipeline/inputs/exec-wasi.md
index b094f3a01..dabd0a065 100644
--- a/pipeline/inputs/exec-wasi.md
+++ b/pipeline/inputs/exec-wasi.md
@@ -10,11 +10,14 @@ The plugin supports the following configuration parameters:
| :--- | :--- |
| WASI\_Path | The place of a WASM program file. |
| Parser | Specify the name of a parser to interpret the entry as a structured message. |
-| Accessible\_Paths | Specify the whilelist of paths to be able to access paths from WASM programs. |
+| Accessible\_Paths | Specify the whitelist of paths to be able to access paths from WASM programs. |
| Interval\_Sec | Polling interval \(seconds\). |
| Interval\_NSec | Polling interval \(nanosecond\). |
-| Buf\_Size | Size of the buffer \(check [unit sizes](https://docs.fluentbit.io/manual/configuration/unit_sizes) for allowed values\) |
+| Wasm\_Heap\_Size | Size of the heap size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values. |
+| Wasm\_Stack\_Size | Size of the stack size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values. |
+| Buf\_Size | Size of the buffer \(check [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values\) |
| Oneshot | Only run once at startup. This allows collection of data precedent to fluent-bit's startup (bool, default: false) |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). Default: `false`. |
## Configuration Examples
diff --git a/pipeline/inputs/exec.md b/pipeline/inputs/exec.md
index 2f7a32416..73b4c3450 100644
--- a/pipeline/inputs/exec.md
+++ b/pipeline/inputs/exec.md
@@ -21,10 +21,11 @@ The plugin supports the following configuration parameters:
| Parser | Specify the name of a parser to interpret the entry as a structured message. |
| Interval\_Sec | Polling interval \(seconds\). |
| Interval\_NSec | Polling interval \(nanosecond\). |
-| Buf\_Size | Size of the buffer \(check [unit sizes](https://docs.fluentbit.io/manual/configuration/unit_sizes) for allowed values\) |
+| Buf\_Size | Size of the buffer \(check [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values\) |
| Oneshot | Only run once at startup. This allows collection of data precedent to fluent-bit's startup (bool, default: false) |
| Exit\_After\_Oneshot | Exit as soon as the one-shot command exits. This allows the exec plugin to be used as a wrapper for another command, sending the target command's output to any fluent-bit sink(s) then exiting. (bool, default: false) |
| Propagate\_Exit\_Code | When exiting due to Exit\_After\_Oneshot, cause fluent-bit to exit with the exit code of the command exited by this plugin. Follows [shell conventions for exit code propagation](https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html). (bool, default: false) |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). Default: `false`. |
## Getting Started
diff --git a/pipeline/inputs/fluentbit-metrics.md b/pipeline/inputs/fluentbit-metrics.md
index 9ef8c604e..358ac92b3 100644
--- a/pipeline/inputs/fluentbit-metrics.md
+++ b/pipeline/inputs/fluentbit-metrics.md
@@ -12,12 +12,13 @@ They can be sent to output plugins including [Prometheus Exporter](../outputs/pr
**Important note:** Metrics collected with Node Exporter Metrics flow through a separate pipeline from logs and current filters do not operate on top of metrics.
-## Configuration
+## Configuration
| Key | Description | Default |
| --------------- | --------------------------------------------------------------------------------------------------------- | --------- |
| scrape_interval | The rate at which metrics are collected from the host operating system | 2 seconds |
| scrape_on_start | Scrape metrics upon start, useful to avoid waiting for 'scrape_interval' for the first round of metrics. | false |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
## Getting Started
diff --git a/pipeline/inputs/forward.md b/pipeline/inputs/forward.md
index 2878f5b8b..9e7a61599 100644
--- a/pipeline/inputs/forward.md
+++ b/pipeline/inputs/forward.md
@@ -20,6 +20,7 @@ The plugin supports the following configuration parameters:
| Shared\_Key | Shared key for secure forward authentication. | |
| Self\_Hostname | Hostname for secure forward authentication. | |
| Security.Users | Specify the username and password pairs for secure forward authentication. | |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
## Getting Started
@@ -81,7 +82,7 @@ pipeline:
Since Fluent Bit v3, in\_forward can handle secure forward protocol.
-For using user-password authentication, it needs to specify `secutiry.users` at least an one-pair.
+For using user-password authentication, it needs to specify `security.users` at least an one-pair.
For using shared key, it needs to specify `shared_key` in both of forward output and forward input.
`self_hostname` is not able to specify with the same hostname between fluent servers and clients.
@@ -142,4 +143,3 @@ Copyright (C) Treasure Data
[2016/10/07 21:49:40] [ info] [in_fw] binding 0.0.0.0:24224
[0] my_tag: [1475898594, {"key 1"=>123456789, "key 2"=>"abcdefg"}]
```
-
diff --git a/pipeline/inputs/head.md b/pipeline/inputs/head.md
index b5f4496b4..6537f0ee2 100644
--- a/pipeline/inputs/head.md
+++ b/pipeline/inputs/head.md
@@ -16,6 +16,7 @@ The plugin supports the following configuration parameters:
| Key | Rename a key. Default: head. |
| Lines | Line number to read. If the number N is set, in\_head reads first N lines like head\(1\) -n. |
| Split\_line | If enabled, in\_head generates key-value pair per line. |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). Default: `false`. |
### Split Line Mode
@@ -84,7 +85,7 @@ pipeline:
Output is
```bash
-$ bin/fluent-bit -c head.conf
+$ bin/fluent-bit -c head.conf
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
@@ -162,4 +163,3 @@ pipeline:
Note: Total interval \(sec\) = Interval\_Sec + \(Interval\_Nsec / 1000000000\).
e.g. 1.5s = 1s + 500000000ns
-
diff --git a/pipeline/inputs/health.md b/pipeline/inputs/health.md
index 0e5694338..51028e735 100644
--- a/pipeline/inputs/health.md
+++ b/pipeline/inputs/health.md
@@ -15,6 +15,7 @@ The plugin supports the following configuration parameters:
| Alert | If enabled, it will only generate messages if the target TCP service is down. By default this option is disabled. |
| Add\_Host | If enabled, hostname is appended to each records. Default value is _false_. |
| Add\_Port | If enabled, port number is appended to each records. Default value is _false_. |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). Default: `false`. |
## Getting Started
@@ -87,4 +88,3 @@ Fluent Bit v1.8.0
[2] health.0: [1624145990.306498573, {"alive"=>true}]
[3] health.0: [1624145991.305595498, {"alive"=>true}]
```
-
diff --git a/pipeline/inputs/http.md b/pipeline/inputs/http.md
index 2e8afc5aa..52150a24b 100644
--- a/pipeline/inputs/http.md
+++ b/pipeline/inputs/http.md
@@ -15,6 +15,7 @@ description: The HTTP input plugin allows you to send custom records to an HTTP
| buffer_chunk_size | This sets the chunk size for incoming incoming JSON messages. These chunks are then stored/managed in the space available by buffer_max_size. | 512K |
| successful_response_code | It allows to set successful response code. `200`, `201` and `204` are supported. | 201 |
| success_header | Add an HTTP header key/value pair on success. Multiple headers can be set. Example: `X-Custom custom-answer` | |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
### TLS / SSL
diff --git a/pipeline/inputs/kafka.md b/pipeline/inputs/kafka.md
index e779b6f07..a0c83b97d 100644
--- a/pipeline/inputs/kafka.md
+++ b/pipeline/inputs/kafka.md
@@ -14,8 +14,8 @@ This plugin uses the official [librdkafka C library](https://github.com/edenhill
| group\_id | Group id passed to librdkafka. | fluent-bit |
| poll\_ms | Kafka brokers polling interval in milliseconds. | 500 |
| Buffer\_Max\_Size | Specify the maximum size of buffer per cycle to poll kafka messages from subscribed topics. To increase throughput, specify larger size. | 4M |
-| poll\_ms | Kafka brokers polling interval in milliseconds. | 500 |
| rdkafka.{property} | `{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) | |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
## Getting Started
@@ -46,7 +46,8 @@ In your main configuration file append the following _Input_ & _Output_ sections
#### Example of using kafka input/output plugins
-The fluent-bit source repository contains a full example of using fluent-bit to process kafka records:
+The Fluent Bit source repository contains a full example of using Fluent Bit to
+process Kafka records:
```text
[INPUT]
diff --git a/pipeline/inputs/kernel-logs.md b/pipeline/inputs/kernel-logs.md
index 7fa9cf143..9614391a5 100644
--- a/pipeline/inputs/kernel-logs.md
+++ b/pipeline/inputs/kernel-logs.md
@@ -7,6 +7,7 @@ The **kmsg** input plugin reads the Linux Kernel log buffer since the beginning,
| Key | Description | Default |
| :--- | :--- | :--- |
| Prio_Level | The log level to filter. The kernel log is dropped if its priority is more than prio_level. Allowed values are 0-8. Default is 8. 8 means all logs are saved. | 8 |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
## Getting Started
@@ -60,4 +61,3 @@ pipeline:
```
{% endtab %}
{% endtabs %}
-
diff --git a/pipeline/inputs/kubernetes-events.md b/pipeline/inputs/kubernetes-events.md
index 164c4eb60..1bfb7d134 100644
--- a/pipeline/inputs/kubernetes-events.md
+++ b/pipeline/inputs/kubernetes-events.md
@@ -14,8 +14,8 @@ Kubernetes exports it events through the API server. This input plugin allows to
|---------------------|---------------------------------------------------------------------------------------|------------------------------------------------------|
| db | Set a database file to keep track of recorded Kubernetes events | |
| db.sync | Set a database sync method. values: extra, full, normal and off | normal |
-| interval_sec | Set the polling interval for each channel. | 0 |
-| interval_nsec | Set the polling interval for each channel (sub seconds: nanoseconds) | 500000000 |
+| interval_sec | Set the reconnect interval (seconds)* | 0 |
+| interval_nsec | Set the reconnect interval (sub seconds: nanoseconds)* | 500000000 |
| kube_url | API Server end-point | https://kubernetes.default.svc |
| kube_ca_file | Kubernetes TLS CA file | /var/run/secrets/kubernetes.io/serviceaccount/ca.crt |
| kube_ca_path | Kubernetes TLS ca path | |
@@ -28,26 +28,45 @@ Kubernetes exports it events through the API server. This input plugin allows to
| tls.verify | Enable or disable verification of TLS peer certificate. | On |
| tls.vhost | Set optional TLS virtual host. | |
+
+- _* As of Fluent-Bit 3.1, this plugin uses a Kubernetes watch stream instead of polling. In versions before 3.1, the interval parameters are used for reconnecting the Kubernetes watch stream._
+
+## Threading
+
+This input always runs in its own [thread](../../administration/multithreading.md#inputs).
+
## Getting Started
+### Kubernetes Service Account
+The Kubernetes service account used by Fluent Bit must have `get`, `list`, and `watch`
+permissions to `namespaces` and `pods` for the namespaces watched in the
+`kube_namespace` configuration parameter. If you're using the helm chart to configure
+Fluent Bit, this role is included.
+
### Simple Configuration File
In the following configuration file, the input plugin *kubernetes_events* collects events every 5 seconds (default for *interval_nsec*) and exposes them through the [standard output plugin](../outputs/standard-output.md) on the console.
```text
[SERVICE]
-flush 1
-log_level info
+ flush 1
+ log_level info
[INPUT]
-name kubernetes_events
-tag k8s_events
-kube_url https://kubernetes.default.svc
+ name kubernetes_events
+ tag k8s_events
+ kube_url https://kubernetes.default.svc
[OUTPUT]
-name stdout
-match *
+ name stdout
+ match *
```
### Event Timestamp
-Event timestamp will be created from the first existing field in the following order of precendence: lastTimestamp, firstTimestamp, metadata.creationTimestamp
+
+Event timestamps are created from the first existing field, based on the following
+order of precedence:
+
+1. `lastTimestamp`
+1. `firstTimestamp`
+1. `metadata.creationTimestamp`
diff --git a/pipeline/inputs/memory-metrics.md b/pipeline/inputs/memory-metrics.md
index 0f380c614..de04a3719 100644
--- a/pipeline/inputs/memory-metrics.md
+++ b/pipeline/inputs/memory-metrics.md
@@ -23,6 +23,11 @@ Fluent Bit v1.x.x
[3] memory: [1488543159, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
```
+## Threading
+
+You can enable the `threaded` setting to run this input in its own
+[thread](../../administration/multithreading.md#inputs).
+
### Configuration File
In your main configuration file append the following _Input_ & _Output_ sections:
diff --git a/pipeline/inputs/mqtt.md b/pipeline/inputs/mqtt.md
index 5ed4295fc..9d67bdbea 100644
--- a/pipeline/inputs/mqtt.md
+++ b/pipeline/inputs/mqtt.md
@@ -6,11 +6,12 @@ The **MQTT** input plugin, allows to retrieve messages/data from MQTT control pa
The plugin supports the following configuration parameters:
-| Key | Description |
-| :--- | :--- |
-| Listen | Listener network interface, default: 0.0.0.0 |
-| Port | TCP port where listening for connections, default: 1883 |
-| Payload_Key | Specify the key where the payload key/value will be preserved. |
+| Key | Description | Default |
+| :---------- | :------------------------------------------------------------- | :------ |
+| Listen | Listener network interface. | `0.0.0.0` |
+| Port | TCP port where listening for connections. | `1883` |
+| Payload_Key | Specify the key where the payload key/value will be preserved. | _none_ |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
## Getting Started
@@ -53,4 +54,3 @@ In your main configuration file append the following _Input_ & _Output_ sections
Name stdout
Match *
```
-
diff --git a/pipeline/inputs/network-io-metrics.md b/pipeline/inputs/network-io-metrics.md
index 64fb21f26..bbfbcc5de 100644
--- a/pipeline/inputs/network-io-metrics.md
+++ b/pipeline/inputs/network-io-metrics.md
@@ -2,7 +2,8 @@
The **netif** input plugin gathers network traffic information of the running system every certain interval of time, and reports them.
-The Network I/O Metrics plugin creates metrics that are log-based \(I.e. JSON payload\). If you are looking for Prometheus-based metrics please see the Node Exporter Metrics input plugin.
+The Network I/O Metrics plugin creates metrics that are log-based, such as JSON
+payload. For Prometheus-based metrics, see the Node Exporter Metrics input plugin.
## Configuration Parameters
@@ -15,6 +16,7 @@ The plugin supports the following configuration parameters:
| Interval\_NSec | Polling interval \(nanosecond\). | 0 |
| Verbose | If true, gather metrics precisely. | false |
| Test\_At\_Init | If true, testing if the network interface is valid at initialization. | false |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
## Getting Started
@@ -75,4 +77,3 @@ pipeline:
Note: Total interval \(sec\) = Interval\_Sec + \(Interval\_Nsec / 1000000000\).
e.g. 1.5s = 1s + 500000000ns
-
diff --git a/pipeline/inputs/nginx.md b/pipeline/inputs/nginx.md
index d56fc375b..1735e5ef3 100644
--- a/pipeline/inputs/nginx.md
+++ b/pipeline/inputs/nginx.md
@@ -12,6 +12,7 @@ The plugin supports the following configuration parameters:
| Port | Port of the target nginx service to connect to. | 80 |
| Status_URL | The URL of the Stub Status Handler. | /status |
| Nginx_Plus | Turn on NGINX plus mode. | true |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
## Getting Started
@@ -35,8 +36,8 @@ server {
### Configuration with NGINX Plus REST API
-A much more powerful and flexible metrics API is available with NGINX Plus. A path needs to be configured
-in NGINX Plus first.
+Another metrics API is available with NGINX Plus. You must first configure a path in
+NGINX Plus.
```
server {
@@ -130,8 +131,9 @@ Fluent Bit v2.x.x
## Exported Metrics
-This documentation is copied from the nginx prometheus exporter metrics documentation:
-[https://github.com/nginxinc/nginx-prometheus-exporter/blob/master/README.md].
+This documentation is copied from the
+[NGINX Prometheus Exporter metrics documentation](https://github.com/nginxinc/nginx-prometheus-exporter/blob/main/README.md)
+on GitHub.
### Common metrics:
Name | Type | Description | Labels
diff --git a/pipeline/inputs/node-exporter-metrics.md b/pipeline/inputs/node-exporter-metrics.md
index 097f1ac9c..2ac3eff5c 100644
--- a/pipeline/inputs/node-exporter-metrics.md
+++ b/pipeline/inputs/node-exporter-metrics.md
@@ -81,6 +81,10 @@ The following table describes the available collectors as part of this plugin. A
| nvme | Exposes nvme statistics from `/proc`. | Linux | v2.2.0 |
| processes | Exposes processes statistics from `/proc`. | Linux | v2.2.0 |
+## Threading
+
+This input always runs in its own [thread](../../administration/multithreading.md#inputs).
+
## Getting Started
### Simple Configuration File
@@ -114,7 +118,7 @@ In the following configuration file, the input plugin _node_exporter_metrics col
host 0.0.0.0
port 2021
-
+
```
{% endtab %}
@@ -201,4 +205,3 @@ docker-compose down
Our current plugin implements a sub-set of the available collectors in the original Prometheus Node Exporter, if you would like that we prioritize a specific collector please open a Github issue by using the following template:\
\
\- [in_node_exporter_metrics](https://github.com/fluent/fluent-bit/issues/new?assignees=\&labels=\&template=feature_request.md\&title=in_node_exporter_metrics:%20add%20ABC%20collector)
-
diff --git a/pipeline/inputs/opentelemetry.md b/pipeline/inputs/opentelemetry.md
index b6dd0ad78..05e0bad92 100644
--- a/pipeline/inputs/opentelemetry.md
+++ b/pipeline/inputs/opentelemetry.md
@@ -20,6 +20,7 @@ Our compliant implementation fully supports OTLP/HTTP and OTLP/GRPC. Note that t
| buffer_chunk_size | Initial size and allocation strategy to store the payload (advanced users only) | 512K |
|successful_response_code | It allows to set successful response code. `200`, `201` and `204` are supported.| 201 |
| tag_from_uri | If true, tag will be created from uri. e.g. v1_metrics from /v1/metrics . | true |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
Important note: Raw traces means that any data forwarded to the traces endpoint (`/v1/traces`) will be packed and forwarded as a log message, and will NOT be processed by Fluent Bit. The traces endpoint by default expects a valid protobuf encoded payload, but you can set the `raw_traces` option in case you want to get trace telemetry data to any of Fluent Bit's supported outputs.
@@ -50,13 +51,13 @@ __OTLP/GRPC__
## Getting started
-The OpenTelemetry plugin currently supports the following telemetry data types:
+The OpenTelemetry input plugin supports the following telemetry data types:
-| Type | HTTP/JSON | HTTP/Protobuf |
-| ----------- | ------------- | --------------- |
-| Logs | Stable | Stable |
-| Metrics | Unimplemented | Stable |
-| Traces | Unimplemented | Stable |
+| Type | HTTP1/JSON | HTTP1/Protobuf | HTTP2/GRPC |
+| ------- | ---------- | -------------- | ---------- |
+| Logs | Stable | Stable | Stable |
+| Metrics | Unimplemented | Stable | Stable |
+| Traces | Unimplemented | Stable | Stable |
A sample config file to get started will look something like the following:
@@ -79,13 +80,13 @@ pipeline:
{% tab title="fluent-bit.conf" %}
```
[INPUT]
- name opentelemetry
- listen 127.0.0.1
- port 4318
+ name opentelemetry
+ listen 127.0.0.1
+ port 4318
[OUTPUT]
- name stdout
- match *
+ name stdout
+ match *
```
{% endtab %}
diff --git a/pipeline/inputs/podman-metrics.md b/pipeline/inputs/podman-metrics.md
index fb51e3328..4d6181eb9 100644
--- a/pipeline/inputs/podman-metrics.md
+++ b/pipeline/inputs/podman-metrics.md
@@ -13,6 +13,7 @@ description: The Podman Metrics input plugin allows you to collect metrics from
| path.config | Custom path to podman containers configuration file | /var/lib/containers/storage/overlay-containers/containers.json |
| path.sysfs | Custom path to sysfs subsystem directory | /sys/fs/cgroup |
| path.procfs | Custom path to proc subsystem directory | /proc |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
## Getting Started
diff --git a/pipeline/inputs/process-exporter-metrics.md b/pipeline/inputs/process-exporter-metrics.md
index 5c933bce2..f74c0996c 100644
--- a/pipeline/inputs/process-exporter-metrics.md
+++ b/pipeline/inputs/process-exporter-metrics.md
@@ -42,6 +42,10 @@ macOS does not have the `proc` filesystem so this plugin will not work for it.
| thread\_wchan | Exposes thread\_wchan from `/proc`. |
| thread | Exposes thread statistics from `/proc`. |
+## Threading
+
+This input always runs in its own [thread](../../administration/multithreading.md#inputs).
+
## Getting Started
### Simple Configuration File
@@ -83,7 +87,8 @@ curl http://127.0.0.1:2021/metrics
### Container to Collect Host Metrics
When deploying Fluent Bit in a container you will need to specify additional settings to ensure that Fluent Bit has access to the process details.
-The following `docker` command deploys Fluent Bit with a specific mount path for `procfs` and settings enabled to ensure that Fluent Bit can collect from the host.
+The following `docker` command deploys Fluent Bit with a specific mount path for
+`procfs` and settings enabled to ensure that Fluent Bit can collect from the host.
These are then exposed over port 2021.
```
diff --git a/pipeline/inputs/process.md b/pipeline/inputs/process.md
index f4a618466..06ba33913 100644
--- a/pipeline/inputs/process.md
+++ b/pipeline/inputs/process.md
@@ -1,8 +1,10 @@
# Process Metrics
+
_Process_ input plugin allows you to check how healthy a process is. It does so by performing a service check at every certain interval of time specified by the user.
-The Process metrics plugin creates metrics that are log-based \(I.e. JSON payload\). If you are looking for Prometheus-based metrics please see the Node Exporter Metrics input plugin.
+The Process metrics plugin creates metrics that are log-based, such as JSON payload.
+For Prometheus-based metrics, see the Node Exporter Metrics input plugin.
## Configuration Parameters
@@ -16,6 +18,7 @@ The plugin supports the following configuration parameters:
| Alert | If enabled, it will only generate messages if the target process is down. By default this option is disabled. |
| Fd | If enabled, a number of fd is appended to each records. Default value is true. |
| Mem | If enabled, memory usage of the process is appended to each records. Default value is true. |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). Default: `false`. |
## Getting Started
@@ -63,4 +66,3 @@ Fluent Bit v1.x.x
[2] proc.0: [1485780299, {"alive"=>true, "proc_name"=>"fluent-bit", "pid"=>10964, "mem.VmPeak"=>14740000, "mem.VmSize"=>14740000, "mem.VmLck"=>0, "mem.VmHWM"=>1152000, "mem.VmRSS"=>1148000, "mem.VmData"=>2276000, "mem.VmStk"=>88000, "mem.VmExe"=>1768000, "mem.VmLib"=>2328000, "mem.VmPTE"=>68000, "mem.VmSwap"=>0, "fd"=>18}]
[3] proc.0: [1485780300, {"alive"=>true, "proc_name"=>"fluent-bit", "pid"=>10964, "mem.VmPeak"=>14740000, "mem.VmSize"=>14740000, "mem.VmLck"=>0, "mem.VmHWM"=>1152000, "mem.VmRSS"=>1148000, "mem.VmData"=>2276000, "mem.VmStk"=>88000, "mem.VmExe"=>1768000, "mem.VmLib"=>2328000, "mem.VmPTE"=>68000, "mem.VmSwap"=>0, "fd"=>18}]
```
-
diff --git a/pipeline/inputs/prometheus-remote-write.md b/pipeline/inputs/prometheus-remote-write.md
index f23ecc40b..b149977b7 100644
--- a/pipeline/inputs/prometheus-remote-write.md
+++ b/pipeline/inputs/prometheus-remote-write.md
@@ -17,6 +17,7 @@ This input plugin allows you to ingest a payload in the Prometheus remote-write
|successful\_response\_code | It allows to set successful response code. `200`, `201` and `204` are supported.| 201 |
| tag\_from\_uri | If true, tag will be created from uri, e.g. api\_prom\_push from /api/prom/push, and any tag specified in the config will be ignored. If false then a tag must be provided in the config for this input. | true |
| uri | Specify an optional HTTP URI for the target web server listening for prometheus remote write payloads, e.g: /api/prom/push | |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
A sample config file to get started will look something like the following:
@@ -26,14 +27,14 @@ A sample config file to get started will look something like the following:
{% tab title="fluent-bit.conf" %}
```
[INPUT]
- name prometheus_remote_write
- listen 127.0.0.1
- port 8080
- uri /api/prom/push
+ name prometheus_remote_write
+ listen 127.0.0.1
+ port 8080
+ uri /api/prom/push
[OUTPUT]
- name stdout
- match *
+ name stdout
+ match *
```
{% endtab %}
@@ -65,13 +66,13 @@ Communicating with TLS, you will need to use the tls related parameters:
```
[INPUT]
- Name prometheus_remote_write
- Listen 127.0.0.1
- Port 8080
- Uri /api/prom/push
- Tls On
- tls.crt_file /path/to/certificate.crt
- tls.key_file /path/to/certificate.key
+ Name prometheus_remote_write
+ Listen 127.0.0.1
+ Port 8080
+ Uri /api/prom/push
+ Tls On
+ tls.crt_file /path/to/certificate.crt
+ tls.key_file /path/to/certificate.key
```
Now, you should be able to send data over TLS to the remote write input.
diff --git a/pipeline/inputs/prometheus-scrape-metrics.md b/pipeline/inputs/prometheus-scrape-metrics.md
index d068de3ac..5f3305b44 100644
--- a/pipeline/inputs/prometheus-scrape-metrics.md
+++ b/pipeline/inputs/prometheus-scrape-metrics.md
@@ -12,6 +12,7 @@ The initial release of the Prometheus Scrape metric allows you to collect metric
| port | The port of the prometheus metric endpoint that you want to scrape | |
| scrape\_interval | The interval to scrape metrics | 10s |
| metrics\_path | The metrics URI endpoint, that must start with a forward slash.
Note: Parameters can also be added to the path by using ?
Similar to the functionality exposed by filters, this processor presents a unified mechanism to perform such operations for data manipulation. The most significant difference is that processors perform better than filters, and when chaining them, there are no encoding/decoding performance penalties.
Note that processors and this specific component can only be enabled using the new YAML configuration format. Classic mode configuration format doesn't support processors.
+## Contexts
+
+The processor, works on top of what we call a __context__, meaning _the place_ where the content modification will happen. We provide different contexts to manipulate the desired information, the following contexts are available:
+
+| Context Name | Signal | Description |
+| -- | -- | -- |
+| `attributes` | Logs | Modify the attributes or metadata of a Log record. |
+| `body` | Logs | Modify the content of a Log record. |
+| `span_name` | Traces | Modify the name of a Span. |
+| `span_kind` | Traces | Modify the kind of a Span. |
+| `span_status` | Traces | Modify the status of a Span. |
+| `span_attributes` | Traces | Modify the attributes of a Span. |
+
+
+### OpenTelemetry Contexts
+
+In addition, we provide special contexts to operate on data that follows an __OpenTelemetry Log Schema__, all of them operates on shared data across a group of records:
+
+| Context Name | Signal | Description |
+| -- | -- | -- |
+| `otel_resource_attributes` | Logs | Modify the attributes of the Log Resource. |
+| `otel_scope_name` | Logs | Modify the name of a Log Scope. |
+| `otel_scope_version` | Logs | Modify version of a Log Scope. |
+| `otel_scope_attributes` | Logs | Modify the attributes of a Log Scope. |
+
+> TIP: if your data is not following the OpenTelemetry Log Schema and your backend or destination for your logs expects to be in an OpenTelemetry schema, take a look at the processor called OpenTelemetry Envelope that you can use in conjunbction with this processor to transform your data to be compatible with OpenTelemetry Log schema.
+
## Configuration Parameters
| Key | Description |
| :---------- | :--- |
-| action | Define the operation to run on the target content. This field is mandatory; for more details about the actions available, check the table below. |
-| context | Specify which component of the Telemetry type will be affected. When processing Logs the following contexts are available: `attributes` or `body`. When processing Traces the following contexts are available: `span_name`, `span_kind`, `span_status`, `span_attributes`. |
+| context | Specify the context where the modifications will happen (more details above).The following contexts are available: `attributes`, `body`, `span_name`, `span_kind`, `span_status`, `span_attributes`, `otel_resource_attributes`, `otel_scope_name`, `otel_scope_version`, `otel_scope_attributes`. |
| key | Specify the name of the key that will be used to apply the modification. |
| value | Based on the action type, `value` might required and represent different things. Check the detailed information for the specific actions. |
| pattern | Defines a regular expression pattern. This property is only used by the `extract` action. |
@@ -23,13 +51,13 @@ The actions specify the type of operation to run on top of a specific key or con
| Action | Description |
| ------- | ------------------------------------------------------------ |
-| insert | Insert a new key with a value into the target context. The `key` and `value` parameters are required. |
-| upsert | Given a specific key with a value, the `upsert` operation will try to update the value of the key. If the key does not exist, the key will be created. The `key` and `value` parameters are required. |
-| delete | Delete a key from the target context. The `key` parameter is required. |
-| rename | Change the name of a key. The `value` set in the configuration will represent the new name. The `key` and `value` parameters are required. |
-| hash | Replace the key value with a hash generated by the SHA-256 algorithm, the binary value generated is finally set as an hex string representation. The `key` parameter is required. |
-| extract | Allows to extact the value of a single key as a list of key/value pairs. This action needs the configuration of a regular expression in the `pattern` property . The `key` and `pattern` parameters are required. For more details check the examples below. |
-| convert | Convert the data type of a key value. The `key` and `converted_type` parameters are required. |
+| `insert` | Insert a new key with a value into the target context. The `key` and `value` parameters are required. |
+| `upsert` | Given a specific key with a value, the `upsert` operation will try to update the value of the key. If the key does not exist, the key will be created. The `key` and `value` parameters are required. |
+| `delete` | Delete a key from the target context. The `key` parameter is required. |
+| `rename` | Change the name of a key. The `value` set in the configuration will represent the new name. The `key` and `value` parameters are required. |
+| `hash` | Replace the key value with a hash generated by the SHA-256 algorithm, the binary value generated is finally set as an hex string representation. The `key` parameter is required. |
+| `extract` | Allows to extact the value of a single key as a list of key/value pairs. This action needs the configuration of a regular expression in the `pattern` property . The `key` and `pattern` parameters are required. For more details check the examples below. |
+| `convert` | Convert the data type of a key value. The `key` and `converted_type` parameters are required. |
#### Insert example
@@ -74,7 +102,7 @@ pipeline:
action: upsert
key: "key2"
value: "example"
-
+
outputs:
- name : stdout
match: '*'
@@ -97,8 +125,8 @@ pipeline:
logs:
- name: content_modifier
action: delete
- key: "key2"
-
+ key: "key2"
+
outputs:
- name : stdout
match: '*'
@@ -168,7 +196,7 @@ pipeline:
action: extract
key: "http.url"
pattern: ^(?
+
+The **labels** processor lets you manipulate the labels of metrics.
+
+Similar to filters, this processor presents a enriching/modifying mechanism to
+perform operations for labels manipulation. The most significant difference is
+that processors perform better than filters, and when chaining them there are no
+encoding or decoding performance penalties.
+
+{% hint style="info" %}
+**Note:** Both processors and this specific component can be enabled only by using
+the YAML configuration format. Classic mode configuration format doesn't support
+processors.
+{% endhint %}
+
+## Configuration Parameters
+
+| Key | Description |
+| :----- | :---------- |
+| update | Update an existing key with a value into metrics. The key/value pair is required. If the specified key doesn't exist, the operation silently fails and has no effect. |
+| insert | Insert a new key with a value into metrics. The key/value pair is required. |
+| upsert | Upsert a specific key with a value, the `upsert` operation will try to update the value of the key. If the key does not exist, the key will be created. The key-value pair is required. |
+| delete | Delete a key from the labels of metrics. The key/value pair is required. If the specified key doesn't exist, the operation silently fails and has no effect. |
+| hash | Replace the key value with a hash generated by the SHA-256 algorithm from the specified label name. The generated binary value is set as a hex string. |
+
+#### Update example
+
+Change the value of the `name` to `fluentbit`:
+
+```yaml
+pipeline:
+ inputs:
+ - name: fluentbit_metrics
+ processors:
+ metrics:
+ - name: labels
+ update: name fluentbit
+ outputs:
+ - name : stdout
+ match: '*'
+```
+
+#### Insert example
+
+The following example appends the key `agent` with the value `fluentbit` as the label
+of metrics:
+
+```yaml
+pipeline:
+ inputs:
+ - name: fluentbit_metrics
+ processors:
+ metrics:
+ - name: labels
+ insert: agent fluentbit
+ outputs:
+ - name : stdout
+ match: '*'
+```
+
+#### Upsert example
+
+Upsert the value of `name` and insert `fluentbit`:
+
+```yaml
+pipeline:
+ inputs:
+ - name: fluentbit_metrics
+ processors:
+ metrics:
+ - name: labels
+ upsert: name fluentbit
+ outputs:
+ - name : stdout
+ match: '*'
+```
+
+#### Delete example
+
+Delete containing `name` key from metrics:
+
+```yaml
+pipeline:
+ inputs:
+ - name: fluentbit_metrics
+ processors:
+ metrics:
+ - name: labels
+ delete: name
+ outputs:
+ - name : stdout
+ match: '*'
+```
+
+#### Hash example
+
+Apply the SHA-1 algorithm for the value of the key `hostname`:
+
+```yaml
+pipeline:
+ inputs:
+ - name: fluentbit_metrics
+ processors:
+ metrics:
+ - name: labels
+ hash: hostname
+ outputs:
+ - name : stdout
+ match: '*'
+```
diff --git a/pipeline/processors/metrics-selector.md b/pipeline/processors/metrics-selector.md
index 962577ad3..262075f9a 100644
--- a/pipeline/processors/metrics-selector.md
+++ b/pipeline/processors/metrics-selector.md
@@ -2,6 +2,8 @@
The **metric_selector** processor allows you to select metrics to include or exclude (similar to the `grep` filter for logs).
+
+
## Configuration Parameters
The native processor plugin supports the following configuration parameters:
@@ -9,9 +11,10 @@ The native processor plugin supports the following configuration parameters:
| Key | Description | Default |
| :---------- | :--- | :--- |
| Metric\_Name | Keep metrics in which the metric of name matches with the actual name or the regular expression. | |
-| Context | Specify matching context. Currently, metric_name is only supported. | `Metrics_Name` |
+| Context | Specify matching context. Currently, metric\_name and delete\_label\_value are only supported. | `Metrics_Name` |
| Action | Specify the action for specified metrics. INCLUDE and EXCLUDE are allowed. | |
| Operation\_Type | Specify the operation type of action for metrics payloads. PREFIX and SUBSTRING are allowed. | |
+| Label | Specify a label key and value pair. | |
## Configuration Examples
@@ -44,6 +47,35 @@ pipeline:
delete: name
+ outputs:
+ - name: stdout
+ match: '*'
+```
+{% endtab %}
+
+{% tab title="context-delete\_label\_value.yaml" %}
+```yaml
+service:
+ flush: 5
+ daemon: off
+ log_level: info
+
+pipeline:
+ inputs:
+ - name: fluentbit_metrics
+ tag: fluentbit.metrics
+ scrape_interval: 10
+
+ processors:
+ metrics:
+ - name: metrics_selector
+ context: delete_label_value
+ label: name stdout.0
+
+ - name: labels
+ delete: name
+
+
outputs:
- name: stdout
match: '*'
diff --git a/pipeline/processors/opentelemetry-envelope.md b/pipeline/processors/opentelemetry-envelope.md
new file mode 100644
index 000000000..f9df45a3c
--- /dev/null
+++ b/pipeline/processors/opentelemetry-envelope.md
@@ -0,0 +1,165 @@
+# OpenTelemetry Envelope
+
+The _OpenTelemetry Envelope_ processor is used to transform your data to be compatible with the OpenTelemetry Log schema. If your data was __not__ generated by [OpenTelemetry input](../inputs/opentelemetry.md) and your backend or destination for your logs expects to be in an OpenTelemetry schema.
+
+
+
+## Configuration Parameters
+
+The processor does not provide any extra configuration parameter, it can be used directly in your _processors_ Yaml directive.
+
+## Usage Example
+
+In this example, we will use the Dummy input plugin to generate a sample message per second, right after is created the processor `opentelemetry_envelope` is used to transform the data to be compatible with the OpenTelemetry Log schema. The output is sent to the standard output and also to an OpenTelemetry collector which is receiving data in port 4318.
+
+
+__fluent-bit.yaml__
+
+```yaml
+service:
+ flush: 1
+ log_level: info
+
+pipeline:
+ inputs:
+ - name: dummy
+ dummy: '{"message": "Hello World"}'
+
+ processors:
+ logs:
+ - name: opentelemetry_envelope
+
+ outputs:
+ - name : stdout
+ match: '*'
+
+ - name: opentelemetry
+ match: '*'
+ host: 127.0.0.1
+ port: 4318
+```
+
+__otel-collector.yaml__
+
+```yaml
+receivers:
+ otlp:
+ protocols:
+ http:
+ endpoint: 127.0.0.1:4318
+
+exporters:
+ file:
+ path: out.json
+ logging:
+ loglevel: info
+
+service:
+ telemetry:
+ logs:
+ level: debug
+ pipelines:
+ logs:
+ receivers: [otlp]
+ exporters: [file, logging]
+```
+
+ You will notice in the standard output of FLuent Bit will print the raw representation of the schema, however, the OpenTelemetry collector will receive the data in the OpenTelemetry Log schema.
+
+Inspecting the output file `out.json` you will see the data in the OpenTelemetry Log schema:
+
+
+```json
+{
+ "resourceLogs": [
+ {
+ "resource": {},
+ "scopeLogs": [
+ {
+ "scope": {},
+ "logRecords": [
+ {
+ "timeUnixNano": "1722904188085758000",
+ "body": {
+ "stringValue": "dummy"
+ },
+ "traceId": "",
+ "spanId": ""
+ }
+ ]
+ }
+ ]
+ }
+ ]
+}
+```
+
+While OpenTelemetry Envelope enrich your logs with the Schema, you might be interested into take a step further and use the [Content Modifier](../processors/content-modifier.md) processor to modify the content of your logs. Here is a quick example that will allow you to add some resource and scope attributes to your logs:
+
+```yaml
+service:
+ flush: 1
+ log_level: info
+
+pipeline:
+ inputs:
+ - name: dummy
+ dummy: '{"message": "Hello World"}'
+
+ processors:
+ logs:
+ - name: opentelemetry_envelope
+
+ - name: content_modifier
+ context: otel_resource_attributes
+ action: upsert
+ key: service.name
+ value: my-service
+
+ outputs:
+ - name : stdout
+ match: '*'
+
+ - name: opentelemetry
+ match: '*'
+ host: 127.0.0.1
+ port: 4318
+```
+
+The collector JSON output will look like this:
+
+```json
+{
+ "resourceLogs": [
+ {
+ "resource": {
+ "attributes": [
+ {
+ "key": "service.name",
+ "value": {
+ "stringValue": "my-service"
+ }
+ }
+ ]
+ },
+ "scopeLogs": [
+ {
+ "scope": {},
+ "logRecords": [
+ {
+ "timeUnixNano": "1722904465173450000",
+ "body": {
+ "stringValue": "Hello World"
+ },
+ "traceId": "",
+ "spanId": ""
+ }
+ ]
+ }
+ ]
+ }
+ ]
+}
+```
+
+For more details about further processing, read the [Content Modifier](../processors/content-modifier.md) processor documentation.
diff --git a/pipeline/processors/sql.md b/pipeline/processors/sql.md
index d1b06d461..47482f80a 100644
--- a/pipeline/processors/sql.md
+++ b/pipeline/processors/sql.md
@@ -2,6 +2,8 @@
The **sql** processor provides a simple interface to select content from Logs by also supporting conditional expressions.
+
+
Our SQL processor does not depend on a database or indexing; it runs everything on the fly (this is good). We don't have the concept of tables but you run the query on the STREAM.
Note that this processor differs from the "stream processor interface" that runs after the filters; this one can only be used in the processor's section of the input plugins when using YAML configuration mode.
diff --git a/vale-styles/FluentBit/AMPM.yml b/vale-styles/FluentBit/AMPM.yml
new file mode 100644
index 000000000..5b696b3b8
--- /dev/null
+++ b/vale-styles/FluentBit/AMPM.yml
@@ -0,0 +1,10 @@
+
+extends: existence
+message: "Use 'AM' or 'PM' (preceded by a space)."
+link: 'https://developers.google.com/style/word-list'
+level: suggestion
+nonword: true
+tokens:
+ - '\d{1,2}[AP]M'
+ - '\d{1,2} ?[ap]m'
+ - '\d{1,2} ?[aApP]\.[mM]\.'
\ No newline at end of file
diff --git a/vale-styles/FluentBit/Acronyms.yml b/vale-styles/FluentBit/Acronyms.yml
new file mode 100644
index 000000000..19936b25c
--- /dev/null
+++ b/vale-styles/FluentBit/Acronyms.yml
@@ -0,0 +1,95 @@
+extends: conditional
+message: "Spell out '%s', if it's unfamiliar to the audience."
+link: 'https://developers.google.com/style/abbreviations'
+level: suggestion
+ignorecase: false
+# Ensures that the existence of 'first' implies the existence of 'second'.
+first: '\b([A-Z]{3,5})\b'
+second: '(?:\b[A-Z][a-z]+ )+\(([A-Z]{3,5})\)'
+# ... with the exception of these:
+exceptions:
+ - ACL
+ - API
+ - ARN
+ - ASC
+ - ASP
+ - AWS
+ - CIDR
+ - CLI
+ - CPU
+ - CRD
+ - CSS
+ - CSV
+ - DEBUG
+ - DESC
+ - DOM
+ - DNS
+ - DPI
+ - DPPS
+ - FAQ
+ - FIPS
+ - GCC
+ - GCP
+ - GDB
+ - GET
+ - GNU
+ - GPG
+ - GPU
+ - GTK
+ - GUI
+ - GZIP
+ - HPA
+ - IAM
+ - HTML
+ - HTTP
+ - HTTPS
+ - IDE
+ - JAR
+ - JSON
+ - JSX
+ - LESS
+ - LLDB
+ - LTS
+ - NET
+ - NOTE
+ - NVDA
+ - OSS
+ - PATH
+ - PEM
+ - PDF
+ - PHP
+ - POSIX
+ - POST
+ - RAM
+ - REPL
+ - REST
+ - RHEL
+ - RPC
+ - RSA
+ - SASL
+ - SCM
+ - SCSS
+ - SDK
+ - SIEM
+ - SLA
+ - SQL
+ - SSH
+ - SSL
+ - SSO
+ - SVG
+ - TBD
+ - TCP
+ - TLS
+ - TRE
+ - TODO
+ - UDP
+ - URI
+ - URL
+ - USB
+ - UTC
+ - UTF
+ - UUID
+ - XML
+ - XSS
+ - YAML
+ - ZIP
diff --git a/vale-styles/FluentBit/AmSpelling.yml b/vale-styles/FluentBit/AmSpelling.yml
new file mode 100644
index 000000000..9a8ea2e30
--- /dev/null
+++ b/vale-styles/FluentBit/AmSpelling.yml
@@ -0,0 +1,8 @@
+extends: existence
+message: "In general, use American spelling instead of '%s'."
+link: 'https://developers.google.com/style/spelling'
+ignorecase: true
+level: suggestion
+tokens:
+ - '(?:\w+)nised?'
+ - '(?:\w+)logue'
\ No newline at end of file
diff --git a/vale-styles/FluentBit/Ampersand.yml b/vale-styles/FluentBit/Ampersand.yml
new file mode 100644
index 000000000..75117bc08
--- /dev/null
+++ b/vale-styles/FluentBit/Ampersand.yml
@@ -0,0 +1,9 @@
+---
+extends: existence
+message: "Don't use an ampersand in place of the word 'and'. Always write out 'and' unless the ampersand is part of a proper name."
+nonword: true
+ignorecase: false
+level: suggestion
+scope: sentence
+tokens:
+ - '[^\*{2}].*.&.*[^\*{2}]\n'
diff --git a/vale-styles/FluentBit/Colons.yml b/vale-styles/FluentBit/Colons.yml
new file mode 100644
index 000000000..aee9281c3
--- /dev/null
+++ b/vale-styles/FluentBit/Colons.yml
@@ -0,0 +1,8 @@
+extends: existence
+message: "'%s' should be in lowercase."
+link: 'https://developers.google.com/style/colons'
+nonword: true
+level: suggestion
+scope: sentence
+tokens:
+ - ':\s[A-Z]'
\ No newline at end of file
diff --git a/vale-styles/FluentBit/Contractions.yml b/vale-styles/FluentBit/Contractions.yml
new file mode 100644
index 000000000..525687ca6
--- /dev/null
+++ b/vale-styles/FluentBit/Contractions.yml
@@ -0,0 +1,30 @@
+extends: substitution
+message: "Feel free to use '%s' instead of '%s'."
+link: 'https://developers.google.com/style/contractions'
+level: suggestion
+ignorecase: true
+action:
+ name: replace
+swap:
+ are not: aren't
+ cannot: can't
+ could not: couldn't
+ did not: didn't
+ do not: don't
+ does not: doesn't
+ has not: hasn't
+ have not: haven't
+ how is: how's
+ is not: isn't
+ it is: it's
+ should not: shouldn't
+ that is: that's
+ they are: they're
+ was not: wasn't
+ we are: we're
+ we have: we've
+ were not: weren't
+ what is: what's
+ when is: when's
+ where is: where's
+ will not: won't
\ No newline at end of file
diff --git a/vale-styles/FluentBit/DateFormat.yml b/vale-styles/FluentBit/DateFormat.yml
new file mode 100644
index 000000000..c70b3b7bc
--- /dev/null
+++ b/vale-styles/FluentBit/DateFormat.yml
@@ -0,0 +1,9 @@
+extends: existence
+message: "Use 'July 31, 2016' format, not '%s'."
+link: 'https://developers.google.com/style/dates-times'
+ignorecase: true
+level: suggestion
+nonword: true
+tokens:
+ - '\d{1,2}(?:\.|/)\d{1,2}(?:\.|/)\d{4}'
+ - '\d{1,2} (?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)|May|Jun(?:e)|Jul(?:y)|Aug(?:ust)|Sep(?:tember)?|Oct(?:ober)|Nov(?:ember)?|Dec(?:ember)?) \d{4}'
\ No newline at end of file
diff --git a/vale-styles/FluentBit/Directional.yml b/vale-styles/FluentBit/Directional.yml
new file mode 100644
index 000000000..1d18a0f4d
--- /dev/null
+++ b/vale-styles/FluentBit/Directional.yml
@@ -0,0 +1,8 @@
+---
+extends: existence
+message: "Verify your use of '%s' with the Style Guide."
+level: suggestion
+ignorecase: true
+tokens:
+ - above
+ - below
diff --git a/vale-styles/FluentBit/DontUse.yml b/vale-styles/FluentBit/DontUse.yml
new file mode 100644
index 000000000..8308555d7
--- /dev/null
+++ b/vale-styles/FluentBit/DontUse.yml
@@ -0,0 +1,18 @@
+---
+extends: existence
+message: "We don't use '%s'."
+ignorecase: true
+level: suggestion
+tokens:
+ - a.k.a.
+ - aka
+ - and/or
+ - at this point
+ - desire
+ - it is recommended that
+ - just
+ - note that
+ - please
+ - quite
+ - such that
+ - thus
diff --git a/vale-styles/FluentBit/Drilldown.yml b/vale-styles/FluentBit/Drilldown.yml
new file mode 100644
index 000000000..ca7cb2ed7
--- /dev/null
+++ b/vale-styles/FluentBit/Drilldown.yml
@@ -0,0 +1,8 @@
+---
+extends: sequence
+message: "Use drilldown as an adjective or noun."
+level: suggestion
+ignorecase: true
+tokens:
+ - tag: NN|JJ
+ pattern: '(?:drill down|drill-down)'
diff --git a/vale-styles/FluentBit/DrilldownVerb.yml b/vale-styles/FluentBit/DrilldownVerb.yml
new file mode 100644
index 000000000..3dd6ca047
--- /dev/null
+++ b/vale-styles/FluentBit/DrilldownVerb.yml
@@ -0,0 +1,8 @@
+---
+extends: sequence
+message: "Use drill down as a verb."
+level: suggestion
+ignorecase: true
+tokens:
+ - tag: VB|VBD|VBG|VBN|VBP|VBZ
+ pattern: '(?:drilldown|drill-down)'
diff --git a/vale-styles/FluentBit/Ellipses.yml b/vale-styles/FluentBit/Ellipses.yml
new file mode 100644
index 000000000..6b93a0c44
--- /dev/null
+++ b/vale-styles/FluentBit/Ellipses.yml
@@ -0,0 +1,9 @@
+extends: existence
+message: "In general, don't use an ellipsis."
+link: 'https://developers.google.com/style/ellipses'
+nonword: true
+level: suggestion
+action:
+ name: remove
+tokens:
+ - '\.\.\.'
\ No newline at end of file
diff --git a/vale-styles/FluentBit/EmDash.yml b/vale-styles/FluentBit/EmDash.yml
new file mode 100644
index 000000000..c0de45e89
--- /dev/null
+++ b/vale-styles/FluentBit/EmDash.yml
@@ -0,0 +1,12 @@
+extends: existence
+message: "Don't put a space before or after a dash."
+link: 'https://developers.google.com/style/dashes'
+nonword: true
+level: suggestion
+action:
+ name: edit
+ params:
+ - remove
+ - ' '
+tokens:
+ - '\s[—–]\s'
\ No newline at end of file
diff --git a/vale-styles/FluentBit/EnDash.yml b/vale-styles/FluentBit/EnDash.yml
new file mode 100644
index 000000000..e69e4bcbb
--- /dev/null
+++ b/vale-styles/FluentBit/EnDash.yml
@@ -0,0 +1,13 @@
+extends: existence
+message: "Use an em dash ('—') instead of '–'."
+link: 'https://developers.google.com/style/dashes'
+nonword: true
+level: suggestion
+action:
+ name: edit
+ params:
+ - replace
+ - '-'
+ - '—'
+tokens:
+ - '–'
\ No newline at end of file
diff --git a/vale-styles/FluentBit/Exclamation.yml b/vale-styles/FluentBit/Exclamation.yml
new file mode 100644
index 000000000..b77798361
--- /dev/null
+++ b/vale-styles/FluentBit/Exclamation.yml
@@ -0,0 +1,7 @@
+extends: existence
+message: "Don't use exclamation points in text."
+link: 'https://developers.google.com/style/exclamation-points'
+nonword: true
+level: suggestion
+tokens:
+ - '\w!(?:\s|$)'
diff --git a/vale-styles/FluentBit/FirstPerson.yml b/vale-styles/FluentBit/FirstPerson.yml
new file mode 100644
index 000000000..e8793eddf
--- /dev/null
+++ b/vale-styles/FluentBit/FirstPerson.yml
@@ -0,0 +1,13 @@
+extends: existence
+message: "Avoid first-person pronouns such as '%s'."
+link: 'https://developers.google.com/style/pronouns#personal-pronouns'
+ignorecase: true
+level: suggestion
+nonword: true
+tokens:
+ - (?:^|\s)I\s
+ - (?:^|\s)I,\s
+ - \bI'm\b
+ - \bme\b
+ - \bmy\b
+ - \bmine\b
\ No newline at end of file
diff --git a/vale-styles/FluentBit/FutureTense.yml b/vale-styles/FluentBit/FutureTense.yml
new file mode 100644
index 000000000..17d1bd6a6
--- /dev/null
+++ b/vale-styles/FluentBit/FutureTense.yml
@@ -0,0 +1,10 @@
+---
+extends: existence
+message: "'%s' might be in future tense. Strive for active voice and present tense in your documentation."
+ignorecase: true
+level: suggestion
+raw:
+ - "(going to( |\n|[[:punct:]])[a-zA-Z]*|"
+ - "will( |\n|[[:punct:]])[a-zA-Z]*|"
+ - "won't( |\n|[[:punct:]])[a-zA-Z]*|"
+ - "[a-zA-Z]*'ll( |\n|[[:punct:]])[a-zA-Z]*)"
diff --git a/vale-styles/FluentBit/Gender.yml b/vale-styles/FluentBit/Gender.yml
new file mode 100644
index 000000000..9b5689ebd
--- /dev/null
+++ b/vale-styles/FluentBit/Gender.yml
@@ -0,0 +1,9 @@
+extends: existence
+message: "Don't use '%s' as a gender-neutral pronoun."
+link: 'https://developers.google.com/style/pronouns#gender-neutral-pronouns'
+level: suggestion
+ignorecase: true
+tokens:
+ - he/she
+ - s/he
+ - \(s\)he
\ No newline at end of file
diff --git a/vale-styles/FluentBit/GenderBias.yml b/vale-styles/FluentBit/GenderBias.yml
new file mode 100644
index 000000000..3a3b6985e
--- /dev/null
+++ b/vale-styles/FluentBit/GenderBias.yml
@@ -0,0 +1,45 @@
+extends: substitution
+message: "Consider using '%s' instead of '%s'."
+link: 'https://developers.google.com/style/inclusive-documentation'
+ignorecase: true
+level: suggestion
+swap:
+ (?:alumna|alumnus): graduate
+ (?:alumnae|alumni): graduates
+ air(?:m[ae]n|wom[ae]n): pilot(s)
+ anchor(?:m[ae]n|wom[ae]n): anchor(s)
+ authoress: author
+ camera(?:m[ae]n|wom[ae]n): camera operator(s)
+ chair(?:m[ae]n|wom[ae]n): chair(s)
+ congress(?:m[ae]n|wom[ae]n): member(s) of congress
+ door(?:m[ae]|wom[ae]n): concierge(s)
+ draft(?:m[ae]n|wom[ae]n): drafter(s)
+ fire(?:m[ae]n|wom[ae]n): firefighter(s)
+ fisher(?:m[ae]n|wom[ae]n): fisher(s)
+ fresh(?:m[ae]n|wom[ae]n): first-year student(s)
+ garbage(?:m[ae]n|wom[ae]n): waste collector(s)
+ lady lawyer: lawyer
+ ladylike: courteous
+ landlord: building manager
+ mail(?:m[ae]n|wom[ae]n): mail carriers
+ man and wife: husband and wife
+ man enough: strong enough
+ mankind: human kind
+ manmade: manufactured
+ manpower: personnel
+ men and girls: men and women
+ middle(?:m[ae]n|wom[ae]n): intermediary
+ news(?:m[ae]n|wom[ae]n): journalist(s)
+ ombuds(?:man|woman): ombuds
+ oneupmanship: upstaging
+ poetess: poet
+ police(?:m[ae]n|wom[ae]n): police officer(s)
+ repair(?:m[ae]n|wom[ae]n): technician(s)
+ sales(?:m[ae]n|wom[ae]n): salesperson or sales people
+ service(?:m[ae]n|wom[ae]n): soldier(s)
+ steward(?:ess)?: flight attendant
+ tribes(?:m[ae]n|wom[ae]n): tribe member(s)
+ waitress: waiter
+ woman doctor: doctor
+ woman scientist[s]?: scientist(s)
+ work(?:m[ae]n|wom[ae]n): worker(s)
\ No newline at end of file
diff --git a/vale-styles/FluentBit/HeadingPunctuation.yml b/vale-styles/FluentBit/HeadingPunctuation.yml
new file mode 100644
index 000000000..659260b27
--- /dev/null
+++ b/vale-styles/FluentBit/HeadingPunctuation.yml
@@ -0,0 +1,13 @@
+extends: existence
+message: "Don't put a period at the end of a heading."
+link: 'https://developers.google.com/style/capitalization#capitalization-in-titles-and-headings'
+nonword: true
+level: suggestion
+scope: heading
+action:
+ name: edit
+ params:
+ - remove
+ - '.'
+tokens:
+ - '[a-z0-9][.]\s*$'
\ No newline at end of file
diff --git a/vale-styles/FluentBit/Headings.yml b/vale-styles/FluentBit/Headings.yml
new file mode 100644
index 000000000..2be26b3b3
--- /dev/null
+++ b/vale-styles/FluentBit/Headings.yml
@@ -0,0 +1,73 @@
+extends: capitalization
+message: "'%s' should use sentence-style capitalization."
+link: 'https://developers.google.com/style/capitalization#capitalization-in-titles-and-headings'
+level: suggestion
+scope: heading
+match: $sentence
+indicators:
+ - ':'
+exceptions:
+ - Amazon
+ - Amazon CloudWatch
+ - Amazon Kinesis Firehose
+ - Amazon Kinesis Streams
+ - API
+ - APIs
+ - Azure
+ - BuildKite
+ - CircleCI
+ - CLI
+ - CloudWatch
+ - Code
+ - Collector
+ - Cosmos
+ - Crowdstrike
+ - cURL
+ - Datadog
+ - Docker
+ - DogStatsD
+ - Elastic Cloud
+ - Emmet
+ - EventBridge
+ - Fluent Bit
+ - GCP
+ - GitLab
+ - GitHub
+ - Google
+ - Google Cloud
+ - Google Cloud Platform
+ - Grafana
+ - gRPC
+ - I
+ - InfluxDB
+ - Kinesis
+ - Kubernetes
+ - LaunchDarkly
+ - Linux
+ - macOS
+ - Marketplace
+ - MongoDB
+ - New Relic
+ - Observability Platform
+ - Okta
+ - OpenMetrics
+ - OpenTelemetry
+ - Opsgenie
+ - PagerDuty
+ - Prometheus
+ - PromQL
+ - REPL
+ - ServiceMonitor
+ - SignalFx
+ - Slack
+ - StatsD
+ - Studio
+ - Tanzu
+ - Telemetry Pipeline
+ - Terraform
+ - TypeScript
+ - URLs
+ - VictorOps
+ - Visual
+ - VS
+ - Windows
diff --git a/vale-styles/FluentBit/Latin.yml b/vale-styles/FluentBit/Latin.yml
new file mode 100644
index 000000000..ca6f3abb7
--- /dev/null
+++ b/vale-styles/FluentBit/Latin.yml
@@ -0,0 +1,17 @@
+extends: substitution
+message: "Use '%s' instead of '%s'."
+link: 'https://developers.google.com/style/abbreviations'
+ignorecase: true
+level: suggestion
+nonword: true
+action:
+ name: replace
+swap:
+ '\b(?:eg|e\.g\.)[\s,]': for example
+ '\b(?:ie|i\.e\.)[\s,]': that is
+ 'ad-hoc': if needed
+ '[\s]et al[\s]': and others
+ '[\s]etc[\s|.]': and so on
+ '[\s]via[\s]': through or by using
+ 'vice versa': and the reverse
+ '[\s]vs[\s|.]': versus
diff --git a/vale-styles/FluentBit/LyHyphens.yml b/vale-styles/FluentBit/LyHyphens.yml
new file mode 100644
index 000000000..97e092a33
--- /dev/null
+++ b/vale-styles/FluentBit/LyHyphens.yml
@@ -0,0 +1,14 @@
+extends: existence
+message: "'%s' doesn't need a hyphen."
+link: 'https://developers.google.com/style/hyphens'
+level: suggestion
+ignorecase: false
+nonword: true
+action:
+ name: edit
+ params:
+ - replace
+ - '-'
+ - ' '
+tokens:
+ - '\s[^\s-]+ly-'
\ No newline at end of file
diff --git a/vale-styles/FluentBit/MayMightCan.yml b/vale-styles/FluentBit/MayMightCan.yml
new file mode 100644
index 000000000..56ce95f06
--- /dev/null
+++ b/vale-styles/FluentBit/MayMightCan.yml
@@ -0,0 +1,7 @@
+---
+extends: existence
+message: "Use 'can' for permissions or 'might' for possibility."
+level: suggestion
+ignorecase: true
+tokens:
+ - may
diff --git a/vale-styles/FluentBit/NonStandardQuotes.yml b/vale-styles/FluentBit/NonStandardQuotes.yml
new file mode 100644
index 000000000..40feaafb3
--- /dev/null
+++ b/vale-styles/FluentBit/NonStandardQuotes.yml
@@ -0,0 +1,8 @@
+---
+extends: existence
+message: 'Use standard single quotes or double quotes only. Do not use left or right quotes.'
+level: suggestion
+ignorecase: true
+scope: raw
+raw:
+ - '[‘’“”]'
diff --git a/vale-styles/FluentBit/OptionalPlurals.yml b/vale-styles/FluentBit/OptionalPlurals.yml
new file mode 100644
index 000000000..50bf2c247
--- /dev/null
+++ b/vale-styles/FluentBit/OptionalPlurals.yml
@@ -0,0 +1,12 @@
+extends: existence
+message: "Don't use plurals in parentheses such as in '%s'."
+link: 'https://developers.google.com/style/plurals-parentheses'
+level: suggestion
+nonword: true
+action:
+ name: edit
+ params:
+ - remove
+ - '(s)'
+tokens:
+ - '\b\w+\(s\)'
diff --git a/vale-styles/FluentBit/Ordinal.yml b/vale-styles/FluentBit/Ordinal.yml
new file mode 100644
index 000000000..cd836d5a5
--- /dev/null
+++ b/vale-styles/FluentBit/Ordinal.yml
@@ -0,0 +1,7 @@
+extends: existence
+message: "Spell out all ordinal numbers ('%s') in text."
+link: 'https://developers.google.com/style/numbers'
+level: suggestion
+nonword: true
+tokens:
+ - \d+(?:st|nd|rd|th)
\ No newline at end of file
diff --git a/vale-styles/FluentBit/Passive.yml b/vale-styles/FluentBit/Passive.yml
new file mode 100644
index 000000000..b52c01204
--- /dev/null
+++ b/vale-styles/FluentBit/Passive.yml
@@ -0,0 +1,184 @@
+extends: existence
+link: 'https://developers.google.com/style/voice'
+message: "In general, use active voice instead of passive voice ('%s')."
+ignorecase: true
+level: suggestion
+raw:
+ - \b(am|are|were|being|is|been|was|be)\b\s*
+tokens:
+ - '[\w]+ed'
+ - awoken
+ - beat
+ - become
+ - been
+ - begun
+ - bent
+ - beset
+ - bet
+ - bid
+ - bidden
+ - bitten
+ - bled
+ - blown
+ - born
+ - bought
+ - bound
+ - bred
+ - broadcast
+ - broken
+ - brought
+ - built
+ - burnt
+ - burst
+ - cast
+ - caught
+ - chosen
+ - clung
+ - come
+ - cost
+ - crept
+ - cut
+ - dealt
+ - dived
+ - done
+ - drawn
+ - dreamt
+ - driven
+ - drunk
+ - dug
+ - eaten
+ - fallen
+ - fed
+ - felt
+ - fit
+ - fled
+ - flown
+ - flung
+ - forbidden
+ - foregone
+ - forgiven
+ - forgotten
+ - forsaken
+ - fought
+ - found
+ - frozen
+ - given
+ - gone
+ - gotten
+ - ground
+ - grown
+ - heard
+ - held
+ - hidden
+ - hit
+ - hung
+ - hurt
+ - kept
+ - knelt
+ - knit
+ - known
+ - laid
+ - lain
+ - leapt
+ - learnt
+ - led
+ - left
+ - lent
+ - let
+ - lighted
+ - lost
+ - made
+ - meant
+ - met
+ - misspelt
+ - mistaken
+ - mown
+ - overcome
+ - overdone
+ - overtaken
+ - overthrown
+ - paid
+ - pled
+ - proven
+ - put
+ - quit
+ - read
+ - rid
+ - ridden
+ - risen
+ - run
+ - rung
+ - said
+ - sat
+ - sawn
+ - seen
+ - sent
+ - set
+ - sewn
+ - shaken
+ - shaven
+ - shed
+ - shod
+ - shone
+ - shorn
+ - shot
+ - shown
+ - shrunk
+ - shut
+ - slain
+ - slept
+ - slid
+ - slit
+ - slung
+ - smitten
+ - sold
+ - sought
+ - sown
+ - sped
+ - spent
+ - spilt
+ - spit
+ - split
+ - spoken
+ - spread
+ - sprung
+ - spun
+ - stolen
+ - stood
+ - stridden
+ - striven
+ - struck
+ - strung
+ - stuck
+ - stung
+ - stunk
+ - sung
+ - sunk
+ - swept
+ - swollen
+ - sworn
+ - swum
+ - swung
+ - taken
+ - taught
+ - thought
+ - thrived
+ - thrown
+ - thrust
+ - told
+ - torn
+ - trodden
+ - understood
+ - upheld
+ - upset
+ - wed
+ - wept
+ - withheld
+ - withstood
+ - woken
+ - won
+ - worn
+ - wound
+ - woven
+ - written
+ - wrung
\ No newline at end of file
diff --git a/vale-styles/FluentBit/Periods.yml b/vale-styles/FluentBit/Periods.yml
new file mode 100644
index 000000000..0333c3160
--- /dev/null
+++ b/vale-styles/FluentBit/Periods.yml
@@ -0,0 +1,7 @@
+extends: existence
+message: "Don't use periods with acronyms or initialisms such as '%s'."
+link: 'https://developers.google.com/style/abbreviations'
+level: suggestion
+nonword: true
+tokens:
+ - '\b(?:[A-Z]\.){3,}'
\ No newline at end of file
diff --git a/vale-styles/FluentBit/Possessives.yml b/vale-styles/FluentBit/Possessives.yml
new file mode 100644
index 000000000..1d0a74e0c
--- /dev/null
+++ b/vale-styles/FluentBit/Possessives.yml
@@ -0,0 +1,7 @@
+---
+extends: existence
+message: "Rewrite '%s' to not use 's."
+level: suggestion
+ignorecase: true
+tokens:
+ - Bit's
diff --git a/vale-styles/FluentBit/Quotes.yml b/vale-styles/FluentBit/Quotes.yml
new file mode 100644
index 000000000..f9c927459
--- /dev/null
+++ b/vale-styles/FluentBit/Quotes.yml
@@ -0,0 +1,7 @@
+extends: existence
+message: "Commas and periods go inside quotation marks."
+link: 'https://developers.google.com/style/quotation-marks'
+level: suggestion
+nonword: true
+tokens:
+ - '"[^"]+"[.,?]'
\ No newline at end of file
diff --git a/vale-styles/FluentBit/Ranges.yml b/vale-styles/FluentBit/Ranges.yml
new file mode 100644
index 000000000..78af6f999
--- /dev/null
+++ b/vale-styles/FluentBit/Ranges.yml
@@ -0,0 +1,7 @@
+extends: existence
+message: "Don't add words such as 'from' or 'between' to describe a range of numbers."
+link: 'https://developers.google.com/style/hyphens'
+nonword: true
+level: suggestion
+tokens:
+ - '(?:from|between)\s\d+\s?-\s?\d+'
\ No newline at end of file
diff --git a/vale-styles/FluentBit/Repetition.yml b/vale-styles/FluentBit/Repetition.yml
new file mode 100644
index 000000000..a5158b8b9
--- /dev/null
+++ b/vale-styles/FluentBit/Repetition.yml
@@ -0,0 +1,7 @@
+---
+extends: repetition
+message: '"%s" is repeated.'
+level: suggestion
+alpha: true
+tokens:
+ - '[^\s]+'
diff --git a/vale-styles/FluentBit/SentenceLengthLong.yml b/vale-styles/FluentBit/SentenceLengthLong.yml
new file mode 100644
index 000000000..556580b10
--- /dev/null
+++ b/vale-styles/FluentBit/SentenceLengthLong.yml
@@ -0,0 +1,7 @@
+---
+extends: occurrence
+message: "Improve readability by using fewer than 35 words in this sentence."
+scope: sentence
+level: suggestion
+max: 35
+token: \b(\w+)\b
diff --git a/vale-styles/FluentBit/Simplicity.yml b/vale-styles/FluentBit/Simplicity.yml
new file mode 100644
index 000000000..e9b779763
--- /dev/null
+++ b/vale-styles/FluentBit/Simplicity.yml
@@ -0,0 +1,12 @@
+---
+extends: existence
+message: 'Avoid words like "%s" that imply ease of use, because the user may find this action difficult.'
+level: suggestion
+ignorecase: true
+tokens:
+ - easy
+ - easily
+ - handy
+ - simple
+ - simply
+ - useful
diff --git a/vale-styles/FluentBit/Slang.yml b/vale-styles/FluentBit/Slang.yml
new file mode 100644
index 000000000..b43eeb299
--- /dev/null
+++ b/vale-styles/FluentBit/Slang.yml
@@ -0,0 +1,11 @@
+extends: existence
+message: "Don't use internet slang abbreviations such as '%s'."
+link: 'https://developers.google.com/style/abbreviations'
+ignorecase: true
+level: suggestion
+tokens:
+ - 'tl;dr'
+ - ymmv
+ - rtfm
+ - imo
+ - fwiw
\ No newline at end of file
diff --git a/vale-styles/FluentBit/Spacing.yml b/vale-styles/FluentBit/Spacing.yml
new file mode 100644
index 000000000..57c52f046
--- /dev/null
+++ b/vale-styles/FluentBit/Spacing.yml
@@ -0,0 +1,8 @@
+extends: existence
+message: "'%s' should have one space."
+link: 'https://developers.google.com/style/sentence-spacing'
+level: suggestion
+nonword: true
+tokens:
+ - '[a-z][.?!] {2,}[A-Z]'
+ - '[a-z][.?!][A-Z]'
\ No newline at end of file
diff --git a/vale-styles/FluentBit/Spelling-exceptions.txt b/vale-styles/FluentBit/Spelling-exceptions.txt
new file mode 100644
index 000000000..fa3157ce2
--- /dev/null
+++ b/vale-styles/FluentBit/Spelling-exceptions.txt
@@ -0,0 +1,179 @@
+accessor
+Alertmanager
+allowlist
+API
+APIs
+Appname
+autoscale
+autoscaler
+autoscaling
+backoff
+Blackhole
+blocklist
+Buildkite
+cAdvisor
+Calyptia
+chronotf
+clickstreams
+CloudWatch
+Config
+Coralogix
+Crowdstrike
+CRDs
+DaemonSet
+Datadog
+Datagen
+datapoint
+datapoints
+Datastream
+declaratively
+deduplicate
+Deployer
+deprovision
+deprovisioned
+deprovisioning
+deprovisions
+Devo
+DogStatsD
+downsample
+downsampled
+downsamples
+downsampling
+downscale
+downscaling
+downscales
+dri
+Dynatrace
+Elasticsearch
+endcode
+endhint
+endtab
+endtabs
+Exabeam
+Fargate
+Firehose
+FluentBit
+Fluentd
+Golang
+golib
+Grafana
+Graphite
+Greylog
+grpc_code
+grpc_method
+grpc_service
+gzip
+HashiCorp
+hostname
+Hostname
+Ingester
+Keepalive
+Istio
+keepalive
+Kinesis
+kubectl
+kubelet
+Kubernetes
+Kusto
+labelset
+loadgenerator
+Logstash
+Lua
+Lucene
+macOS
+Mandiant
+matchers
+Minishift
+minikube
+MTTx
+namespace
+namespaces
+Nginx
+OAuth
+Okta
+Oniguruma
+OpenTelemetry
+Opsgenie
+OTel
+PagerDuty
+performant
+persistable
+Postgres
+PowerShell
+prepopulate
+Profiler
+Prometheus
+PromQL
+Protobuf
+proxying
+Pulumi
+Pushgateway
+quantile
+quantiles
+queryable
+Queryable
+rdkafka
+Redpanda
+rollup
+Rollup
+rollups
+Rollups
+routable
+runbook
+runbooks
+Scalyr
+SDKs
+SELinux
+serverless
+ServiceDiscovery
+ServiceMonitor
+ServiceMonitors
+sharding
+SignalFx
+Signup
+sparkline
+sparklines
+Sparklines
+Splunk
+Stackdriver
+StatsD
+stderr
+stdout
+strftime
+subcommand
+subcommands
+subquery
+subrecord
+substring
+syslog
+systemctl
+Systemd
+Tanzu
+Telegraf
+templated
+temporality
+Terraform
+Thanos
+Timeshift
+tolerations
+tooltip
+tooltips
+uber
+unaggregated
+unary
+Unary
+unmuted
+unsort
+UUIDs
+Vectra
+Vercel
+VictoriaMetrics
+VictorOps
+Vivo
+VMs
+Wavefront
+Worldmap
+Zipkin
+Zsh
+Zstandard
+zstd
diff --git a/vale-styles/FluentBit/Spelling.yml b/vale-styles/FluentBit/Spelling.yml
new file mode 100644
index 000000000..46cc6dc74
--- /dev/null
+++ b/vale-styles/FluentBit/Spelling.yml
@@ -0,0 +1,5 @@
+extends: spelling
+message: "Spelling check: '%s'?"
+level: suggestion
+ignore:
+ - FluentBit/Spelling-exceptions.txt
diff --git a/vale-styles/FluentBit/Subjunctive.yml b/vale-styles/FluentBit/Subjunctive.yml
new file mode 100644
index 000000000..f7087389c
--- /dev/null
+++ b/vale-styles/FluentBit/Subjunctive.yml
@@ -0,0 +1,14 @@
+---
+extends: existence
+message: "Use the indicative or imperative moods when writing docs."
+ignorecase: true
+level: suggestion
+tokens:
+ - should
+ - shouldn't
+ - should not
+ - won't
+ - would
+ - wouldn't
+ - could
+ - couldn't
diff --git a/vale-styles/FluentBit/Terms.yml b/vale-styles/FluentBit/Terms.yml
new file mode 100644
index 000000000..6ad4cf2e3
--- /dev/null
+++ b/vale-styles/FluentBit/Terms.yml
@@ -0,0 +1,13 @@
+---
+extends: substitution
+message: Use '%s' instead of '%s'.
+level: suggestion
+ignorecase: true
+scope: paragraph
+action:
+ name: replace
+swap:
+ datapoints: data points
+ Terraform Provider: Terraform provider
+ timeseries: time series
+ topology: placement
diff --git a/vale-styles/FluentBit/Units.yml b/vale-styles/FluentBit/Units.yml
new file mode 100644
index 000000000..786c1d8bb
--- /dev/null
+++ b/vale-styles/FluentBit/Units.yml
@@ -0,0 +1,11 @@
+extends: existence
+message: "Put a nonbreaking space between the number and the unit in '%s'."
+link: 'https://developers.google.com/style/units-of-measure'
+nonword: true
+level: suggestion
+tokens:
+ - \d+(?:B|kB|MB|GB|TB)
+ - \d+(?:ns|ms|s|min|h|d)
+
+exceptions:
+ - k3s
diff --git a/vale-styles/FluentBit/UserFocus.yml b/vale-styles/FluentBit/UserFocus.yml
new file mode 100644
index 000000000..340ad71e7
--- /dev/null
+++ b/vale-styles/FluentBit/UserFocus.yml
@@ -0,0 +1,17 @@
+---
+extends: existence
+message: "Rewrite to put the focus on what the user wants to do rather than on how the document is laid out."
+ignorecase: true
+level: suggestion
+tokens:
+ - The purpose of this document is
+ - This document (?:describes|explains|shows)
+ - This page (?:describes|explains|shows)
+ - This page (?:describes|explains|shows)
+ - This document (?:describes|explains|shows)
+ - This section (?:describes|explains|shows)
+ - This section (?:describes|explains|shows)
+ - The following page (?:describes|explains|shows)
+ - The following document (?:describes|explains|shows)
+ - The following section (?:describes|explains|shows)
+ - This topic (?:describes|explains|shows)
diff --git a/vale-styles/FluentBit/We.yml b/vale-styles/FluentBit/We.yml
new file mode 100644
index 000000000..ffe69e65d
--- /dev/null
+++ b/vale-styles/FluentBit/We.yml
@@ -0,0 +1,11 @@
+extends: existence
+message: "Try to avoid using first-person plural like '%s'."
+link: 'https://developers.google.com/style/pronouns#personal-pronouns'
+level: suggestion
+ignorecase: true
+tokens:
+ - we
+ - we'(?:ve|re)
+ - ours?
+ - us
+ - let's
\ No newline at end of file
diff --git a/vale-styles/FluentBit/WordList.yml b/vale-styles/FluentBit/WordList.yml
new file mode 100644
index 000000000..aca77474a
--- /dev/null
+++ b/vale-styles/FluentBit/WordList.yml
@@ -0,0 +1,79 @@
+extends: substitution
+message: "Use '%s' instead of '%s'."
+link: 'https://developers.google.com/style/word-list'
+level: suggestion
+ignorecase: false
+action:
+ name: replace
+swap:
+ '(?:API Console|dev|developer) key': API key
+ '(?:cell ?phone|smart ?phone)': phone|mobile phone
+ '(?:dev|developer|APIs) console': API console
+ '(?:e-mail|Email|E-mail)': email
+ '(?:file ?path|path ?name)': path
+ '(?:kill|terminate|abort)': stop|exit|cancel|end
+ '(?:OAuth ?2|Oauth)': OAuth 2.0
+ '(?:ok|Okay)': OK|okay
+ '(?:WiFi|wifi)': Wi-Fi
+ '[\.]+apk': APK
+ '3\-D': 3D
+ 'Google (?:I\-O|IO)': Google I/O
+ 'tap (?:&|and) hold': touch & hold
+ 'un(?:check|select)': clear
+ # account name: username
+ action bar: app bar
+ admin: administrator
+ Ajax: AJAX
+ allows you to: lets you
+ Android device: Android-powered device
+ android: Android
+ API explorer: APIs Explorer
+ application: app
+ approx\.: approximately
+ as well as: and
+ authN: authentication
+ authZ: authorization
+ autoupdate: automatically update
+ cellular data: mobile data
+ cellular network: mobile network
+ chapter: documents|pages|sections
+ check box: checkbox
+ check: select
+ click on: click|click in
+ Container Engine: Kubernetes Engine
+ content type: media type
+ curated roles: predefined roles
+ data are: data is
+ Developers Console: Google API Console|API Console
+ ephemeral IP address: ephemeral external IP address
+ fewer data: less data
+ file name: filename
+ firewalls: firewall rules
+ functionality: capability|feature
+ Google account: Google Account
+ Google accounts: Google Accounts
+ Googling: search with Google
+ grayed-out: unavailable
+ HTTPs: HTTPS
+ in order to: to
+ # ingest: import|load
+ k8s: Kubernetes
+ long press: touch & hold
+ network IP address: internal IP address
+ omnibox: address bar
+ open-source: open source
+ overview screen: recents screen
+ regex: regular expression
+ SHA1: SHA-1|HAS-SHA1
+ sign into: sign in to
+ \w* ?sign-?on: single sign-on
+ static IP address: static external IP address
+ stylesheet: style sheet
+ synch: sync
+ tablename: table name
+ tablet: device
+ touch: tap
+ url: URL
+ vs\.: versus
+ wish: want
+ World Wide Web: web