From 3cd2b49ca58f73d20d44aa0bccbf11ebdb78767c Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Thu, 7 Nov 2024 15:23:42 -0800 Subject: [PATCH 1/4] admin: networking: updating for style Signed-off-by: Lynette Miles --- administration/networking.md | 126 +++++++++++------- vale-styles/FluentBit/Spelling-exceptions.txt | 1 + 2 files changed, 77 insertions(+), 50 deletions(-) diff --git a/administration/networking.md b/administration/networking.md index 8179e6d3c..c3abd4a25 100644 --- a/administration/networking.md +++ b/administration/networking.md @@ -1,72 +1,95 @@ # Networking -[Fluent Bit](https://fluentbit.io) implements a unified networking interface that is exposed to components like plugins. This interface abstract all the complexity of general I/O and is fully configurable. +[Fluent Bit](https://fluentbit.io) implements a unified networking interface that's +exposed to components like plugins. This interface abstracts all the complexity of +general I/O and is fully configurable. -A common use case is when a component or plugin needs to connect to a service to send and receive data. Despite the operational mode sounds easy to deal with, there are many factors that can make things hard like unresponsive services, networking latency or any kind of connectivity error. The networking interface aims to abstract and simplify the network I/O handling, minimize risks and optimize performance. +A common use case is when a component or plugin needs to connect to a service to send +and receive data. There are many challenges to handle like unresponsive services, +networking latency, or any kind of connectivity error. The networking interface aims +to abstract and simplify the network I/O handling, minimize risks, and optimize +performance. -## Concepts +## Networking concepts -### TCP Connect Timeout +Fluent Bit uses the following networking concepts: -Most of the time creating a new TCP connection to a remote server is straightforward and takes a few milliseconds. But there are cases where DNS resolving, slow network or incomplete TLS handshakes might create long delays, or incomplete connection statuses. - -The `net.connect_timeout` allows to configure the maximum time to wait for a connection to be established, note that this value already considers the TLS handshake process. +### TCP connect timeout -The `net.connect_timeout_log_error` indicates if an error should be logged in case of connect timeout. If disabled, the timeout is logged as debug level message instead. +Most of the time creating a new TCP connection to a remote server is straightforward and takes a few milliseconds. But there are cases where DNS resolving, slow network or incomplete TLS handshakes might create long delays, or incomplete connection statuses. -### TCP Source Address +`net.connect_timeout` lets you configure the maximum time to wait for a connection +to be established. This value already considers the TLS handshake process. -On environments with multiple network interfaces, might be desired to choose which interface to use for our data that will flow through the network. +`net.connect_timeout_log_error` indicates if an error should be logged in case of +connect timeout. If disabled, the timeout is logged as a debug level message. -The `net.source_address` allows to specify which network address must be used for a TCP connection and data flow. +### TCP source address -### Connection Keepalive +On environments with multiple network interfaces, you can choose which +interface to use for Fluent Bit data that will flow through the network. -TCP is a _connected oriented_ channel, to deliver and receive data from a remote end-point in most of cases we use a TCP connection. This TCP connection can be created and destroyed once is not longer needed, this approach has pros and cons, here we will refer to the opposite case: keep the connection open. +Use `net.source_address` to specify which network address to use for a TCP connection +and data flow. -The concept of `Connection Keepalive` refers to the ability of the client \(Fluent Bit on this case\) to keep the TCP connection open in a persistent way, that means that once the connection is created and used, instead of close it, it can be recycled. This feature offers many benefits in terms of performance since communication channels are always established before hand. +### Connection keepalive -Any component that uses TCP channels like HTTP or [TLS](transport-security.md), can take advantage of this feature. For configuration purposes use the `net.keepalive` property. +A connection keepalive refers to the ability of a client to keep the TCP connection +open in a persistent way. This feature offers many benefits in terms +of performance since communication channels are always established before hand. -### Connection Keepalive Idle Timeout +Any component that uses TCP channels like HTTP or [TLS](transport-security.md), can +take advantage of this feature. For configuration purposes use the `net.keepalive` +property. -If a connection is keepalive enabled, there might be scenarios where the connection can be unused for long periods of time. Having an idle keepalive connection is not helpful and is recommendable to keep them alive if they are used. +### Connection keepalive idle timeout -In order to control how long a keepalive connection can be idle, we expose the configuration property called `net.keepalive_idle_timeout`. +If a connection is keepalive enabled, there might be scenarios where the connection +can be unused for long periods of time. Unused connections can be removed. To control +how long a keepalive connection can be idle, Fluent Bit uses a configuration property +called `net.keepalive_idle_timeout`. ### DNS mode -If a transport layer protocol is specified, the plugin whose configuration section the `net.dns.mode` setting is specified on overrides the global `dns.mode` value and issues DNS requests using the specified protocol which can be either TCP or UDP - -### Max Connections Per Worker - -By default, Fluent Bit tries to deliver data as faster as possible and create TCP connections on-demand and in keepalive mode for performance reasons. In high-scalable environments, the user might want to control how many connections are done in parallel by setting a limit. - -This can be done by the configuration property called `net.max_worker_connections` that can be used in the output plugins sections. -This feature acts at the worker level, e.g., if you have 5 workers and `net.max_worker_connections` is set to 10, a max of 50 connections will be allowed. -If the limit is reached, the output plugin will issue a retry. - - -## Configuration Options - -For plugins that rely on networking I/O, the following section describes the network configuration properties available and how they can be used to optimize performance or adjust to different configuration needs: - -| Property | Description | Default | -| :--- |:------------------------------------------------------------------------------------------------------------------------------------------|:--------------| -| `net.connect_timeout` | Set maximum time expressed in seconds to wait for a TCP connection to be established, this include the TLS handshake time. | 10 | -| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | true | -| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). Can be set in the [SERVICE] section and overridden on a per plugin basis if desired. | | -| `net.dns.prefer_ipv4` | Prioritize IPv4 DNS results when trying to establish a connection. | false | -| `net.dns.resolver`| Select the primary DNS resolver type (LEGACY or ASYNC). | | -| `net.keepalive` | Enable or disable connection keepalive support. Accepts a boolean value: on / off. | on | -| `net.keepalive_idle_timeout` | Set maximum time expressed in seconds for an idle keepalive connection. | 30 | -| `net.keepalive_max_recycle` | Set maximum number of times a keepalive connection can be used before it is retired. | 2000 | -| `net.max_worker_connections` | Set maximum number of TCP connections that can be established per worker. | 0 (unlimited) | -| `net.source_address` | Specify network address to bind for data traffic. | | +The global `dns.mode` value issues DNS requests using the specified protocol, either +TCP or UDP. If a transport layer protocol is specified, plugins that configure the +`net.dns.mode` setting override the global setting. + +### Maximum connections per worker + +Fluent Bit tries to deliver data as faster as possible and create TCP connections +on-demand and in keepalive mode for performance reasons. In highly scalable +environments, you might want to control how many connections are done in +parallel by setting a limit. + +Use the property called `net.max_worker_connections` property in the output plugins +sections. This property acts at the worker level. For example, if you have 5 workers +and `net.max_worker_connections` is set to 10, a maximum of 50 connections will be +allowed. If the limit is reached, the output plugin will issue a retry. + +## Configuration options + +The following table describes the network configuration properties available and +their usage in optimizing performance or adjusting configuration needs for plugins +that rely on networking I/O: + +| Property | Description | Default | +| :------- |:------------|:--------| +| `net.connect_timeout` | Set maximum time expressed in seconds to wait for a TCP connection to be established, this include the TLS handshake time. | `10` | +| `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | +| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). Can be set in the [SERVICE] section and overridden on a per plugin basis if desired. | _none_ | +| `net.dns.prefer_ipv4` | Prioritize IPv4 DNS results when trying to establish a connection. | `false` | +| `net.dns.resolver`| Select the primary DNS resolver type (`LEGACY` or `ASYNC`). | _none_ | +| `net.keepalive` | Enable or disable connection keepalive support. Accepts a Boolean value: `on` or `off`.  | `on` | +| `net.keepalive_idle_timeout` | Set maximum time expressed in seconds for an idle keepalive connection. | `30` | +| `net.keepalive_max_recycle` | Set maximum number of times a keepalive connection can be used before it's retired. | `2000` | +| `net.max_worker_connections` | Set maximum number of TCP connections that can be established per worker. | `0` (unlimited) | +| `net.source_address` | Specify network address to bind for data traffic. | _none_ | ## Example -As an example, we will send 5 random messages through a TCP output connection, in the remote side we will use `nc` \(netcat\) utility to see the data. +This example sends 5 random messages through a TCP output connection. The remote +side uses the `nc` (netcat) utility to see the data. Put the following configuration snippet in a file called `fluent-bit.conf`: @@ -96,10 +119,11 @@ Put the following configuration snippet in a file called `fluent-bit.conf`: In another terminal, start `nc` and make it listen for messages on TCP port 9090: ```text -$ nc -l 9090 +nc -l 9090 ``` -Now start Fluent Bit with the configuration file written above and you will see the data flowing to netcat: +Now start Fluent Bit with the configuration file written previously and you will see +the data flowing to netcat: ```text $ nc -l 9090 @@ -110,6 +134,8 @@ $ nc -l 9090 {"date":1587769736.572277,"rand_value":527581343064950185} ``` -If the `net.keepalive` option is not enabled, Fluent Bit will close the TCP connection and netcat will quit, here we can see how the keepalive connection works. +If the `net.keepalive` option isn't enabled, Fluent Bit will close the TCP connection +and netcat will quit. -After the 5 records arrive, the connection will keep idle and after 10 seconds it will be closed due to `net.keepalive_idle_timeout`. +After the 5 records arrive, the connection will idle. After 10 seconds it will be +closed due to `net.keepalive_idle_timeout`. diff --git a/vale-styles/FluentBit/Spelling-exceptions.txt b/vale-styles/FluentBit/Spelling-exceptions.txt index f2da97b32..5831fdffd 100644 --- a/vale-styles/FluentBit/Spelling-exceptions.txt +++ b/vale-styles/FluentBit/Spelling-exceptions.txt @@ -91,6 +91,7 @@ minikube MTTx namespace namespaces +netcat Nginx OAuth Okta From 3a28b23c7e829bd64e2b9cd54d1142f601e9b33a Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Thu, 7 Nov 2024 15:24:48 -0800 Subject: [PATCH 2/4] networking: forgot a line wrap: Signed-off-by: Lynette Miles --- administration/networking.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/administration/networking.md b/administration/networking.md index c3abd4a25..78d3402d6 100644 --- a/administration/networking.md +++ b/administration/networking.md @@ -16,7 +16,10 @@ Fluent Bit uses the following networking concepts: ### TCP connect timeout -Most of the time creating a new TCP connection to a remote server is straightforward and takes a few milliseconds. But there are cases where DNS resolving, slow network or incomplete TLS handshakes might create long delays, or incomplete connection statuses. +Most of the time creating a new TCP connection to a remote server is straightforward +and takes a few milliseconds. But there are cases where DNS resolving, slow network +or incomplete TLS handshakes might create long delays, or incomplete connection +statuses. `net.connect_timeout` lets you configure the maximum time to wait for a connection to be established. This value already considers the TLS handshake process. From 02d79a31180bb560db9ad28482355feeaac3403a Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Thu, 7 Nov 2024 15:25:58 -0800 Subject: [PATCH 3/4] ANd a typo Signed-off-by: Lynette Miles --- administration/networking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/administration/networking.md b/administration/networking.md index 78d3402d6..f6a4fed90 100644 --- a/administration/networking.md +++ b/administration/networking.md @@ -78,7 +78,7 @@ that rely on networking I/O: | Property | Description | Default | | :------- |:------------|:--------| -| `net.connect_timeout` | Set maximum time expressed in seconds to wait for a TCP connection to be established, this include the TLS handshake time. | `10` | +| `net.connect_timeout` | Set maximum time expressed in seconds to wait for a TCP connection to be established, this includes the TLS handshake time. | `10` | | `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | | `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). Can be set in the [SERVICE] section and overridden on a per plugin basis if desired. | _none_ | | `net.dns.prefer_ipv4` | Prioritize IPv4 DNS results when trying to establish a connection. | `false` | From e615b34bcd9faca6eaba85c8f955f79b3eca21ca Mon Sep 17 00:00:00 2001 From: esmerel <6818907+esmerel@users.noreply.github.com> Date: Fri, 8 Nov 2024 08:28:36 -0800 Subject: [PATCH 4/4] Apply suggestions from code review Co-authored-by: Adam Locke Signed-off-by: esmerel <6818907+esmerel@users.noreply.github.com> --- administration/networking.md | 61 ++++++++++++++++++------------------ 1 file changed, 31 insertions(+), 30 deletions(-) diff --git a/administration/networking.md b/administration/networking.md index f6a4fed90..251645730 100644 --- a/administration/networking.md +++ b/administration/networking.md @@ -1,10 +1,10 @@ # Networking [Fluent Bit](https://fluentbit.io) implements a unified networking interface that's -exposed to components like plugins. This interface abstracts all the complexity of +exposed to components like plugins. This interface abstracts the complexity of general I/O and is fully configurable. -A common use case is when a component or plugin needs to connect to a service to send +A common use case is when a component or plugin needs to connect with a service to send and receive data. There are many challenges to handle like unresponsive services, networking latency, or any kind of connectivity error. The networking interface aims to abstract and simplify the network I/O handling, minimize risks, and optimize @@ -16,16 +16,16 @@ Fluent Bit uses the following networking concepts: ### TCP connect timeout -Most of the time creating a new TCP connection to a remote server is straightforward -and takes a few milliseconds. But there are cases where DNS resolving, slow network -or incomplete TLS handshakes might create long delays, or incomplete connection -statuses. +Typically, creating a new TCP connection to a remote server is straightforward +and takes a few milliseconds. However, there are cases where DNS resolving, a slow +network, or incomplete TLS handshakes might create long delays, or incomplete +connection statuses. -`net.connect_timeout` lets you configure the maximum time to wait for a connection -to be established. This value already considers the TLS handshake process. +- `net.connect_timeout` lets you configure the maximum time to wait for a connection + to be established. This value already considers the TLS handshake process. -`net.connect_timeout_log_error` indicates if an error should be logged in case of -connect timeout. If disabled, the timeout is logged as a debug level message. +- `net.connect_timeout_log_error` indicates if an error should be logged in case of + connect timeout. If disabled, the timeout is logged as a debug level message. ### TCP source address @@ -39,15 +39,15 @@ and data flow. A connection keepalive refers to the ability of a client to keep the TCP connection open in a persistent way. This feature offers many benefits in terms -of performance since communication channels are always established before hand. +of performance because communication channels are always established beforehand. Any component that uses TCP channels like HTTP or [TLS](transport-security.md), can -take advantage of this feature. For configuration purposes use the `net.keepalive` +take use feature. For configuration purposes use the `net.keepalive` property. ### Connection keepalive idle timeout -If a connection is keepalive enabled, there might be scenarios where the connection +If a connection keepalive is enabled, there might be scenarios where the connection can be unused for long periods of time. Unused connections can be removed. To control how long a keepalive connection can be idle, Fluent Bit uses a configuration property called `net.keepalive_idle_timeout`. @@ -60,15 +60,16 @@ TCP or UDP. If a transport layer protocol is specified, plugins that configure t ### Maximum connections per worker -Fluent Bit tries to deliver data as faster as possible and create TCP connections -on-demand and in keepalive mode for performance reasons. In highly scalable -environments, you might want to control how many connections are done in -parallel by setting a limit. +For optimal performance, Fluent Bit tries to deliver data quickly and create +TCP connections on-demand and in keepalive mode. In highly scalable +environments, you might limit how many connections are created in +parallel. -Use the property called `net.max_worker_connections` property in the output plugins -sections. This property acts at the worker level. For example, if you have 5 workers -and `net.max_worker_connections` is set to 10, a maximum of 50 connections will be -allowed. If the limit is reached, the output plugin will issue a retry. +Use the `net.max_worker_connections` property in the output plugin section to set +the maximum number of allowed connections. This property acts at the worker level. +For example, if you have five workers and `net.max_worker_connections` is set +to 10, a maximum of 50 connections is allowed. If the limit is reached, the output +plugin issues a retry. ## Configuration options @@ -78,9 +79,9 @@ that rely on networking I/O: | Property | Description | Default | | :------- |:------------|:--------| -| `net.connect_timeout` | Set maximum time expressed in seconds to wait for a TCP connection to be established, this includes the TLS handshake time. | `10` | +| `net.connect_timeout` | Set maximum time expressed in seconds to wait for a TCP connection to be established, including the TLS handshake time. | `10` | | `net.connect_timeout_log_error` | On connection timeout, specify if it should log an error. When disabled, the timeout is logged as a debug message. | `true` | -| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). Can be set in the [SERVICE] section and overridden on a per plugin basis if desired. | _none_ | +| `net.dns.mode` | Select the primary DNS connection type (TCP or UDP). Can be set in the `[SERVICE]` section and overridden on a per plugin basis if desired. | _none_ | | `net.dns.prefer_ipv4` | Prioritize IPv4 DNS results when trying to establish a connection. | `false` | | `net.dns.resolver`| Select the primary DNS resolver type (`LEGACY` or `ASYNC`). | _none_ | | `net.keepalive` | Enable or disable connection keepalive support. Accepts a Boolean value: `on` or `off`.  | `on` | @@ -91,7 +92,7 @@ that rely on networking I/O: ## Example -This example sends 5 random messages through a TCP output connection. The remote +This example sends five random messages through a TCP output connection. The remote side uses the `nc` (netcat) utility to see the data. Put the following configuration snippet in a file called `fluent-bit.conf`: @@ -125,8 +126,8 @@ In another terminal, start `nc` and make it listen for messages on TCP port 9090 nc -l 9090 ``` -Now start Fluent Bit with the configuration file written previously and you will see -the data flowing to netcat: +Start Fluent Bit with the configuration file you defined previously to see +data flowing to netcat: ```text $ nc -l 9090 @@ -137,8 +138,8 @@ $ nc -l 9090 {"date":1587769736.572277,"rand_value":527581343064950185} ``` -If the `net.keepalive` option isn't enabled, Fluent Bit will close the TCP connection -and netcat will quit. +If the `net.keepalive` option isn't enabled, Fluent Bit closes the TCP connection +and netcat quits. -After the 5 records arrive, the connection will idle. After 10 seconds it will be -closed due to `net.keepalive_idle_timeout`. +After the five records arrive, the connection idles. After 10 seconds, the connection +closes due to `net.keepalive_idle_timeout`.