From c2ad933e4f805e0ec97cbb41cabe06d46f2c1e11 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Thu, 17 Oct 2024 15:43:29 -0700 Subject: [PATCH 1/3] Docs: nest: bringing nest doc to basic style guidelines Signed-off-by: Lynette Miles --- pipeline/filters/nest.md | 124 +++++++++++++++++++++++---------------- 1 file changed, 74 insertions(+), 50 deletions(-) diff --git a/pipeline/filters/nest.md b/pipeline/filters/nest.md index 96990ca81..43f1b14c3 100644 --- a/pipeline/filters/nest.md +++ b/pipeline/filters/nest.md @@ -1,15 +1,16 @@ # Nest -The _Nest Filter_ plugin allows you to operate on or with nested data. Its modes of operation are +The _Nest Filter_ plugin lets you operate on or with nested data. Its modes of operation are -* `nest` - Take a set of records and place them in a map -* `lift` - Take a map by key and lift its records up +- `nest` - Take a set of records and place them in a map +- `lift` - Take a map by key and lift its records up -## Example usage \(nest\) +## Example usage for nest` -As an example using JSON notation, to nest keys matching the `Wildcard` value `Key*` under a new key `NestKey` the transformation becomes, +As an example using JSON notation, to nest keys matching the `Wildcard` value `Key*` +under a new key `NestKey` the transformation becomes: -_Example \(input\)_ +Input: ```text { @@ -19,7 +20,7 @@ _Example \(input\)_ } ``` -_Example \(output\)_ +Output: ```text { @@ -31,11 +32,12 @@ _Example \(output\)_ } ``` -## Example usage \(lift\) +## Example usage for `lift` -As an example using JSON notation, to lift keys nested under the `Nested_under` value `NestKey*` the transformation becomes, +As an example using JSON notation, to lift keys nested under the `Nested_under` value +`NestKey*` the transformation becomes: -_Example \(input\)_ +Input: ```text { @@ -47,7 +49,7 @@ _Example \(input\)_ } ``` -_Example \(output\)_ +Output: ```text { @@ -61,40 +63,47 @@ _Example \(output\)_ The plugin supports the following configuration parameters: -| Key | Value Format | Operation | Description | +| Key | Value format | Operation | Description | | :--- | :--- | :--- | :--- | -| Operation | ENUM \[`nest` or `lift`\] | | Select the operation `nest` or `lift` | -| Wildcard | FIELD WILDCARD | `nest` | Nest records which field matches the wildcard | -| Nest\_under | FIELD STRING | `nest` | Nest records matching the `Wildcard` under this key | -| Nested\_under | FIELD STRING | `lift` | Lift records nested under the `Nested_under` key | -| Add\_prefix | FIELD STRING | ANY | Prefix affected keys with this string | -| Remove\_prefix | FIELD STRING | ANY | Remove prefix from affected keys if it matches this string | +| `Operation` | ENUM [`nest` or `lift`] | | Select the operation `nest` or `lift` | +| `Wildcard` | FIELD WILDCARD | `nest` | Nest records which field matches the wildcard | +| `Nest_under` | FIELD STRING | `nest` | Nest records matching the `Wildcard` under this key | +| `Nested_under` | FIELD STRING | `lift` | Lift records nested under the `Nested_under` key | +| `Add_prefix` | FIELD STRING | ANY | Prefix affected keys with this string | +| `Remove_prefix` | FIELD STRING | ANY | Remove prefix from affected keys if it matches this string | ## Getting Started -In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the [Memory Usage Input Plugin](../inputs/memory-metrics.md), which outputs the following \(example\), +To start filtering records, run the filter from the command line or through the +configuration file. The following example invokes the +[Memory Usage Input Plugin](../inputs/memory-metrics.md), which outputs the +following: ```text [0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}] ``` -## Example \#1 - nest +## Example 1 - nest ### Command Line -> Note: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended. +Using command line mode requires quotes to parse the wildcard properly. The use +of a configuration file is recommended. -The following command will load the _mem_ plugin. Then the _nest_ filter will match the wildcard rule to the keys and nest the keys matching `Mem.*` under the new key `NEST`. +The following command loads the _mem_ plugin. Then the _nest_ filter matches the +wildcard rule to the keys and nests the keys matching `Mem.*` under the new key +`NEST`. -```text -$ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout +```shell copy +bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout ``` ### Configuration File {% tabs %} {% tab title="fluent-bit.conf" %} -```python + +```python copy [INPUT] Name mem Tag mem.local @@ -111,10 +120,12 @@ $ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildc Nest_under Memstats Remove_prefix Mem. ``` + {% endtab %} {% tab title="fluent-bit.yaml" %} -```yaml + +```yaml copy pipeline: inputs: - name: mem @@ -130,6 +141,7 @@ pipeline: - name: stdout match: '*' ``` + {% endtab %} {% endtabs %} @@ -142,15 +154,17 @@ The output of both the command line and configuration invocations should be iden [0] mem.local: [1522978514.007359767, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Memstats"=>{"total"=>4050908, "used"=>714984, "free"=>3335924}}] ``` -## Example \#2 - nest and lift undo +## Example 2 - nest and lift undo -This example nests all `Mem.*` and `Swap,*` items under the `Stats` key and then reverses these actions with a `lift` operation. The output appears unchanged. +This example nests all `Mem.*` and `Swap.*` items under the `Stats` key and then +reverses these actions with a `lift` operation. The output appears unchanged. -### Configuration File +### Example 2 Configuration File {% tabs %} {% tab title="fluent-bit.conf" %} -```python + +```python copy [INPUT] Name mem Tag mem.local @@ -175,10 +189,11 @@ This example nests all `Mem.*` and `Swap,*` items under the `Stats` key and then Nested_under Stats Remove_prefix NESTED ``` -{% endtab %} +{% endtab %} {% tab title="fluent-bit.yaml" %} -```yaml + +```yaml copy pipeline: inputs: - name: mem @@ -201,6 +216,7 @@ pipeline: - name: stdout match: '*' ``` + {% endtab %} {% endtabs %} @@ -211,15 +227,17 @@ pipeline: [0] mem.local: [1529566958.000940636, {"Mem.total"=>8053656, "Mem.used"=>6940380, "Mem.free"=>1113276, "Swap.total"=>16532988, "Swap.used"=>1286772, "Swap.free"=>15246216}] ``` -## Example \#3 - nest 3 levels deep +## Example 3 - nest 3 levels deep -This example takes the keys starting with `Mem.*` and nests them under `LAYER1`, which itself is then nested under `LAYER2`, which is nested under `LAYER3`. +This example takes the keys starting with `Mem.*` and nests them under `LAYER1`, +which is then nested under `LAYER2`, which is nested under `LAYER3`. -### Configuration File +### Example 3 Configuration File {% tabs %} {% tab title="fluent-bit.conf" %} -```python + +```python copy [INPUT] Name mem Tag mem.local @@ -249,10 +267,11 @@ This example takes the keys starting with `Mem.*` and nests them under `LAYER1`, Wildcard LAYER2* Nest_under LAYER3 ``` -{% endtab %} +{% endtab %} {% tab title="fluent-bit.yaml" %} -```yaml + +```yaml copy pipeline: inputs: - name: mem @@ -277,6 +296,7 @@ pipeline: - name: stdout match: '*' ``` + {% endtab %} {% endtabs %} @@ -302,15 +322,19 @@ pipeline: } ``` -## Example \#4 - multiple nest and lift filters with prefix +## Example 4 - multiple nest and lift filters with prefix -This example starts with the 3-level deep nesting of _Example 2_ and applies the `lift` filter three times to reverse the operations. The end result is that all records are at the top level, without nesting, again. One prefix is added for each level that is lifted. +This example uses the 3-level deep nesting of _Example 2_ and applies the +`lift` filter three times to reverse the operations. The end result is that all +records are at the top level, without nesting, again. One prefix is added for each +level that's lifted. ### Configuration file {% tabs %} {% tab title="fluent-bit.conf" %} -```python + +```python copy [INPUT] Name mem Tag mem.local @@ -361,10 +385,12 @@ This example starts with the 3-level deep nesting of _Example 2_ and applies the Nested_under Lifted3_Lifted2_LAYER1 Add_prefix Lifted3_Lifted2_Lifted1_ ``` + {% endtab %} {% tab title="fluent-bit.yaml" %} -```yaml + +```yaml copy pipeline: inputs: - name: mem @@ -404,23 +430,21 @@ pipeline: - name: stdout match: '*' ``` + {% endtab %} {% endtabs %} - ### Result ```text [0] mem.local: [1524862951.013414798, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996}] - { - "Swap.total"=>1046524, - "Swap.used"=>0, - "Swap.free"=>1046524, - "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, - "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, + "Swap.total"=>1046524, + "Swap.used"=>0, + "Swap.free"=>1046524, + "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, + "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996 } ``` - From e7319f5358fdcd66849546684d1221b606be4c93 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Thu, 17 Oct 2024 15:44:50 -0700 Subject: [PATCH 2/3] Docs: nest: fixing missing tick Signed-off-by: Lynette Miles --- pipeline/filters/nest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/filters/nest.md b/pipeline/filters/nest.md index 43f1b14c3..e9b31136b 100644 --- a/pipeline/filters/nest.md +++ b/pipeline/filters/nest.md @@ -5,7 +5,7 @@ The _Nest Filter_ plugin lets you operate on or with nested data. Its modes of o - `nest` - Take a set of records and place them in a map - `lift` - Take a map by key and lift its records up -## Example usage for nest` +## Example usage for `nest` As an example using JSON notation, to nest keys matching the `Wildcard` value `Key*` under a new key `NestKey` the transformation becomes: From 9a0d31bd142fc8853022daa869b8de97d1189b05 Mon Sep 17 00:00:00 2001 From: esmerel <6818907+esmerel@users.noreply.github.com> Date: Fri, 18 Oct 2024 08:34:58 -0700 Subject: [PATCH 3/3] Apply suggestions from code review Co-authored-by: Craig Norris <112565517+cnorris-cs@users.noreply.github.com> Signed-off-by: esmerel <6818907+esmerel@users.noreply.github.com> --- pipeline/filters/nest.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pipeline/filters/nest.md b/pipeline/filters/nest.md index e9b31136b..b0262d324 100644 --- a/pipeline/filters/nest.md +++ b/pipeline/filters/nest.md @@ -1,9 +1,9 @@ # Nest -The _Nest Filter_ plugin lets you operate on or with nested data. Its modes of operation are +The _Nest Filter_ plugin lets you operate on or with nested data. Its modes of operation are: -- `nest` - Take a set of records and place them in a map -- `lift` - Take a map by key and lift its records up +- `nest` - Take a set of records and place them in a map. +- `lift` - Take a map by key and lift its records up. ## Example usage for `nest`