Skip to content

Commit

Permalink
Merge pull request #1476 from fluent/lynettemiles/sc-113441/update-pi…
Browse files Browse the repository at this point in the history
…peline-filters-nest
  • Loading branch information
esmerel authored Oct 18, 2024
2 parents 5e90309 + 9a0d31b commit cc6af37
Showing 1 changed file with 74 additions and 50 deletions.
124 changes: 74 additions & 50 deletions pipeline/filters/nest.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
# Nest

The _Nest Filter_ plugin allows you to operate on or with nested data. Its modes of operation are
The _Nest Filter_ plugin lets you operate on or with nested data. Its modes of operation are:

* `nest` - Take a set of records and place them in a map
* `lift` - Take a map by key and lift its records up
- `nest` - Take a set of records and place them in a map.
- `lift` - Take a map by key and lift its records up.

## Example usage \(nest\)
## Example usage for `nest`

As an example using JSON notation, to nest keys matching the `Wildcard` value `Key*` under a new key `NestKey` the transformation becomes,
As an example using JSON notation, to nest keys matching the `Wildcard` value `Key*`
under a new key `NestKey` the transformation becomes:

_Example \(input\)_
Input:

```text
{
Expand All @@ -19,7 +20,7 @@ _Example \(input\)_
}
```

_Example \(output\)_
Output:

```text
{
Expand All @@ -31,11 +32,12 @@ _Example \(output\)_
}
```

## Example usage \(lift\)
## Example usage for `lift`

As an example using JSON notation, to lift keys nested under the `Nested_under` value `NestKey*` the transformation becomes,
As an example using JSON notation, to lift keys nested under the `Nested_under` value
`NestKey*` the transformation becomes:

_Example \(input\)_
Input:

```text
{
Expand All @@ -47,7 +49,7 @@ _Example \(input\)_
}
```

_Example \(output\)_
Output:

```text
{
Expand All @@ -61,40 +63,47 @@ _Example \(output\)_

The plugin supports the following configuration parameters:

| Key | Value Format | Operation | Description |
| Key | Value format | Operation | Description |
| :--- | :--- | :--- | :--- |
| Operation | ENUM \[`nest` or `lift`\] | | Select the operation `nest` or `lift` |
| Wildcard | FIELD WILDCARD | `nest` | Nest records which field matches the wildcard |
| Nest\_under | FIELD STRING | `nest` | Nest records matching the `Wildcard` under this key |
| Nested\_under | FIELD STRING | `lift` | Lift records nested under the `Nested_under` key |
| Add\_prefix | FIELD STRING | ANY | Prefix affected keys with this string |
| Remove\_prefix | FIELD STRING | ANY | Remove prefix from affected keys if it matches this string |
| `Operation` | ENUM [`nest` or `lift`] | | Select the operation `nest` or `lift` |
| `Wildcard` | FIELD WILDCARD | `nest` | Nest records which field matches the wildcard |
| `Nest_under` | FIELD STRING | `nest` | Nest records matching the `Wildcard` under this key |
| `Nested_under` | FIELD STRING | `lift` | Lift records nested under the `Nested_under` key |
| `Add_prefix` | FIELD STRING | ANY | Prefix affected keys with this string |
| `Remove_prefix` | FIELD STRING | ANY | Remove prefix from affected keys if it matches this string |

## Getting Started

In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the [Memory Usage Input Plugin](../inputs/memory-metrics.md), which outputs the following \(example\),
To start filtering records, run the filter from the command line or through the
configuration file. The following example invokes the
[Memory Usage Input Plugin](../inputs/memory-metrics.md), which outputs the
following:

```text
[0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
```

## Example \#1 - nest
## Example 1 - nest

### Command Line

> Note: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.
Using command line mode requires quotes to parse the wildcard properly. The use
of a configuration file is recommended.

The following command will load the _mem_ plugin. Then the _nest_ filter will match the wildcard rule to the keys and nest the keys matching `Mem.*` under the new key `NEST`.
The following command loads the _mem_ plugin. Then the _nest_ filter matches the
wildcard rule to the keys and nests the keys matching `Mem.*` under the new key
`NEST`.

```text
$ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout
```shell copy
bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout
```

### Configuration File

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python

```python copy
[INPUT]
Name mem
Tag mem.local
Expand All @@ -111,10 +120,12 @@ $ bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildc
Nest_under Memstats
Remove_prefix Mem.
```

{% endtab %}

{% tab title="fluent-bit.yaml" %}
```yaml

```yaml copy
pipeline:
inputs:
- name: mem
Expand All @@ -130,6 +141,7 @@ pipeline:
- name: stdout
match: '*'
```
{% endtab %}
{% endtabs %}
Expand All @@ -142,15 +154,17 @@ The output of both the command line and configuration invocations should be iden
[0] mem.local: [1522978514.007359767, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Memstats"=>{"total"=>4050908, "used"=>714984, "free"=>3335924}}]
```
## Example \#2 - nest and lift undo
## Example 2 - nest and lift undo
This example nests all `Mem.*` and `Swap,*` items under the `Stats` key and then reverses these actions with a `lift` operation. The output appears unchanged.
This example nests all `Mem.*` and `Swap.*` items under the `Stats` key and then
reverses these actions with a `lift` operation. The output appears unchanged.

### Configuration File
### Example 2 Configuration File

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python

```python copy
[INPUT]
Name mem
Tag mem.local
Expand All @@ -175,10 +189,11 @@ This example nests all `Mem.*` and `Swap,*` items under the `Stats` key and then
Nested_under Stats
Remove_prefix NESTED
```
{% endtab %}

{% endtab %}
{% tab title="fluent-bit.yaml" %}
```yaml

```yaml copy
pipeline:
inputs:
- name: mem
Expand All @@ -201,6 +216,7 @@ pipeline:
- name: stdout
match: '*'
```

{% endtab %}
{% endtabs %}

Expand All @@ -211,15 +227,17 @@ pipeline:
[0] mem.local: [1529566958.000940636, {"Mem.total"=>8053656, "Mem.used"=>6940380, "Mem.free"=>1113276, "Swap.total"=>16532988, "Swap.used"=>1286772, "Swap.free"=>15246216}]
```

## Example \#3 - nest 3 levels deep
## Example 3 - nest 3 levels deep

This example takes the keys starting with `Mem.*` and nests them under `LAYER1`, which itself is then nested under `LAYER2`, which is nested under `LAYER3`.
This example takes the keys starting with `Mem.*` and nests them under `LAYER1`,
which is then nested under `LAYER2`, which is nested under `LAYER3`.

### Configuration File
### Example 3 Configuration File

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python

```python copy
[INPUT]
Name mem
Tag mem.local
Expand Down Expand Up @@ -249,10 +267,11 @@ This example takes the keys starting with `Mem.*` and nests them under `LAYER1`,
Wildcard LAYER2*
Nest_under LAYER3
```
{% endtab %}

{% endtab %}
{% tab title="fluent-bit.yaml" %}
```yaml

```yaml copy
pipeline:
inputs:
- name: mem
Expand All @@ -277,6 +296,7 @@ pipeline:
- name: stdout
match: '*'
```

{% endtab %}
{% endtabs %}

Expand All @@ -302,15 +322,19 @@ pipeline:
}
```

## Example \#4 - multiple nest and lift filters with prefix
## Example 4 - multiple nest and lift filters with prefix

This example starts with the 3-level deep nesting of _Example 2_ and applies the `lift` filter three times to reverse the operations. The end result is that all records are at the top level, without nesting, again. One prefix is added for each level that is lifted.
This example uses the 3-level deep nesting of _Example 2_ and applies the
`lift` filter three times to reverse the operations. The end result is that all
records are at the top level, without nesting, again. One prefix is added for each
level that's lifted.

### Configuration file

{% tabs %}
{% tab title="fluent-bit.conf" %}
```python

```python copy
[INPUT]
Name mem
Tag mem.local
Expand Down Expand Up @@ -361,10 +385,12 @@ This example starts with the 3-level deep nesting of _Example 2_ and applies the
Nested_under Lifted3_Lifted2_LAYER1
Add_prefix Lifted3_Lifted2_Lifted1_
```

{% endtab %}

{% tab title="fluent-bit.yaml" %}
```yaml

```yaml copy
pipeline:
inputs:
- name: mem
Expand Down Expand Up @@ -404,23 +430,21 @@ pipeline:
- name: stdout
match: '*'
```

{% endtab %}
{% endtabs %}


### Result

```text
[0] mem.local: [1524862951.013414798, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996}]
{
"Swap.total"=>1046524,
"Swap.used"=>0,
"Swap.free"=>1046524,
"Lifted3_Lifted2_Lifted1_Mem.total"=>4050908,
"Lifted3_Lifted2_Lifted1_Mem.used"=>1253912,
"Swap.total"=>1046524,
"Swap.used"=>0,
"Swap.free"=>1046524,
"Lifted3_Lifted2_Lifted1_Mem.total"=>4050908,
"Lifted3_Lifted2_Lifted1_Mem.used"=>1253912,
"Lifted3_Lifted2_Lifted1_Mem.free"=>2796996
}
```

0 comments on commit cc6af37

Please sign in to comment.