Skip to content

Commit

Permalink
[4.5] Adds postgresql csv parser
Browse files Browse the repository at this point in the history
  • Loading branch information
eldarnash committed Jan 4, 2024
1 parent 0ee7218 commit 54fc494
Show file tree
Hide file tree
Showing 4 changed files with 60 additions and 12 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -133,11 +133,13 @@ Using the `greedy` flag will assign the remainder of the message to the last col
};
```

{{% include-headless "chunk/option-parser-on-type-error.md" %}}

## quote-pairs() {#csv-parser-quote-pairs}

| | |
| --------- | ------------------------------- |
| Synopsis: | quote-pairs('<quote_pairs>) |
| Synopsis: | `quote-pairs(<quote_pairs>)` |

*Description:* List quote-pairs between single quotes. Delimiter characters or strings enclosed between quote characters are ignored. Note that the beginning and ending quote character does not have to be identical, for example, `[}` can also be a quote-pair. For an example of using `quote-pairs()` to parse Apache log files, see [Example: Parsing Apache log files]({{< relref "/chapter-parsers/csv-parser/_index.md" >}}).

Expand Down
38 changes: 38 additions & 0 deletions content/chapter-parsers/postgresql-csvlog-parser/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
title: "PostgreSQL csvlog"
weight: 1650
---
<!-- This file is under the copyright of Axoflow, and licensed under Apache License 2.0, except for using the Axoflow and AxoSyslog trademarks. -->

Available in {{% param "product_name" %}} version 4.5.0 and later.

This parser processes messages in the [PostgreSQL csvlog](https://www.postgresql.org/docs/current/runtime-config-logging.html#RUNTIME-CONFIG-LOGGING-CSVLOG) format.
The following sample message is a multi-line message with embedded NL characters. This is a single, multi-line log entry that starts with the timestamp.

```sh
2023-08-08 12:05:52.805 UTC,,,22113,,64d22fa0.5661,1,,2023-08-08 12:05:52 UTC,23/74060,0,LOG,00000,"automatic vacuum of table ""tablename"": index scans: 0
pages: 0 removed, 4 remain, 0 skipped due to pins, 0 skipped frozen
tuples: 114 removed, 268 remain, 0 are dead but not yet removable, oldest xmin: 149738000
buffer usage: 97 hits, 0 misses, 6 dirtied
avg read rate: 0.000 MB/s, avg write rate: 114.609 MB/s
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s",,,,,,,,,""
```

The `postgresql-csvlog-parser()` extracts the information from this message into a set of name-value pairs. By default, the name-value pairs have the `.pgsql` prefix.

```sh
@version: current

log {
source { file("/var/log/pgsql.log" follow-freq(1) flags(no-parse)); };
parser { postgresql-csvlog-parser() };
destination { ... };
};

```
The `postgresql-csvlog-parser()` driver is actually a reusable configuration snippet configured to parse log messages using the `csv-parser()`. For details on using or writing such configuration snippets, see {{% xref "/chapter-configuration-file/large-configs/config-blocks/_index.md" %}}. You can find the source of this configuration snippet on [GitHub](https://github.com/syslog-ng/syslog-ng/blob/master/scl/pgsql/pgsql.conf).
{{% include-headless "chunk/option-parser-on-type-error.md" %}}
{{% include-headless "chunk/option-parser-prefix.md" %}}
17 changes: 6 additions & 11 deletions content/headless/chunk/option-description-destination-on-error.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,9 @@
<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->
*Description:* Controls what happens when type-casting fails and {{% param "product.abbrev" %}} cannot convert some data to the specified type. By default, {{% param "product.abbrev" %}} drops the entire message and logs the error. Currently the `value-pairs()` option uses the settings of `on-error()`.

- `drop-message`: Drop the entire message and log an error message to the `internal()` source. This is the default behavior of {{% param "product.abbrev" %}}.

- `drop-property`: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the `internal()` source.

- `fallback-to-string`: Convert the property to string and log an error message to the `internal()` source.

- `silently-drop-message`: Drop the entire message silently, without logging the error.

- `silently-drop-property`: Omit the affected property (macro, template, or message-field) silently, without logging the error.

- `silently-fallback-to-string`: Convert the property to string silently, without logging the error.
- `drop-message`: Drop the entire message and log an error message to the `internal()` source. This is the default behavior of {{% param "product.abbrev" %}}.
- `drop-property`: Omit the affected property (macro, template, or message-field) from the log message and log an error message to the `internal()` source.
- `fallback-to-string`: Convert the property to string and log an error message to the `internal()` source.
- `silently-drop-message`: Drop the entire message silently, without logging the error.
- `silently-drop-property`: Omit the affected property (macro, template, or message-field) silently, without logging the error.
- `silently-fallback-to-string`: Convert the property to string silently, without logging the error.
13 changes: 13 additions & 0 deletions content/headless/chunk/option-parser-on-type-error.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
---
<!-- This file is under the copyright of Axoflow, and licensed under Apache License 2.0, except for using the Axoflow and AxoSyslog trademarks. -->

## on-type-error()

| | |
| --------- | ------ |
| Synopsis: | string |

*Description:* Specifies what to do when casting a parsed value to a specific data type fails. Note that the [`flags(drop-invalid)`](#csv-parser-flags) option and the [`on-error()` global option]({{< relref "/chapter-global-options/reference-options/_index.md#global-option-typecasting-on-error" >}}) also affects the behavior.

Accepts the same values as the [`on-error()` global option]({{< relref "/chapter-global-options/reference-options/_index.md#global-option-typecasting-on-error" >}}).

0 comments on commit 54fc494

Please sign in to comment.