From 17d3d6a484ff71c0aa6e7989708111d4fd55c28e Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Mon, 18 Nov 2024 16:32:16 +0100 Subject: [PATCH 1/5] Adds missing store-legacy-msghdr flag --- content/headless/chunk/option-source-flags.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/content/headless/chunk/option-source-flags.md b/content/headless/chunk/option-source-flags.md index 0911fb23..7f8caff6 100644 --- a/content/headless/chunk/option-source-flags.md +++ b/content/headless/chunk/option-source-flags.md @@ -6,13 +6,16 @@ | | | | -------- | ----- | -| Type: | assume-utf8, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8 | +| Type: | assume-utf8, dont-store-legacy-msghdr, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8 | | Default: | empty set | *Description:* Specifies the log parsing options of the source. - *assume-utf8*: The `assume-utf8` flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the `validate-utf8` flag. - *dont-store-legacy-msghdr*: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before `msg` in the following message: `Jan 22 10:06:11 host program:msg`). If you do not want to store the original header of the message, enable the `dont-store-legacy-msghdr` flag. + + For Python sources, see `store-legacy-msghdr`. + - *empty-lines*: Use the `empty-lines` flag to keep the empty lines of the messages. By default, {{% param "product.abbrev" %}} removes empty lines automatically. - *exit-on-eof*: If this flag is set on a source, {{< product >}} stops when an EOF (end of file) is received. Available in version 4.9 and later. - *expect-hostname*: If the `expect-hostname` flag is enabled, {{% param "product.abbrev" %}} will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the `no-hostname` flag by default. @@ -58,6 +61,10 @@ Essentially, the `no-header` flag signals {{% param "product.abbrev" %}} that th Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages. +- *store-legacy-msghdr*: Available only when using the Python sources ([`python()`]({{< relref "/chapter-sources/python-source/_index.md" >}}) and [`python-fetcher()`]({{< relref "/chapter-sources/python-fetcher-source/_index.md" >}})). If set, {{< product >}} stores the original incoming header of the log message. + + For other sources, see `dont-store-legacy-msghdr`. + - *store-raw-message*: Save the original message as received from the client in the `${RAWMSG}` macro. You can forward this raw message in its original form to another AxoSyslog node using the [`syslog-ng()` destination]({{< relref "/chapter-destinations/destination-syslog-ng/_index.md" >}}), or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later. - *syslog-protocol*: The `syslog-protocol` flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the `syslog` driver, which handles only messages that have a frame header. From 91bea72d2ac08f3885f6ec2971712913fc24456d Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Mon, 18 Nov 2024 16:52:00 +0100 Subject: [PATCH 2/5] [4.8.1] Adds no-piggyback-errors flag --- .../parser-syslog/parser-syslog-options/_index.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/content/chapter-parsers/parser-syslog/parser-syslog-options/_index.md b/content/chapter-parsers/parser-syslog/parser-syslog-options/_index.md index dcfc1053..c31e2c56 100644 --- a/content/chapter-parsers/parser-syslog/parser-syslog-options/_index.md +++ b/content/chapter-parsers/parser-syslog/parser-syslog-options/_index.md @@ -14,7 +14,12 @@ The `syslog-parser()` has the following options: {{< include-headless "chunk/option-source-flags.md" >}} -For the `syslog-parser()` you can also set the `check-hostname` flag, which is equivalent with the [`check-hostname()` global option]({{< relref "/chapter-global-options/reference-options/_index.md#global-option-check-hostname" >}}), but only applies to this parser. +For the `syslog-parser()` you can also set the following flags: + +- `check-hostname`: Equivalent with the [`check-hostname()` global option]({{< relref "/chapter-global-options/reference-options/_index.md#global-option-check-hostname" >}}), but only applies to this parser. +- `no-piggyback-errors`: Do not attribute the message to {{< product >}} in case of errors. Things already processed or extracted are retained, for example: `${MESSAGE}` retains its value (potentially the raw message), other macros like `${HOST}`, `${PROGRAM}`, or `${PID}` may or may not be extracted. The error is indicated by setting `${MSGFORMAT}` set to "syslog:error". + + Available in {{< product >}} 4.8.1 and later. ## sdata-prefix() From a7362e3def2f9fb3c2e83ef7bba9cda49e72d0e9 Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Tue, 19 Nov 2024 15:37:01 +0100 Subject: [PATCH 3/5] [4.8.1] Adds the ${.tls.x509_fp} macro --- .../tlsoptions/_index.md | 6 +++--- .../reference-macros/_index.md | 11 ++++------- 2 files changed, 7 insertions(+), 10 deletions(-) diff --git a/content/chapter-encrypted-transport-tls/tlsoptions/_index.md b/content/chapter-encrypted-transport-tls/tlsoptions/_index.md index e6dd124e..1c7b0232 100644 --- a/content/chapter-encrypted-transport-tls/tlsoptions/_index.md +++ b/content/chapter-encrypted-transport-tls/tlsoptions/_index.md @@ -372,9 +372,9 @@ To find the fingerprint of a certificate, you can use the following command: `op When using the `trusted-keys()` and `trusted-dn()` parameters, note the following: - - First, the `trusted-keys()` parameter is checked. If the fingerprint of the peer is listed, the certificate validation is performed. - - - If the fingerprint of the peer is not listed in the `trusted-keys()` parameter, the `trusted-dn()` parameter is checked. If the DN of the peer is not listed in the `trusted-dn()` parameter, the authentication of the peer fails and the connection is closed. +- First, the `trusted-keys()` parameter is checked. If the fingerprint of the peer is listed, the certificate validation is performed. +- If the fingerprint of the peer is not listed in the `trusted-keys()` parameter, the `trusted-dn()` parameter is checked. If the DN of the peer is not listed in the `trusted-dn()` parameter, the authentication of the peer fails and the connection is closed. {{% /alert %}} +Starting with version 4.8.1, if `trusted-keys()` is set, {{% param "product.abbrev" %}} automatically adds the key fingerprint of the peer to the `${.tls.x509_fp}` name-value pair. diff --git a/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md b/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md index 680fa9eb..b8832135 100644 --- a/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md +++ b/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md @@ -439,13 +439,10 @@ Available in {{% param "product.abbrev" %}} version 4.5 and later. *Description:* When using a transport that uses TLS, these macros contain information about the peer's certificate. That way, you can use information from the client certificate in filenames, database values, or as other metadata. If you clients have their own certificates, then these values are unique per client, but unchangeable by the client. The following macros are available in {{% param "product.abbrev" %}} version 3.9 and later. - - `.tls.x509_cn`: The Common Name of the certificate. - - - `.tls.x509_o`: The value of the Organization field. - - - `.tls.x509_ou`: The value of the Organization Unit field. - - +- `.tls.x509_cn`: The Common Name of the certificate. +- `.tls.x509_o`: The value of the Organization field. +- `.tls.x509_ou`: The value of the Organization Unit field. +- `.tls.x509_fp`: The key fingerprint of the peer, if the [`trusted-keys()` option]({{< relref "/chapter-encrypted-transport-tls/tlsoptions/_index.md#tls-options-trusted-keys" >}}) is used. Available in version 4.8.1 and later. ## UNIQID {#macro-uniqid} From 8a18afc941956067edf9084f64f4408729b87ae5 Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Tue, 19 Nov 2024 16:03:20 +0100 Subject: [PATCH 4/5] [4.8.1] Adds the MSGFORMAT macro --- .../reference-macros/_index.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md b/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md index b8832135..0a1e4f5e 100644 --- a/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md +++ b/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md @@ -232,6 +232,17 @@ The `${MSG}` macro is an alias of the `${MESSAGE}` macro, using `${MSG}` in {{% {{% include-headless "chunk/macro-msghdr.md" %}} +## MSGFORMAT {#macro-msgformat} + +Available in {{% param "product.abbrev" %}} version 4.8.1 and later. + +*Description:* Stores the original format of the incoming message. Possible values: + +- `linux:devkmsg`: Linux kernel message. +- `linux:pacct`: [Linux process accounting log]({{< relref "/chapter-sources/source-pacct/_index.md" >}}) format. +- `raw`: {{% param "product.abbrev" %}} didn't parse the message, for example, because the `no-parse` flag was set. +- `syslog:rfc3164`: Syslog message formatted as RFC3164. +- `syslog:rfc5424`: Syslog message formatted as RFC5424. ## MSGID {#macro-msgid} From a406223570d50d3818a9f9379a90384f3c65c7b9 Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Tue, 19 Nov 2024 16:19:51 +0100 Subject: [PATCH 5/5] store-legacy-msghdr clarifications --- content/headless/chunk/option-source-flags.md | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/content/headless/chunk/option-source-flags.md b/content/headless/chunk/option-source-flags.md index 7f8caff6..ec04264c 100644 --- a/content/headless/chunk/option-source-flags.md +++ b/content/headless/chunk/option-source-flags.md @@ -12,10 +12,7 @@ *Description:* Specifies the log parsing options of the source. - *assume-utf8*: The `assume-utf8` flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the `validate-utf8` flag. -- *dont-store-legacy-msghdr*: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before `msg` in the following message: `Jan 22 10:06:11 host program:msg`). If you do not want to store the original header of the message, enable the `dont-store-legacy-msghdr` flag. - - For Python sources, see `store-legacy-msghdr`. - +- *dont-store-legacy-msghdr*: By default, {{< product >}} stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained ({{< product >}} automatically corrects minor header errors, for example, adds a whitespace before `msg` in the following message: `Jan 22 10:06:11 host program:msg`). If you do not want to store the original header of the message, enable the `dont-store-legacy-msghdr` flag. - *empty-lines*: Use the `empty-lines` flag to keep the empty lines of the messages. By default, {{% param "product.abbrev" %}} removes empty lines automatically. - *exit-on-eof*: If this flag is set on a source, {{< product >}} stops when an EOF (end of file) is received. Available in version 4.9 and later. - *expect-hostname*: If the `expect-hostname` flag is enabled, {{% param "product.abbrev" %}} will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the `no-hostname` flag by default. @@ -61,10 +58,7 @@ Essentially, the `no-header` flag signals {{% param "product.abbrev" %}} that th Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages. -- *store-legacy-msghdr*: Available only when using the Python sources ([`python()`]({{< relref "/chapter-sources/python-source/_index.md" >}}) and [`python-fetcher()`]({{< relref "/chapter-sources/python-fetcher-source/_index.md" >}})). If set, {{< product >}} stores the original incoming header of the log message. - - For other sources, see `dont-store-legacy-msghdr`. - +- *store-legacy-msghdr*: By default, {{< product >}} stores the original incoming header of the log message, so this flag is active. To disable it, use the `dont-store-legacy-msghdr` flag. - *store-raw-message*: Save the original message as received from the client in the `${RAWMSG}` macro. You can forward this raw message in its original form to another AxoSyslog node using the [`syslog-ng()` destination]({{< relref "/chapter-destinations/destination-syslog-ng/_index.md" >}}), or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later. - *syslog-protocol*: The `syslog-protocol` flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the `syslog` driver, which handles only messages that have a frame header.