syslogreceiver

Syslog Receiver

Status
Stability	alpha: logs
Distributions	contrib
Issues
Code Owners	@djaglowski, @andrzej-stencel | Seeking more code owners!

Parses Syslogs received over TCP or UDP.

Configuration

Field	Default	Description
tcp	nil	Defined tcp_input operator. (see the TCP configuration section)
udp	nil	Defined udp_input operator. (see the UDP configuration section)
protocol	required	The protocol to parse the syslog messages as. Options are rfc3164 and rfc5424
location	UTC	The geographic location (timezone) to use when parsing the timestamp (Syslog RFC 3164 only). The available locations depend on the local IANA Time Zone database. This page contains many examples, such as America/New_York.
enable_octet_counting	false	Wether or not to enable RFC 6587 Octet Counting on syslog parsing (Syslog RFC 5424 and TCP only).
max_octets	8192	The maximum octets for messages using RFC 6587 Octet Counting on syslog parsing (Syslog RFC 5424 and TCP only).
allow_skip_pri_header	false	Allow parsing records without the PRI header. If this setting is enabled, messages without the PRI header will be successfully parsed. The SeverityNumber and SeverityText fields as well as the priority and facility attributes will not be set on the log record. If this setting is disabled (the default), messages without PRI header will throw an exception. To set this setting to true, the enable_octet_counting setting must be false.
non_transparent_framing_trailer	nil	The framing trailer, either LF or NUL, when using RFC 6587 Non-Transparent-Framing (Syslog RFC 5424 and TCP only).
attributes	{}	A map of key: value labels to add to the entry's attributes
resource	{}	A map of key: value labels to add to the entry's resource
operators	[]	An array of operators. See below for more details
retry_on_failure.enabled	false	If true, the receiver will pause reading a file and attempt to resend the current batch of logs if it encounters an error from downstream components.
retry_on_failure.initial_interval	1 second	Time to wait after the first failure before retrying.
retry_on_failure.max_interval	30 seconds	Upper bound on retry backoff interval. Once this value is reached the delay between consecutive retries will remain constant at the specified value.
retry_on_failure.max_elapsed_time	5 minutes	Maximum amount of time (including retries) spent trying to send a logs batch to a downstream consumer. Once this value is reached, the data is discarded. Retrying never stops if set to 0.

Operators

Each operator performs a simple responsibility, such as parsing a timestamp or JSON. Chain together operators to process logs into a desired format.

• Every operator has a type.
• Every operator can be given a unique id. If you use the same type of operator more than once in a pipeline, you must specify an id. Otherwise, the id defaults to the value of type.
• Operators will output to the next operator in the pipeline. The last operator in the pipeline will emit from the receiver. Optionally, the output parameter can be used to specify the id of another operator to which logs will be passed directly.
• Only parsers and general purpose operators should be used.

UDP Configuration

Field	Default	Description
listen_address	required	A listen address of the form <ip>:<port>.
add_attributes	false	Adds net.* attributes according to OpenTelemetry semantic conventions.
multiline		A multiline configuration block. See below for details.
one_log_per_packet	false	Skip log tokenization, set to true if logs contain one log per record and multiline is not used. This will improve performance.
preserve_leading_whitespaces	false	Whether to preserve leading whitespaces.
preserve_trailing_whitespaces	false	Whether to preserve trailing whitespaces.
encoding	utf-8	The encoding of the file being read. See the list of supported encodings below for available options.
async	nil	An async configuration block. See below for details.

TCP Configuration

Field	Default	Description
max_log_size	1MiB	The maximum size of a log entry to read before failing. Protects against reading large amounts of data into memory.
listen_address	required	A listen address of the form <ip>:<port>.
tls	nil	An optional TLS configuration (see the TLS configuration section).
add_attributes	false	Adds net.* attributes according to OpenTelemetry semantic conventions.
multiline		A multiline configuration block. See below for details.
one_log_per_packet	false	Skip log tokenization, set to true if logs contain one log per record and multiline is not used. This will improve performance.
preserve_leading_whitespaces	false	Whether to preserve leading whitespaces.
preserve_trailing_whitespaces	false	Whether to preserve trailing whitespaces.
encoding	utf-8	The encoding of the file being read. See the list of supported encodings below for available options.

TLS Configuration

The tcp_input operator supports TLS, disabled by default.

Field	Default	Description
cert_file		Path to the TLS cert to use for TLS required connections.
key_file		Path to the TLS key to use for TLS required connections.
ca_file		Path to the CA cert. For a client this verifies the server certificate. For a server this verifies client certificates. If empty, the system root CA is used.
client_ca_file		(optional) Path to the TLS cert to use by the server to verify a client certificate. This sets the ClientCAs and ClientAuth to RequireAndVerifyClientCert in the TLSConfig. Please refer to godoc.org/crypto/tls#Config for more information.

multiline configuration

If set, the multiline configuration block instructs the udp_input operator to split log entries on a pattern other than newlines.

note If multiline is not set at all, it won't split log entries at all. Every UDP packet is going to be treated as a log. note multiline detection works per UDP packet due to protocol limitations.

The multiline configuration block must contain exactly one of line_start_pattern or line_end_pattern. These are regex patterns that match either the beginning of a new log entry, or the end of a log entry.

The omit_pattern setting can be used to omit the start/end pattern from each entry.

Supported encodings

Key	Description
nop	No encoding validation. Treats the file as a stream of raw bytes
utf-8	UTF-8 encoding
utf-16le	UTF-16 encoding with little-endian byte order
utf-16be	UTF-16 encoding with big-endian byte order
ascii	ASCII encoding
big5	The Big5 Chinese character encoding

Other less common encodings are supported on a best-effort basis. See https://www.iana.org/assignments/character-sets/character-sets.xhtml for other encodings available.

async configuration

If set, the async configuration block instructs the udp_input operator to read and process logs asynchronously and concurrently.

note If async is not set at all, a single thread will read & process lines synchronously.

Field	Default	Description
readers	1	Concurrency level - Determines how many go routines read from UDP port and push to channel (to be handled by processors).
processors	1	Concurrency level - Determines how many go routines read from channel (pushed by readers) and process logs before sending downstream.
max_queue_length	100	Determines max number of messages which may be waiting for a processor. While the queue is full, the readers will wait until there's room (readers will not drop messages, but they will not read additional incoming messages during that period).

Additional Terminology and Features

• An entry is the base representation of log data as it moves through a pipeline. All operators either create, modify, or consume entries.
• A field is used to reference values in an entry.
• A common expression syntax is used in several operators. For example, expressions can be used to filter or route entries.

Parsers with Embedded Operations

Many parsers operators can be configured to embed certain followup operations such as timestamp and severity parsing. For more information, see complex parsers.

Example Configurations

TCP Configuration:

receivers:
  syslog:
    tcp:
      listen_address: "0.0.0.0:54526"
    protocol: rfc5424

UDP Configuration:

receivers:
  syslog:
    udp:
      listen_address: "0.0.0.0:54526"
    protocol: rfc3164
    location: UTC