diff --git a/_data/external_links.yml b/_data/external_links.yml index c08dd40b..b01a2e4e 100644 --- a/_data/external_links.yml +++ b/_data/external_links.yml @@ -906,6 +906,21 @@ opensearch: url: https://opensearch.org title: [ "OpenSearch" ] +op-tel: + id: op-tel + url: https://opentelemetry.io/ + title: [ "OpenTelemetry" ] + +op-tel-prtcl: + id: op-tel-prtcl + url: https://opentelemetry.io/docs/specs/otlp/ + title: [ "OpenTelemetry Protocol (OTLP/gRPC)" ] + +gh-op-tel-prot: + id: gh-op-tel-prot + url: https://github.com/open-telemetry/opentelemetry-proto/tree/v0.20.0/opentelemetry/proto + title: [ "OpenTelemetry proto files" ] + osquery: id: osquery url: https://osquery.io @@ -1165,3 +1180,4 @@ w3c-xml: id: w3c-xml url: https://www.w3.org/TR/REC-xml/ title: [ "W3C Recommendation: Extensible Markup Language (XML)" ] + diff --git a/doc/_admin-guide/060_Sources/102_OpenTelemetry/000_opentelemetry_source_options.md b/doc/_admin-guide/060_Sources/102_OpenTelemetry/000_opentelemetry_source_options.md new file mode 100644 index 00000000..f9c01d26 --- /dev/null +++ b/doc/_admin-guide/060_Sources/102_OpenTelemetry/000_opentelemetry_source_options.md @@ -0,0 +1,81 @@ +--- +title: opentelemetry() source options +id: adm-src-optel-opt +--- + +The following options are available for the `opentelemetry()` source. + +## auth() + +The `auth()` option can be used to set the authentication of the driver. The default state of this option is `insecure`, as it is not defined. + +The following sub-options are available for `auth()`. + +### adc() + +This option is an authentication method that is only available for destinations. For more information, see Application Default Credentials. + +### alts() + +This option is an accessible authentication available for Google infrastructures. Service accounts can be listed with the nested `target-service-account()` option, to match these against the server. + +#### Example: configure an opentelemetry() source using auth(alts()) + +```config +source { + opentelemetry( + port(4317) + auth(alts()) + ); + }; + +``` + +### insecure() + +This option can be used to disable authentication: `auth(insecure())`. + +### tls() + +The `tls()` option accepts the following nested sub-options. +* ca-file() +* key-file() +* cert-file() + +#### Example: configure an OpenTelemetry source using auth(tls()) + +```config +destination { + opentelemetry( + url("your-otel-server:1234") + auth( + tls( + ca-file("/path/to/ca.pem") + key-file("/path/to/key.pem") + cert-file("/path/to/cert.pem") + ) + ) + ); + }; + +``` + +{% include doc/admin-guide/options/channel-args.md %} + +## concurrent-requests() + +| Type:| integer| +|Default:| 2| + +*Description:* This option configures the upper limit of in-flight gRPC requests per worker. It is advisd to set this value in the range of 10s or 100s when there are a high number of clients sending simultaneously. In an optimzed solution, the number of `workers()` and `concurrent-requests()` is greater than or equal to the number of clients. However, this can cause an increase in memory usage. + +## log-fetch-limit() + +| Type:| number| +|Default:| 100| + +*Description:* This option specifies the upper limit of messages received from a source during a single poll loop. The destination queues can fill up before flow-control could stop reading if the defined `log-fetch-limit()` is too high. + +{% include doc/admin-guide/options/port.md %} + +{% include doc/admin-guide/options/workers.md %} diff --git a/doc/_admin-guide/060_Sources/102_OpenTelemetry/README.md b/doc/_admin-guide/060_Sources/102_OpenTelemetry/README.md new file mode 100644 index 00000000..cfc029f0 --- /dev/null +++ b/doc/_admin-guide/060_Sources/102_OpenTelemetry/README.md @@ -0,0 +1,32 @@ +--- +title: 'OpenTelemetry source' +short_title: OpenTelemetry +id: adm-src-optel +description: >- + In syslog-ng OSE 4.3 and later versions it is possible to collect logs, metrics and traces from OpenTelemetry clients. For more information see the OpenTelemetry Protocol (OTLP/gRPC). +--- + +In the example below, an OpenTelemetry source and destination are defined. The incoming data is forwarded to a receiver. + +### Example: Forwarding OpenTelemetry data + +```config +log otel_forward_mode_alts { + source { + opentelemetry( + port(4317) + auth(alts()) + ); + }; + + destination { + opentelemetry( + url("otel-server:1234") + auth(alts()) + ); + }; +}; +``` + +**NOTE:** The syslog-ng OSE application does not parse the fields of the incoming messages into name-value pairs. It is only capable of forwarding messages using the `opentelemetry()` destination. For information on parsing OpenTelemetry messages, see the OpenTelemetry parser section. +{: .notice--info} diff --git a/doc/_admin-guide/070_Destinations/157_OpenTelemetry/000_opentelemetry-destination-options.md b/doc/_admin-guide/070_Destinations/157_OpenTelemetry/000_opentelemetry-destination-options.md new file mode 100644 index 00000000..86d84be2 --- /dev/null +++ b/doc/_admin-guide/070_Destinations/157_OpenTelemetry/000_opentelemetry-destination-options.md @@ -0,0 +1,76 @@ +--- +title: opentelemetry() destination options +id: adm-dest-optel-opt +--- + +The following options are available for the opentelemetry() destination. + +## auth() + +The `auth()` option can be used to set the authentication of the driver. The default state of this option is `insecure`, as it is not defined. + +The following sub-options are available for `auth()`. + +### adc() + +This option is an authentication method that is only available for destinations. For more information, see Application Default Credentials. + +### alts() + +This option is an accessible authentication available for Google infrastructures. Service accounts can be listed with the nested `target-service-account()` option, to match these against the server. + +#### Example: configure an opentelemetry() destination using auth(alts()) + +```config +destination { + opentelemetry( + url("otel-server:1234") + auth(alts()) + ); +}; + +``` + +### insecure() + +This option can be used to disable authentication: `auth(insecure())`. + +### tls() + +The `tls()` option accepts the following nested sub-options. +* ca-file() +* key-file() +* cert-file() + +```config +destination { + opentelemetry( + url("your-otel-server:12346") + auth( + tls( + ca-file("/path/to/ca.pem") + key-file("/path/to/key.pem") + cert-file("/path/to/cert.pem") + ) + ) + ); + }; +``` +{% include doc/admin-guide/options/batch-bytes.md %} + +{% include doc/admin-guide/options/batch-lines.md %} + +{% include doc/admin-guide/options/batch-timeout.md %} + +{% include doc/admin-guide/options/channel-args.md %} + +## compression() + +| Type:| boolean| +|Default:| `no`| + +Available in syslog-ng OSE 4.5 and later versions. + +*Description:* This option enables compression in gRPC requests. Currently, only the deflate compression method is supported. + +{% include doc/admin-guide/options/workers.md %} diff --git a/doc/_admin-guide/070_Destinations/157_OpenTelemetry/README.md b/doc/_admin-guide/070_Destinations/157_OpenTelemetry/README.md new file mode 100644 index 00000000..749db5a6 --- /dev/null +++ b/doc/_admin-guide/070_Destinations/157_OpenTelemetry/README.md @@ -0,0 +1,57 @@ +--- +title: 'OpenTelemetry destination' +short_title: OpenTelemetry +id: adm-dest-optel +description: >- + In syslog-ng OSE 4.3 and later versions it is possible to send logs, metrics and traces from OpenTelemetry clients. For more information see the OpenTelemetry Protocol (OTLP/gRPC). +--- + +In the example below, an OpenTelemetry source and destination are defined. It receives data and forwards it to a receiver. + +### Example: Forwarding OpenTelemetry data + +```config +log otel_forward_mode_alts { + source { + opentelemetry( + port(12345) + auth(alts()) + ); + }; + + destination { + opentelemetry( + url("my-otel-server:12345") + auth(alts()) + ); + }; +}; +``` + +**NOTE:** Only the `url()` option is mandatory to definie in the destination, which includes the port number. +{: .notice--info} + +### Example: Sending log messages to an OpenTelemetry destination + +```config +log non_otel_to_otel_tls { + source { + network( + port(12346) + ); + }; + + destination { + opentelemetry( + url("my-otel-server:12346") + auth( + tls( + ca-file("/path/to/ca.pem") + key-file("/path/to/key.pem") + cert-file("/path/to/cert.pem") + ) + ) + ); + }; +}; +``` diff --git a/doc/_admin-guide/120_Parser/024_opentelemetry_parser.md b/doc/_admin-guide/120_Parser/024_opentelemetry_parser.md new file mode 100644 index 00000000..f85c9962 --- /dev/null +++ b/doc/_admin-guide/120_Parser/024_opentelemetry_parser.md @@ -0,0 +1,51 @@ +--- +title: OpenTelemetry parser +parser: OpenTelemetry +id: adm-parser-optel +--- + +The syslog-ng OSE application does not parse the fields of the incoming messages into name-value pairs. It is only capable of forwarding messages using the `opentelemetry()` destination. To parse the fields into name-value pairs, use the `opentelemetry()` parser. + +This parser parses the fields into name-value pairs starting with the `.otel.` prefix. + +* The type of the message is stored in `.otel.type`. Accepted values: + * `log` + * `metric` + * `span` +* Resource information is mapped into `.otel.resource.<...>`. Examples: + * `.otel.resource.dropped_attributes_count` + * `.otel.resource.schema_url` +* Scope information is mapped into `.otel.scope.<...>`. Examples: + * `.otel.scope.name` + * `.otel.scope.schema_url` +* The fields of log records are mapped into `.otel.log.<...>`. Examples: + * `.otel.log.body` + * `.otel.log.severity_text` +* The fields of metrics are mapped into `.otel.metric.<...>`. Example: + * `.otel.metric.name` + * `.otel.metric.unit` + * The type of the metric is mapped into `.otel.metric.data.type`. Possible values: + * `gauge` + * `sum` + * `histogram` + * `exponential_histogram` + * `summary` + * The actual data is mapped into `.otel.metric.data..<...>`. Example: + * `.otel.metric.data.gauge.data_points.0.time_unix_nano` +* The fields of traces are mapped into `.otel.span.<...>`. Example: + * `.otel.span.name` + * `.otel.span.trace_state` + Repeated fields have an index, for example, `.otel.span.events.5.time_unix_nano`. + +For more information on the parsed fields, the OpenTelemetry proto files can be accessed on GitHub. + +## Mapping data types + +String, bool, int64, double, and bytes values are mapped to their respective syslog-ng OSE name-value type, for example, `.otel.resource.attributes.string_key` becomes a string value. + +The mapping of AnyValue type fields is limited. + +`ArrayValue` and `KeyValueList` types are stored serialized with the `protobuf` type. + +**NOTE:** `protobuf` and bytes types are only available, unless explicitly type cast. For example, `bytes(${.otel.log.span_id})`. When using template functions, use --include-bytes, for example, `$({format-json} .otel.* --include-bytes)`. In the case of `$({format-json})`, the content is base64-encoded into the bytes content. +{: .notice--info}