OpenTelemetry (syslog-ng#42)

Documented OpenTelemetry source, destination and parser.

Signed-off-by: Zsolt Gyulai (zgyulai) <[email protected]>

_data/external_links.yml

@@ -906,6 +906,21 @@ opensearch:
   url: https://opensearch.org
   title: [ "OpenSearch" ]
 
+op-tel:
+  id: op-tel
+  url: https://opentelemetry.io/
+  title: [ "OpenTelemetry" ]
+
+op-tel-prtcl:
+  id: op-tel-prtcl
+  url: https://opentelemetry.io/docs/specs/otlp/
+  title: [ "OpenTelemetry Protocol (OTLP/gRPC)" ]
+
+gh-op-tel-prot:
+  id: gh-op-tel-prot
+  url: https://github.com/open-telemetry/opentelemetry-proto/tree/v0.20.0/opentelemetry/proto
+  title: [ "OpenTelemetry proto files" ]
+
 osquery:
   id: osquery
   url: https://osquery.io
@@ -1165,3 +1180,4 @@ w3c-xml:
   id: w3c-xml
   url: https://www.w3.org/TR/REC-xml/
   title: [ "W3C Recommendation: Extensible Markup Language (XML)" ]
+

doc/_admin-guide/060_Sources/102_OpenTelemetry/000_opentelemetry_source_options.md

@@ -0,0 +1,81 @@
+---
+title: opentelemetry() source options
+id: adm-src-optel-opt
+---
+
+The following options are available for the `opentelemetry()` source.
+
+## auth()
+
+The `auth()` option can be used to set the authentication of the driver. The default state of this option is `insecure`, as it is not defined.
+
+The following sub-options are available for `auth()`.
+
+### adc()
+
+This option is an authentication method that is only available for destinations. For more information, see Application Default Credentials.
+
+### alts()
+
+This option is an accessible authentication available for Google infrastructures. Service accounts can be listed with the nested `target-service-account()` option, to match these against the server.
+
+#### Example: configure an opentelemetry() source using auth(alts())
+
+```config
+source {
+    opentelemetry(
+      port(4317)
+      auth(alts())
+    );
+  };
+
+```
+
+### insecure()
+
+This option can be used to disable authentication: `auth(insecure())`.
+
+### tls()
+
+The `tls()` option accepts the following nested sub-options.
+* ca-file()
+* key-file()
+* cert-file()
+
+#### Example: configure an OpenTelemetry source using auth(tls())
+
+```config
+destination {
+    opentelemetry(
+      url("your-otel-server:1234")
+      auth(
+        tls(
+          ca-file("/path/to/ca.pem")
+          key-file("/path/to/key.pem")
+          cert-file("/path/to/cert.pem")
+        )
+      )
+    );
+  };
+
+```
+
+{% include doc/admin-guide/options/channel-args.md %}
+
+## concurrent-requests()
+
+|   Type:|       integer|
+|Default:|             2|
+
+*Description:* This option configures the upper limit of in-flight gRPC requests per worker. It is advisd to set this value in the range of 10s or 100s when there are a high number of clients sending simultaneously. In an optimzed solution, the number of `workers()` and `concurrent-requests()` is greater than or equal to the number of clients. However, this can cause an increase in memory usage.
+
+## log-fetch-limit()
+
+|   Type:|       number|
+|Default:|          100|
+
+*Description:* This option specifies the upper limit of messages received from a source during a single poll loop. The destination queues can fill up before flow-control could stop reading if the defined `log-fetch-limit()` is too high.
+
+{% include doc/admin-guide/options/port.md %}
+
+{% include doc/admin-guide/options/workers.md %}

doc/_admin-guide/060_Sources/102_OpenTelemetry/README.md

@@ -0,0 +1,32 @@
+---
+title: 'OpenTelemetry source'
+short_title: OpenTelemetry
+id: adm-src-optel
+description: >-
+    In syslog-ng OSE 4.3 and later versions it is possible to collect logs, metrics and traces from OpenTelemetry clients. For more information see the OpenTelemetry Protocol (OTLP/gRPC).
+---
+
+In the example below, an OpenTelemetry source and destination are defined. The incoming data is forwarded to a receiver.
+
+### Example: Forwarding OpenTelemetry data
+
+```config
+log otel_forward_mode_alts {
+  source {
+    opentelemetry(
+      port(4317)
+      auth(alts())
+    );
+  };
+
+  destination {
+    opentelemetry(
+      url("otel-server:1234")
+      auth(alts())
+    );
+  };
+};
+```
+
+**NOTE:** The syslog-ng OSE application does not parse the fields of the incoming messages into name-value pairs. It is only capable of forwarding messages using the `opentelemetry()` destination. For information on parsing OpenTelemetry messages, see the OpenTelemetry parser section.
+{: .notice--info}

doc/_admin-guide/070_Destinations/157_OpenTelemetry/000_opentelemetry-destination-options.md

@@ -0,0 +1,76 @@
+---
+title: opentelemetry() destination options 
+id: adm-dest-optel-opt
+---
+
+The following options are available for the opentelemetry() destination.
+
+## auth()
+
+The `auth()` option can be used to set the authentication of the driver. The default state of this option is `insecure`, as it is not defined.
+
+The following sub-options are available for `auth()`.
+
+### adc()
+
+This option is an authentication method that is only available for destinations. For more information, see Application Default Credentials.
+
+### alts()
+
+This option is an accessible authentication available for Google infrastructures. Service accounts can be listed with the nested `target-service-account()` option, to match these against the server.
+
+#### Example: configure an opentelemetry() destination using auth(alts())
+
+```config
+destination {
+  opentelemetry(
+    url("otel-server:1234")
+    auth(alts())
+  );
+};
+
+```
+
+### insecure()
+
+This option can be used to disable authentication: `auth(insecure())`.
+
+### tls()
+
+The `tls()` option accepts the following nested sub-options.
+* ca-file()
+* key-file()
+* cert-file()
+
+```config
+destination {
+    opentelemetry(
+      url("your-otel-server:12346")
+      auth(
+        tls(
+          ca-file("/path/to/ca.pem")
+          key-file("/path/to/key.pem")
+          cert-file("/path/to/cert.pem")
+        )
+      )
+    );
+  };
+```
+{% include doc/admin-guide/options/batch-bytes.md %}
+
+{% include doc/admin-guide/options/batch-lines.md %}
+
+{% include doc/admin-guide/options/batch-timeout.md %}
+
+{% include doc/admin-guide/options/channel-args.md %}
+
+## compression()
+
+|   Type:|       boolean|
+|Default:|          `no`|
+
+Available in syslog-ng OSE 4.5 and later versions.
+
+*Description:* This option enables compression in gRPC requests. Currently, only the deflate compression method is supported.
+
+{% include doc/admin-guide/options/workers.md %}

doc/_admin-guide/070_Destinations/157_OpenTelemetry/README.md

@@ -0,0 +1,57 @@
+---
+title: 'OpenTelemetry destination'
+short_title: OpenTelemetry
+id: adm-dest-optel
+description: >-
+    In syslog-ng OSE 4.3 and later versions it is possible to send logs, metrics and traces from OpenTelemetry clients. For more information see the OpenTelemetry Protocol (OTLP/gRPC).
+---
+
+In the example below, an OpenTelemetry source and destination are defined. It receives data and forwards it to a receiver.
+
+### Example: Forwarding OpenTelemetry data
+
+```config
+log otel_forward_mode_alts {
+  source {
+    opentelemetry(
+      port(12345)
+      auth(alts())
+    );
+  };
+
+  destination {
+    opentelemetry(
+      url("my-otel-server:12345")
+      auth(alts())
+    );
+  };
+};
+```
+
+**NOTE:** Only the `url()` option is mandatory to definie in the destination, which includes the port number.
+{: .notice--info}
+
+### Example: Sending log messages to an OpenTelemetry destination
+
+```config
+log non_otel_to_otel_tls {
+  source {
+    network(
+      port(12346)
+    );
+  };
+
+  destination {
+    opentelemetry(
+      url("my-otel-server:12346")
+      auth(
+        tls(
+          ca-file("/path/to/ca.pem")
+          key-file("/path/to/key.pem")
+          cert-file("/path/to/cert.pem")
+        )
+      )
+    );
+  };
+};
+```

doc/_admin-guide/120_Parser/024_opentelemetry_parser.md

@@ -0,0 +1,51 @@
+---
+title: OpenTelemetry parser
+parser: OpenTelemetry
+id: adm-parser-optel
+---
+
+The syslog-ng OSE application does not parse the fields of the incoming messages into name-value pairs. It is only capable of forwarding messages using the `opentelemetry()` destination. To parse the fields into name-value pairs, use the `opentelemetry()` parser.
+
+This parser parses the fields into name-value pairs starting with the `.otel.` prefix.
+
+* The type of the message is stored in `.otel.type`. Accepted values:
+  * `log`
+  * `metric`
+  * `span`
+* Resource information is mapped into `.otel.resource.<...>`. Examples:
+  * `.otel.resource.dropped_attributes_count`
+  * `.otel.resource.schema_url`
+* Scope information is mapped into `.otel.scope.<...>`. Examples:
+  * `.otel.scope.name`
+  * `.otel.scope.schema_url`
+* The fields of log records are mapped into `.otel.log.<...>`. Examples:
+  * `.otel.log.body`
+  * `.otel.log.severity_text`
+* The fields of metrics are mapped into `.otel.metric.<...>`. Example:
+  * `.otel.metric.name`
+  * `.otel.metric.unit` 
+  * The type of the metric is mapped into `.otel.metric.data.type`. Possible values: 
+    * `gauge`
+    * `sum`
+    * `histogram`
+    * `exponential_histogram`
+    * `summary`
+  * The actual data is mapped into `.otel.metric.data.<type>.<...>`. Example:
+    * `.otel.metric.data.gauge.data_points.0.time_unix_nano`
+* The fields of traces are mapped into `.otel.span.<...>`. Example:
+  * `.otel.span.name`
+  * `.otel.span.trace_state`
+  Repeated fields have an index, for example, `.otel.span.events.5.time_unix_nano`.
+
+For more information on the parsed fields, the OpenTelemetry proto files can be accessed on GitHub.
+
+## Mapping data types
+
+String, bool, int64, double, and bytes values are mapped to their respective syslog-ng OSE name-value type, for example, `.otel.resource.attributes.string_key` becomes a string value.
+
+The mapping of AnyValue type fields is limited.
+
+`ArrayValue` and `KeyValueList` types are stored serialized with the `protobuf` type. 
+
+**NOTE:** `protobuf` and bytes types are only available, unless explicitly type cast. For example, `bytes(${.otel.log.span_id})`. When using template functions, use --include-bytes, for example, `$({format-json} .otel.* --include-bytes)`. In the case of `$({format-json})`, the content is base64-encoded into the bytes content.
+{: .notice--info}