Skip to content

Commit

Permalink
[4.5][http] Adds url templating and worker-partition-key
Browse files Browse the repository at this point in the history
  • Loading branch information
eldarnash committed Jan 4, 2024
1 parent 54fc494 commit 35ed848
Show file tree
Hide file tree
Showing 4 changed files with 40 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -75,5 +75,6 @@ The following destination sends [log messages to an Elasticsearch server using t
{{% include-headless "chunk/http-load-balance-workers.md" %}}
## Templates in the url()
{{< include-headless "chunk/option-destination-http-url-templating.md" >}}
Original file line number Diff line number Diff line change
Expand Up @@ -362,6 +362,9 @@ In case the server on the specified URL returns a redirect request, {{% param "p

{{< include-headless "chunk/destination-load-balancing-url.md" >}}

### Templates in the url()

{{< include-headless "chunk/option-destination-http-url-templating.md" >}}

{{% include-headless "chunk/option-destination-http-user-agent.md" %}}

Expand All @@ -378,6 +381,8 @@ In case the server on the specified URL returns a redirect request, {{% param "p

{{% include-headless "chunk/option-destination-http-use-system-cert-store.md" %}}

{{< include-headless "chunk/option-destination-http-worker-partition-key.md" >}}

{{< include-headless "chunk/option-destination-threaded-workers.md" >}}

{{% include-headless "chunk/http-load-balance-workers.md" %}}
Expand Down
16 changes: 16 additions & 0 deletions content/headless/chunk/option-destination-http-url-templating.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
---
<!-- This file is under the copyright of Axoflow, and licensed under Apache License 2.0, except for using the Axoflow and AxoSyslog trademarks. -->
Available in {{% param "product_name" %}} version 4.5.0 and later.

In {{% param "product_name" %}}, a template can only be resolved on a single message, because the same template might have different resolutions on different messages. As a batch consists of multiple messages, it's not trivial to decide which message should be used for the resolution.

When batching is enabled and multiple workers are configured, it's important to add only those messages to a batch which generate identical URLs. To achieve this, set the [`worker-partition-key()` option](#worker-partition-key) with a template that contains all the templates used in the `url()` option, otherwise messages will be mixed.

For security reasons, all the templated contents in the `url()` option are URL-encoded automatically. The following parts of the URL cannot be templated:

- scheme
- host
- port
- user
- password
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
---
<!-- This file is under the copyright of Axoflow, and licensed under Apache License 2.0, except for using the Axoflow and AxoSyslog trademarks. -->
## worker-partition-key()

| | |
| -------- | ------ |
| Type: | template |
| Default: | |

*Description:* The `worker-partition-key()` option specifies a template: messages that expand the template to the same value are mapped to the same partition. When batching is enabled and multiple workers are configured, it's important to add only those messages to a batch which generate identical URLs. To achieve this, set the `worker-partition-key()` option with a template that contains all the templates used in the `url()` option, otherwise messages will be mixed.

For example, you can partition messages based on the destination host:

```shell
worker-partition-key("$HOST")
```

0 comments on commit 35ed848

Please sign in to comment.