From 39701610ca36e0a44023dc6089774f85584db3af Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Wed, 30 Oct 2024 12:43:28 -0300 Subject: [PATCH 1/4] rpk: add --compression flag to rpk transform deploy --- .../rpk-transform/rpk-transform-deploy.adoc | 28 ++++++++++++++++--- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc b/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc index 645b2a4df..42012e115 100644 --- a/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc +++ b/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc @@ -8,10 +8,8 @@ When run in the same directory as a `transform.yaml`, this reads the configurati You can specify environment variables for the transform using the `--var` flag. Variables are separated by an equal sign. For example: `--var=KEY=VALUE`. The `--var` flag can be repeated to specify multiple variables. - You can specify the `--from-offset` flag to identify where on the input topic the transform should begin processing. Expressed as: - - `@T` - Begin reading records with committed timestamp >= T (UNIX time, ms from epoch) - `+N` - Begin reading N records from the start of each input partition @@ -20,7 +18,6 @@ You can specify the `--from-offset` flag to identify where on the input topic th Note that the broker will only respect `--from-offset` on the first deploy for a given transform. Re-deploying the transform will cause processing to pick up at the last committed offset. This state is maintained until the transform is deleted. - == Usage [,bash] @@ -34,8 +31,12 @@ rpk transform deploy [flags] |=== |*Value* |*Type* |*Description* +|--compression |string |Output batch compression type. + |--file |string |The WebAssembly module to deploy. +|--from-offset |string |Process an input topic partition from a relative offset; check help text for more information. + |-h, --help |- |Help for deploy. |-i, --input-topic |string |The input topic to apply the transform to. @@ -48,7 +49,6 @@ rpk transform deploy [flags] |--config |string |Redpanda or `rpk` config file; default search paths are `/var/lib/redpanda/.config/rpk/rpk.yaml`, `$PWD/redpanda.yaml`, and `/etc/redpanda/redpanda.yaml`. - |-X, --config-opt |stringArray |Override `rpk` configuration settings. See xref:reference:rpk/rpk-x-options.adoc[`rpk -X`] or execute `rpk -X help` for inline detail or `rpk -X list` for terser detail. |--profile |string |Profile to use. See xref:reference:rpk/rpk-profile.adoc[`rpk profile`] for more details. @@ -76,4 +76,24 @@ Deploy a transformation with multiple environment variables: rpk transform deploy --var FOO=BAR --var FIZZ=BUZZ ---- +Configure compression for batches output by data transforms. The default setting is "no compression," but you can choose from the following options: + +* none +* gzip +* snappy +* lz4 +* zstd + +Configure this at deployment using `rpk` with the `--compression` flag: + +[,bash] +---- +rpk transform deploy --compression +---- + +Enabling compression may increase computation costs and could impact latency at the output topic. + +For more details, see xref:modules:deploy/deployment-option/self-hosted/manual/sizing.adoc[]. + + // end::single-source[] \ No newline at end of file From 10c53fdc5c2009f51b0e03748e74f5a97da73188 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Wed, 30 Oct 2024 17:07:27 -0300 Subject: [PATCH 2/4] Update modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc --- .../pages/rpk/rpk-transform/rpk-transform-deploy.adoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc b/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc index 42012e115..31f921995 100644 --- a/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc +++ b/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc @@ -35,7 +35,8 @@ rpk transform deploy [flags] |--file |string |The WebAssembly module to deploy. -|--from-offset |string |Process an input topic partition from a relative offset; check help text for more information. +|--from-offset |string |Process an input topic partition from a relative offset. Run `rpk transform deploy --from-offset -h` for more information. + |-h, --help |- |Help for deploy. From 5f7898ce87c344d0f1332ff6e45f4d3175dbcad9 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Wed, 30 Oct 2024 17:38:01 -0300 Subject: [PATCH 3/4] Update modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc --- .../reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc b/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc index 31f921995..2d61322a0 100644 --- a/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc +++ b/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc @@ -35,7 +35,7 @@ rpk transform deploy [flags] |--file |string |The WebAssembly module to deploy. -|--from-offset |string |Process an input topic partition from a relative offset. Run `rpk transform deploy --from-offset -h` for more information. +|--from-offset |string |Process an input topic partition from a relative offset. |-h, --help |- |Help for deploy. From 70d2784720909fd67ea839cfa37698eab9b50805 Mon Sep 17 00:00:00 2001 From: Paulo Borges Date: Wed, 30 Oct 2024 18:31:52 -0300 Subject: [PATCH 4/4] remove extra line --- .../reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc | 1 - 1 file changed, 1 deletion(-) diff --git a/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc b/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc index 2d61322a0..f557c8eec 100644 --- a/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc +++ b/modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy.adoc @@ -37,7 +37,6 @@ rpk transform deploy [flags] |--from-offset |string |Process an input topic partition from a relative offset. - |-h, --help |- |Help for deploy. |-i, --input-topic |string |The input topic to apply the transform to.