Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rpk: add --compression flag to rpk transform deploy #832

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,8 @@ When run in the same directory as a `transform.yaml`, this reads the configurati

You can specify environment variables for the transform using the `--var` flag. Variables are separated by an equal sign. For example: `--var=KEY=VALUE`. The `--var` flag can be repeated to specify multiple variables.


You can specify the `--from-offset` flag to identify where on the input topic the transform should begin processing. Expressed as:


- `@T` - Begin reading records with committed timestamp >= T (UNIX time, ms from epoch)

- `+N` - Begin reading N records from the start of each input partition
Expand All @@ -20,7 +18,6 @@ You can specify the `--from-offset` flag to identify where on the input topic th

Note that the broker will only respect `--from-offset` on the first deploy for a given transform. Re-deploying the transform will cause processing to pick up at the last committed offset. This state is maintained until the transform is deleted.


== Usage

[,bash]
Expand All @@ -34,8 +31,12 @@ rpk transform deploy [flags]
|===
|*Value* |*Type* |*Description*

|--compression |string |Output batch compression type.

|--file |string |The WebAssembly module to deploy.

|--from-offset |string |Process an input topic partition from a relative offset.

|-h, --help |- |Help for deploy.

|-i, --input-topic |string |The input topic to apply the transform to.
Expand All @@ -48,7 +49,6 @@ rpk transform deploy [flags]

|--config |string |Redpanda or `rpk` config file; default search paths are `/var/lib/redpanda/.config/rpk/rpk.yaml`, `$PWD/redpanda.yaml`, and `/etc/redpanda/redpanda.yaml`.


|-X, --config-opt |stringArray |Override `rpk` configuration settings. See xref:reference:rpk/rpk-x-options.adoc[`rpk -X`] or execute `rpk -X help` for inline detail or `rpk -X list` for terser detail.

|--profile |string |Profile to use. See xref:reference:rpk/rpk-profile.adoc[`rpk profile`] for more details.
Expand Down Expand Up @@ -76,4 +76,24 @@ Deploy a transformation with multiple environment variables:
rpk transform deploy --var FOO=BAR --var FIZZ=BUZZ
----

Configure compression for batches output by data transforms. The default setting is "no compression," but you can choose from the following options:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is "no compression" the same as none listed below?


* none
* gzip
* snappy
* lz4
* zstd

Configure this at deployment using `rpk` with the `--compression` flag:

[,bash]
----
rpk transform deploy --compression <compression_type>
----

Enabling compression may increase computation costs and could impact latency at the output topic.

For more details, see xref:modules:deploy/deployment-option/self-hosted/manual/sizing.adoc[].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
For more details, see xref:modules:deploy/deployment-option/self-hosted/manual/sizing.adoc[].
For more details, see xref:deploy:deployment-option/self-hosted/manual/sizing.adoc[].

Please double check - maybe ROOT needs to be added?



// end::single-source[]