-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rpk: add --compression flag to rpk transform deploy #832
Open
Deflaimun
wants to merge
4
commits into
main
Choose a base branch
from
DOC-369-Docs-update-for-data-transforms-output-batch-compression
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
3970161
rpk: add --compression flag to rpk transform deploy
Deflaimun 10c53fd
Update modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy…
Deflaimun 5f7898c
Update modules/reference/pages/rpk/rpk-transform/rpk-transform-deploy…
Deflaimun 70d2784
remove extra line
Deflaimun File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -8,10 +8,8 @@ When run in the same directory as a `transform.yaml`, this reads the configurati | |||||
|
||||||
You can specify environment variables for the transform using the `--var` flag. Variables are separated by an equal sign. For example: `--var=KEY=VALUE`. The `--var` flag can be repeated to specify multiple variables. | ||||||
|
||||||
|
||||||
You can specify the `--from-offset` flag to identify where on the input topic the transform should begin processing. Expressed as: | ||||||
|
||||||
|
||||||
- `@T` - Begin reading records with committed timestamp >= T (UNIX time, ms from epoch) | ||||||
|
||||||
- `+N` - Begin reading N records from the start of each input partition | ||||||
|
@@ -20,7 +18,6 @@ You can specify the `--from-offset` flag to identify where on the input topic th | |||||
|
||||||
Note that the broker will only respect `--from-offset` on the first deploy for a given transform. Re-deploying the transform will cause processing to pick up at the last committed offset. This state is maintained until the transform is deleted. | ||||||
|
||||||
|
||||||
== Usage | ||||||
|
||||||
[,bash] | ||||||
|
@@ -34,8 +31,12 @@ rpk transform deploy [flags] | |||||
|=== | ||||||
|*Value* |*Type* |*Description* | ||||||
|
||||||
|--compression |string |Output batch compression type. | ||||||
|
||||||
|--file |string |The WebAssembly module to deploy. | ||||||
|
||||||
|--from-offset |string |Process an input topic partition from a relative offset. | ||||||
|
||||||
|-h, --help |- |Help for deploy. | ||||||
|
||||||
|-i, --input-topic |string |The input topic to apply the transform to. | ||||||
|
@@ -48,7 +49,6 @@ rpk transform deploy [flags] | |||||
|
||||||
|--config |string |Redpanda or `rpk` config file; default search paths are `/var/lib/redpanda/.config/rpk/rpk.yaml`, `$PWD/redpanda.yaml`, and `/etc/redpanda/redpanda.yaml`. | ||||||
|
||||||
|
||||||
|-X, --config-opt |stringArray |Override `rpk` configuration settings. See xref:reference:rpk/rpk-x-options.adoc[`rpk -X`] or execute `rpk -X help` for inline detail or `rpk -X list` for terser detail. | ||||||
|
||||||
|--profile |string |Profile to use. See xref:reference:rpk/rpk-profile.adoc[`rpk profile`] for more details. | ||||||
|
@@ -76,4 +76,24 @@ Deploy a transformation with multiple environment variables: | |||||
rpk transform deploy --var FOO=BAR --var FIZZ=BUZZ | ||||||
---- | ||||||
|
||||||
Configure compression for batches output by data transforms. The default setting is "no compression," but you can choose from the following options: | ||||||
|
||||||
* none | ||||||
* gzip | ||||||
* snappy | ||||||
* lz4 | ||||||
* zstd | ||||||
|
||||||
Configure this at deployment using `rpk` with the `--compression` flag: | ||||||
|
||||||
[,bash] | ||||||
---- | ||||||
rpk transform deploy --compression <compression_type> | ||||||
---- | ||||||
|
||||||
Enabling compression may increase computation costs and could impact latency at the output topic. | ||||||
|
||||||
For more details, see xref:modules:deploy/deployment-option/self-hosted/manual/sizing.adoc[]. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Please double check - maybe ROOT needs to be added? |
||||||
|
||||||
|
||||||
// end::single-source[] |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is "no compression" the same as
none
listed below?