Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update pipeline config files section #48

Merged
merged 10 commits into from
Mar 12, 2024
34 changes: 18 additions & 16 deletions docs/pipeline-configuration-files/specifications.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -43,20 +43,21 @@ pipelines: # A list of pipeline configurations
name: my-file-destination
settings:
path: ./file2.txt
processors: # A list of processor configurations, processors are attached to the connector
- id: proc1 # Processor ID [required]
type: js # Processor type [required]
workers: 2 # Number of parallel workers
settings: # A map of configuration keys and values for the processor (specific to the chosen type)
processors: # A list of processor configurations, processors are attached to the connector
- id: proc1 # Processor ID [required]
plugin: custom.javascript # Processor plugin name [required]
workers: 2 # Number of parallel workers
settings: # A map of configuration keys and values for the processor (specific to the chosen type)
script: >
function process(record) {
return record; // pass through
}
processors: # A list of processor configurations, processors are attached to the pipeline
- id: proc2 # Processor ID [required]
type: hoistfieldpayload # Processor type [required]
settings: # A map of configuration keys and values for the processor (specific to the chosen type)
field: ${ENV_VAR} # You can use environment variables by wrapping them in a dollar sign and curly braces ${}
processors: # A list of processor configurations, processors are attached to the pipeline
- id: proc2 # Processor ID [required]
plugin: field.set # Processor type [required]
settings: # A map of configuration keys and values for the processor (specific to the chosen type)
field: .Payload.After.key # You can use environment variables by wrapping them in a dollar sign and curly braces ${}
value: ${ENV_VAR} # You can use environment variables by wrapping them in a dollar sign and curly braces ${}
dead-letter-queue: # Dead-letter queue (DLQ) configuration
plugin: "builtin:file" # DLQ Connector plugin
settings: # A map of configuration keys and values for the plugin (specific to the chosen plugin)
Expand Down Expand Up @@ -267,7 +268,7 @@ beginning.
- **Default**: None
- **Description**: This node contains the configuration of a single processor.

### id
### plugin
maha-hajja marked this conversation as resolved.
Show resolved Hide resolved

- **Data Type**: String
- **Required**: Yes
Expand All @@ -276,22 +277,23 @@ beginning.
identify the processor inside the pipeline. If multiple processors inside the
pipeline use the same ID, provisioning will fail.

### type
### plugin
maha-hajja marked this conversation as resolved.
Show resolved Hide resolved

- **Data Type**: String
- **Required**: Yes
- **Default**: None
- **Description**: Defines the processor type (e.g. `js`). Check out the
[processors documentation](/docs/processors/getting-started) to find out which
processors are supported.
- **Description**: Defines the processor's plugin name (e.g. `field.set`). Check out the
[processors documentation](/docs/processors/builtin) to find the list of builtin processors
we provide. Also, check [Building processors](/docs/processors/building-processors)
maha-hajja marked this conversation as resolved.
Show resolved Hide resolved
for how to build and use your own WASM processor.

### settings

- **Data Type**: Mapping node
- **Required**: Yes
- **Default**: None
- **Description**: A map of configuration keys and values for the processor. The
values in this map depend on the chosen processor [`type`](#type-1).
values in this map depend on the chosen processor [`plugin`](#plugin-1).
maha-hajja marked this conversation as resolved.
Show resolved Hide resolved

### workers

Expand Down