Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update pipeline config files section #48

Merged
merged 10 commits into from
Mar 12, 2024
41 changes: 26 additions & 15 deletions docs/pipeline-configuration-files/specifications.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -43,20 +43,22 @@ pipelines: # A list of pipeline configurations
name: my-file-destination
settings:
path: ./file2.txt
processors: # A list of processor configurations, processors are attached to the connector
- id: proc1 # Processor ID [required]
type: js # Processor type [required]
workers: 2 # Number of parallel workers
settings: # A map of configuration keys and values for the processor (specific to the chosen type)
processors: # A list of processor configurations, processors are attached to the connector
- id: proc1 # Processor ID [required]
plugin: custom.javascript # Processor plugin name [required]
condition: '{{ eq .Metadata.foo "bar" }}' # Condition (Go template expression) that dictates if the record will be passed to the processor or not.
workers: 2 # Number of parallel workers
settings: # A map of configuration keys and values for the processor (specific to the chosen processor plugin)
script: >
function process(record) {
return record; // pass through
}
processors: # A list of processor configurations, processors are attached to the pipeline
- id: proc2 # Processor ID [required]
type: hoistfieldpayload # Processor type [required]
settings: # A map of configuration keys and values for the processor (specific to the chosen type)
field: ${ENV_VAR} # You can use environment variables by wrapping them in a dollar sign and curly braces ${}
processors: # A list of processor configurations, processors are attached to the pipeline
- id: proc2 # Processor ID [required]
plugin: field.set # Processor type [required]
settings: # A map of configuration keys and values for the processor (specific to the chosen processor plugin)
field: .Payload.After.key
value: ${ENV_VAR} # You can use environment variables by wrapping them in a dollar sign and curly braces ${}
dead-letter-queue: # Dead-letter queue (DLQ) configuration
plugin: "builtin:file" # DLQ Connector plugin
settings: # A map of configuration keys and values for the plugin (specific to the chosen plugin)
Expand Down Expand Up @@ -276,22 +278,31 @@ beginning.
identify the processor inside the pipeline. If multiple processors inside the
pipeline use the same ID, provisioning will fail.

### type
### plugin
maha-hajja marked this conversation as resolved.
Show resolved Hide resolved

- **Data Type**: String
- **Required**: Yes
- **Default**: None
- **Description**: Defines the processor type (e.g. `js`). Check out the
[processors documentation](/docs/processors/getting-started) to find out which
processors are supported.
- **Description**: Defines the processor's plugin name (e.g. `field.set`). Check out the
[processors documentation](/docs/processors/builtin) to find the list of builtin processors
we provide.

### condition

- **Data Type**: String
- **Required**: No
- **Default**: `true` (all records will be passed to the processor)
- **Description**: A [Go template expression](https://pkg.go.dev/text/template) that will be used as a condition to pass the record to the processor or not.
The Go template expression will be evaluated using each record and needs to produce `true` or `false`. If it produces `true`
the record will be passed to the processor, `false` means it will continue down the pipeline without getting processed.

### settings

- **Data Type**: Mapping node
- **Required**: Yes
- **Default**: None
- **Description**: A map of configuration keys and values for the processor. The
values in this map depend on the chosen processor [`type`](#type-1).
values in this map depend on the chosen processor [`plugin`](#plugin-2).

### workers

Expand Down