Skip to content

Commit

Permalink
update pipeline config files section (#48)
Browse files Browse the repository at this point in the history
* update pipeline config files section

* update type to plugin

* add condition

* delete referencing the "building processor" doc

* Update docs/pipeline-configuration-files/specifications.mdx

Co-authored-by: Lovro Mažgon <[email protected]>

* Update docs/pipeline-configuration-files/specifications.mdx

Co-authored-by: Lovro Mažgon <[email protected]>

* Update docs/pipeline-configuration-files/specifications.mdx

Co-authored-by: Lovro Mažgon <[email protected]>

* Update docs/pipeline-configuration-files/specifications.mdx

Co-authored-by: Lovro Mažgon <[email protected]>

---------

Co-authored-by: Lovro Mažgon <[email protected]>
  • Loading branch information
maha-hajja and lovromazgon authored Mar 12, 2024
1 parent 2df862e commit 678b960
Showing 1 changed file with 26 additions and 15 deletions.
41 changes: 26 additions & 15 deletions docs/pipeline-configuration-files/specifications.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -43,20 +43,22 @@ pipelines: # A list of pipeline configurations
name: my-file-destination
settings:
path: ./file2.txt
processors: # A list of processor configurations, processors are attached to the connector
- id: proc1 # Processor ID [required]
type: js # Processor type [required]
workers: 2 # Number of parallel workers
settings: # A map of configuration keys and values for the processor (specific to the chosen type)
processors: # A list of processor configurations, processors are attached to the connector
- id: proc1 # Processor ID [required]
plugin: custom.javascript # Processor plugin name [required]
condition: '{{ eq .Metadata.foo "bar" }}' # Condition (Go template expression) that dictates if the record will be passed to the processor or not.
workers: 2 # Number of parallel workers
settings: # A map of configuration keys and values for the processor (specific to the chosen processor plugin)
script: >
function process(record) {
return record; // pass through
}
processors: # A list of processor configurations, processors are attached to the pipeline
- id: proc2 # Processor ID [required]
type: hoistfieldpayload # Processor type [required]
settings: # A map of configuration keys and values for the processor (specific to the chosen type)
field: ${ENV_VAR} # You can use environment variables by wrapping them in a dollar sign and curly braces ${}
processors: # A list of processor configurations, processors are attached to the pipeline
- id: proc2 # Processor ID [required]
plugin: field.set # Processor type [required]
settings: # A map of configuration keys and values for the processor (specific to the chosen processor plugin)
field: .Payload.After.key
value: ${ENV_VAR} # You can use environment variables by wrapping them in a dollar sign and curly braces ${}
dead-letter-queue: # Dead-letter queue (DLQ) configuration
plugin: "builtin:file" # DLQ Connector plugin
settings: # A map of configuration keys and values for the plugin (specific to the chosen plugin)
Expand Down Expand Up @@ -276,22 +278,31 @@ beginning.
identify the processor inside the pipeline. If multiple processors inside the
pipeline use the same ID, provisioning will fail.

### type
### plugin

- **Data Type**: String
- **Required**: Yes
- **Default**: None
- **Description**: Defines the processor type (e.g. `js`). Check out the
[processors documentation](/docs/processors/getting-started) to find out which
processors are supported.
- **Description**: Defines the processor's plugin name (e.g. `field.set`). Check out the
[processors documentation](/docs/processors/builtin) to find the list of builtin processors
we provide.

### condition

- **Data Type**: String
- **Required**: No
- **Default**: `true` (all records will be passed to the processor)
- **Description**: A [Go template expression](https://pkg.go.dev/text/template) that will be used as a condition to pass the record to the processor or not.
The Go template expression will be evaluated using each record and needs to produce `true` or `false`. If it produces `true`
the record will be passed to the processor, `false` means it will continue down the pipeline without getting processed.

### settings

- **Data Type**: Mapping node
- **Required**: Yes
- **Default**: None
- **Description**: A map of configuration keys and values for the processor. The
values in this map depend on the chosen processor [`type`](#type-1).
values in this map depend on the chosen processor [`plugin`](#plugin-2).

### workers

Expand Down

0 comments on commit 678b960

Please sign in to comment.