Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: add rate limit config param #123

Merged
merged 8 commits into from
Aug 12, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions docs/connectors/configuration-parameters/rate-limiting.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
---
title: "Rate Limiting"
sidebar_position: 2
---

Destination connectors can be configured to limit the rate at which records can be written. This is especially useful when the destination resource has a rate limit to ensure that the connector does not exceed it. By default, Conduit does not limit the rate at which records are written.

## Configuration parameters

* `sdk.rate.perSecond`: Maximum number of records written per second (0 means no rate limit).
* `sdk.rate.burst`: Allow bursts of at most X records (0 or less means that bursts are not limited). Only takes effect if a rate limit per second is set. Note that if `sdk.batch.size` is bigger than `sdk.rate.burst`, the effective batch size will be equal to `sdk.rate.burst`.

## Example

The pipeline will generate an infinite number of structured records in bursts during 5 seconds at a maximum rate, and write them to the log. When you run it, you'll notice that by using both `sdk.rate.perSecond` and `sdk.rate.burst`, the log destination connector will limit the rate of processed records to 1 burst per second.

```yaml
version: 2.2
pipelines:
- id: generator-to-log
status: running
description: >
Example pipeline using the generator source connector and the log destination connector.
Showing how to limit the rate of processing records.
connectors:
- id: example
type: source
plugin: generator
settings:
rate: 0
recordCount: 0
burst.generateTime: 5s
burst.sleepTime: 0s
lovromazgon marked this conversation as resolved.
Show resolved Hide resolved
format.type: structured
format.options.id: int
format.options.name: string
operations: create
- id: log
type: destination
plugin: log
settings:
level: info

# limit the rate of proccessing records
sdk.rate.perSecond: 1
sdk.rate.burst: 1
raulb marked this conversation as resolved.
Show resolved Hide resolved
```
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Schema Extraction"
sidebar_position: 2
sidebar_position: 3
---

Source and destination connectors can be configured to automatically extract the
Expand Down