Skip to content

Commit

Permalink
Add pages about storage, configuration (#107)
Browse files Browse the repository at this point in the history
* Add pages about storage, configuration

* sort alphabetically (#110)

---------

Co-authored-by: Raúl Barroso <[email protected]>
  • Loading branch information
hariso and raulb authored Jul 31, 2024
1 parent aa8fa97 commit 13ededd
Show file tree
Hide file tree
Showing 8 changed files with 66 additions and 6 deletions.
32 changes: 32 additions & 0 deletions docs/features/configuration.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
title: 'Configuration'
sidebar_position: 1
---


Conduit accepts CLI flags, environment variables and a configuration file to
configure its behavior. Each CLI flag has a corresponding environment variable
and a corresponding field in the configuration file. Conduit uses the value for
each configuration option based on the following priorities:

- **CLI flags** (highest priority) - if a CLI flag is provided it will always be
respected, regardless of the environment variable or configuration file. To
see a full list of available flags run `conduit --help`.
- **Environment variables** (lower priority) - an environment variable is only
used if no CLI flag is provided for the same option. Environment variables
have the prefix `CONDUIT` and contain underscores instead of dots and
hyphens (e.g. the flag `-db.postgres.connection-string` corresponds
to `CONDUIT_DB_POSTGRES_CONNECTION_STRING`).
- **Configuration file** (lowest priority) - Conduit by default loads the
file `conduit.yaml` placed in the same folder as Conduit. The path to the file
can be customized using the CLI flag `-config`. It is not required to provide
a configuration file and any value in the configuration file can be overridden
by an environment variable or a flag. The file content should be a YAML
document where keys can be hierarchically split on `.`. For example:

```yaml
db:
type: postgres # corresponds to flag -db.type and env variable CONDUIT_DB_TYPE
postgres:
connection-string: postgres://localhost:5432/conduitdb # -db.postgres.connection-string or CONDUIT_DB_POSTGRES_CONNECTION_STRING
```
2 changes: 1 addition & 1 deletion docs/features/dead-letter-queue.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: 'Dead-letter queue'
sidebar_position: 1
sidebar_position: 2
---

A dead-letter queue is a connector to which Conduit redirects records that can't
Expand Down
2 changes: 1 addition & 1 deletion docs/features/metrics.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: 'Metrics'
sidebar_position: 2
sidebar_position: 3
---

Conduit comes with a number of already defined metrics. The metrics available
Expand Down
2 changes: 1 addition & 1 deletion docs/features/opencdc-record.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: 'OpenCDC record'
sidebar_position: 3
sidebar_position: 4
---

An OpenCDC record in Conduit aims to standardize the format of data records exchanged between different connectors within a data processing pipeline. The primary objective is to ensure compatibility between various combinations of source and destination connectors.
Expand Down
2 changes: 1 addition & 1 deletion docs/features/record-routing.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: 'Record routing'
sidebar_position: 4
sidebar_position: 6
keywords: [ 'synchronization', 'collections', 'routing' ]
---

Expand Down
28 changes: 28 additions & 0 deletions docs/features/storage.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
title: 'Storage'
sidebar_position: 7
---

Conduit's own data (information about pipelines, connectors, etc.) can be stored
in the following databases:
* BadgerDB (default)
* PostgreSQL
* SQLite

It's also possible to store all the data in memory, which is sometimes useful
for development purposes.

The database type used can be configured with the `db.type` parameter (through
any of the [configuration](/docs/features/configuration) options in Conduit).
For example, the CLI flag to use a PostgresSQL database with Conduit is as
follows: `-db.type=postgres`.

Changing database parameters (e.g. the PostgreSQL connection string) is done
through parameters of the following form: `db.<db type>.<parameter name>`. For
example, the CLI flag to use a PostgreSQL instance listening on `localhost:5432`
would be: `-db.postgres.connection-string=postgres://localhost:5432/conduitdb`.

The full example in our case would be:
```shell
./conduit -db.type=postgres -db.postgres.connection-string="postgresql://localhost:5432/conduitdb"
```
2 changes: 1 addition & 1 deletion docs/features/stream-inspector.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: 'Stream Inspector'
sidebar_position: 6
sidebar_position: 8
tag: ['Stream Inspector', 'Connector']
keywords: ['stream', 'inspector']
---
Expand Down
2 changes: 1 addition & 1 deletion docs/features/web-ui.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: 'Web UI'
sidebar_position: 7
sidebar_position: 9
---

Conduit comes with a web UI that allows you to build a data pipeline using a
Expand Down

0 comments on commit 13ededd

Please sign in to comment.