Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Customize Conduit's built-in connectors #105

Merged
merged 8 commits into from
Jul 31, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions docs/connectors/additional-built-in-plugins.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
---
title: "Adding built-in Connectors"
sidebar_position: 2
---

Built-in connectors offer better performance when compared to standalone ones,
which is why in some cases it's desirable to have a custom build of Conduit that
includes additional built-in connectors.

The simplest way to achieve so is to write a small application that embeds
Conduit (i.e. uses Conduit as a library) and adds one or more connectors to its
default configuration.

In the example below we will add
the [HTTP connector](https://github.com/conduitio-labs/conduit-connector-http)
to Conduit as a built-in connector.

First, we initialize a Go module with `go mod init github.com/conduitio-labs/custom-conduit`.

Then , we need to add Conduit and the HTTP connector as dependencies:
```shell
go get github.com/conduitio/conduit
go get github.com/conduitio-labs/conduit-connector-http
go mod tidy
```

Once that is done, we need to write a `main` function that:
1. Adds the HTTP connector the default Conduit configuration
2. Runs Conduit with the custom configuration.

That's done in the code below:

```go
package main

import (
http "github.com/conduitio-labs/conduit-connector-http"
"github.com/conduitio/conduit/pkg/conduit"
)

func main() {
// Get the default configuration, including all built-in connectors
cfg := conduit.DefaultConfig()

// Add the HTTP connector to list of built-in connectors
cfg.ConnectorPlugins["http"] = http.Connector

conduit.Serve(cfg)
}
```

This custom version of Conduit can be built with `go build main.go -o custom-conduit`. If
you run the built binary, you can check that the HTTP connector has been
included in the build by listing all the connector plugins:
```shell
curl 'http://localhost:8080/v1/connectors/plugins'

[
{
"name": "builtin:http@(devel)",
"summary": "HTTP source and destination connectors for Conduit.",
"description": "Conduit HTTP source and destination connectors, they connect to an HTTP URL and send HTTP requests.",
"version": "(devel)",
"author": "",
"destinationParams": {},
"sourceParams": {}
}
// other plugins
]
```
2 changes: 1 addition & 1 deletion docs/connectors/connector-lifecycle.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Connector Lifecycle"
sidebar_position: 7
sidebar_position: 6
---

:::info
Expand Down
2 changes: 1 addition & 1 deletion docs/connectors/kafka-connect-connector.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Kafka Connect Connectors with Conduit"
sidebar_position: 8
sidebar_position: 9
---

# Using Kafka Connect Connectors with Conduit
Expand Down
2 changes: 1 addition & 1 deletion docs/connectors/output-formats.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Output Formats"
sidebar_position: 6
sidebar_position: 5
---

One of the challenges to be solved when integrating Conduit with other systems, such as Kafka Connect and Debezium, is
Expand Down