feat: release 0.11.0 (#109)

* doc: update to 0.11.0 * fix: announcement bar dismissal * document metadata fields * add link for source * combine metadata fields * combine differently * refactor building connector This is so the each function could be anchored and shows up in the menu of the right hand-side. I considered this was more relevant than what was linked before: before: ![](https://d.pr/i/QtqZUO.png) after: ![](https://d.pr/i/3WKPvo.png) * Update docs/features/opencdc-record.mdx Co-authored-by: Maha Hajja <[email protected]> * Update docs/features/opencdc-record.mdx Co-authored-by: Maha Hajja <[email protected]> * Update docs/features/opencdc-record.mdx Co-authored-by: Maha Hajja <[email protected]> * Update docs/features/opencdc-record.mdx Co-authored-by: Maha Hajja <[email protected]> * Include additional options to middleware * update how it works docs with schema host functions (#111) * update how it works docs with schema host functions * Update docs/processors/standalone/how-it-works.mdx Co-authored-by: Lovro Mažgon <[email protected]> * Update docs/processors/standalone/how-it-works.mdx Co-authored-by: Lovro Mažgon <[email protected]> * Update docs/processors/standalone/how-it-works.mdx Co-authored-by: Raúl Barroso <[email protected]> --------- Co-authored-by: Lovro Mažgon <[email protected]> Co-authored-by: Raúl Barroso <[email protected]> * Update processor specs and regenerate docs (#113) * update processor specs and regenerate docs * fix example name * updates architecture * includes schema service on architecture page * updates architecture page * generate processor specs (#115) * adds new schema registry feature page * tweaks * updates * more accurate * address feedback * add schema utility functions to building.mdx (#114) * include github repo * tweaks * fix link * regenerate processor docs with latest specs (#117) * General info about schema support (#116) * Add general info about schema support * connectors and processors * schema reg: include github repo * fix link * update architecture diagram * Update docs/features/opencdc-record.mdx Co-authored-by: Lovro Mažgon <[email protected]> * back to lowercase * fix json encoding md * Update docs/features/schema-support.mdx Co-authored-by: Lovro Mažgon <[email protected]> * Update docs/features/schema-support.mdx Co-authored-by: Lovro Mažgon <[email protected]> * Update docs/features/storage.mdx Co-authored-by: Lovro Mažgon <[email protected]> * Update docs/features/schema-support.mdx Co-authored-by: Lovro Mažgon <[email protected]> * Update docs/processors/standalone/building.mdx Co-authored-by: Lovro Mažgon <[email protected]> * fix indentation and complete snippet * Fix wording and link * more accurate * fix go code * Update docs/connectors/building-connectors/developing-destination-connectors.mdx Co-authored-by: Lovro Mažgon <[email protected]> * Update docs/connectors/building-connectors/developing-source-connectors.mdx Co-authored-by: Lovro Mažgon <[email protected]> * Update docs/processors/standalone/building.mdx Co-authored-by: Lovro Mažgon <[email protected]> * doc: configuration parameters (#119) * fix link * configuration parameters * add batch size * add schema extraction * explain batching * uses output format --------- Co-authored-by: Lovro Mažgon <[email protected]> * Add info about supported data types (#118) * Add info about supported data types * add link * move page * Update docs/features/opencdc-record.mdx * Update docs/features/opencdc-record.mdx * Update docs/features/opencdc-record.mdx --------- Co-authored-by: Lovro Mažgon <[email protected]> * updates * update * fix formatting * update architecture page * Add default values * fix configuration links * add example for schema extraction * explain example * capitalize * Update docs/connectors/configuration-parameters/output-format.mdx Co-authored-by: Maha Hajja <[email protected]> --------- Co-authored-by: Maha Hajja <[email protected]> Co-authored-by: Lovro Mažgon <[email protected]> Co-authored-by: Haris Osmanagić <[email protected]>
ConduitIO · Aug 7, 2024 · 6b51ee3 · 6b51ee3
1 parent 1992b1b
commit 6b51ee3
Show file tree

Hide file tree

Showing 61 changed files with 2,205 additions and 442 deletions.
diff --git a/docs/connectors/_category_.json b/docs/connectors/_category_.json
@@ -1,4 +1,4 @@
 {
   "label": "Connectors",
-  "position": 3
+  "position": 2
 }
diff --git a/docs/connectors/additional-built-in-plugins.mdx b/docs/connectors/additional-built-in-plugins.mdx
@@ -1,6 +1,6 @@
 ---
 title: "Adding built-in Connectors"
-sidebar_position: 2
+sidebar_position: 4
 ---
 
 Built-in connectors offer better performance when compared to standalone ones,

diff --git a/docs/connectors/behavior.mdx b/docs/connectors/behavior.mdx
@@ -1,6 +1,6 @@
 ---
 title: "Connector Behavior"
-sidebar_position: 4
+sidebar_position: 5
 ---
 
 This document provides insights on how Conduit communicates with a connector.

diff --git a/docs/connectors/building-connectors/_category_.json b/docs/connectors/building-connectors/_category_.json
@@ -1,5 +1,5 @@
 {
   "label": "Building Connectors",
-  "position": 5,
+  "position": 4,
   "collapsed": true
 }
diff --git a/docs/connectors/building-connectors/conduit-sdk.mdx b/docs/connectors/building-connectors/conduit-sdk.mdx
@@ -6,35 +6,14 @@ sidebar_position: 0
 Conduit connectors can be built in any programming language that supports gRPC.
 To make it easier to write connectors we provide
 a [Connector SDK](https://github.com/ConduitIO/conduit-connector-sdk) written in
-Go. Using the SDK is the recommended way of writing a Conduit connector.
-
-## Supported data types
-
-There are no limitations when it comes to data types a source connector can read
-from a source. However, if a standalone source connector uses `record.StructuredData`
-in its key or any part of the payload, then there are certain limitations in the
-data types it can send to Conduit.
-
-The following data types are supported:
-
-* `bool`
-* `int`, `int32`, `int64`, `uint`, `uint32`, `uint64`
-* `float32`, `float64`
-* `string`
-* `[]byte` (stored as a string, base64-encoded)
-* `map[string]interface{}` (a map of strings to any of the values that are supported)
-* `[]interface{}` (a slice of any value that is supported)
-
-A notable limitation is timestamps, i.e. `time.Time` values are not supported.
-
-One way to support other values is to encode source data to a `[]byte` (e.g. using
-a JSON encoding) and then store the value as `record.RawData`.
-
+Go. Using the SDK is the recommended way of writing a Conduit connector. If you
+are implementing a connector in a different language you should refer to
+the [Connector Protocol](https://github.com/ConduitIO/conduit-connector-protocol).
 
 ## Conduit connector template
 
 The easiest way to start implementing your own Conduit connector is by using the
-[Conduit connector template](https://github.com/ConduitIO/conduit-connector-template).
+[Conduit connector template](/docs/connectors/building-connectors/conduit-connector-template).
 It contains the basic project structure as well as some additional utilities
 like GitHub actions and a Makefile.
 

diff --git a/docs/connectors/building-connectors/developing-destination-connectors.mdx b/docs/connectors/building-connectors/developing-destination-connectors.mdx
@@ -5,13 +5,11 @@ sidebar_position: 4
 
 A Destination is responsible for writing [Record](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#Record) to third party systems.
 
-You need to implement the functions required by Destination and provide your own implementations. Information about individual functions are listed below.
+You need to implement the functions required by Destination and provide your own implementations. Information about individual functions are listed below. The **`destination.go`** file is the main file where the functionality of your Destination Connector is implemented.
 
-## `destination.go`
+## `Destination struct`
 
-This file provides the main functionality of your Destination Connector.
-
-- **`Destination`** Struct: Every Destination implementation needs to include an [UnimplementedDestination](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#UnimplementedDestination) to satisfy the interface. This allows us to potentially change the interface in the future while remaining backward compatible with existing Destination implementations.  This struct can be modified to add additional fields that can be accessed throughout the lifecycle of the Connector.
+Every Destination implementation needs to include an [UnimplementedDestination](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#UnimplementedDestination) to satisfy the interface. This allows us to potentially change the interface in the future while remaining backward compatible with existing Destination implementations.  This struct can be modified to add additional fields that can be accessed throughout the lifecycle of the Connector.
 
     ```go
     type Destination struct {
@@ -21,24 +19,39 @@ This file provides the main functionality of your Destination Connector.
     }
     ```
 
-- **`NewDestination()`**: A constructor function for your Destination struct. Note that this is the same function that should be set as the value of `Connector.NewDestination`. The constructor should be used to wrap your Destination in the default middleware. You can add additional middleware, but unless you have a very good reason, you should always include the default middleware.
+## Destination Connector Lifecycle Functions
 
-    ```go
-    func NewDestination() sdk.Destination {
-        // Create Destination and wrap it in the default middleware.
-        return sdk.DestinationWithMiddleware(&Destination{}, sdk.DefaultDestinationMiddleware()...)
-    }
-    ```
+### `NewDestination()`
+
+A constructor function for your Destination struct. Note that this is the same function that should be set as the value of `Connector.NewDestination`. The constructor should be used to wrap your Destination in the default middleware. You can add additional middleware, but unless you have a very good reason, you should always include the default middleware.
+
+```go
+func NewDestination() sdk.Destination {
+    // Create Destination and wrap it in the default middleware.
+    return sdk.DestinationWithMiddleware(
+      &Destination{}, 
+      sdk.DefaultDestinationMiddleware()...,
+    )
+}
+```
+
+**Additional options via `DestinationMiddlewareOption`**:
 
-- **`Parameters()`**: A map of named Parameters that describe how to configure the connector. This map is typically generated using [`paramgen`](https://github.com/ConduitIO/conduit-commons/tree/main/paramgen).
+Currently, the available destination middleware options can be found [here](https://github.com/ConduitIO/conduit-connector-sdk/blob/1cbe778fabc8f903e075872560e6a91049d2e978/destination_middleware.go#L44-L50).
+
+### `Parameters()`
+
+A map of named Parameters that describe how to configure the connector. This map is typically generated using [`paramgen`](https://github.com/ConduitIO/conduit-commons/tree/main/paramgen).
 
     ```go
     func (d *Destination) Parameters() map[string]sdk.Parameter {
         return d.config.Parameters()
     }
     ```
 
-- **`Configure()`**: Validates and stores configuration data for the connector. Any complex validation logic should be implemented here.
+### `Configure()`
+
+Validates and stores configuration data for the connector. Any complex validation logic should be implemented here.
 
     ```go
     func (d *Destination) Configure(ctx context.Context, cfg map[string]string) error {
@@ -51,7 +64,9 @@ This file provides the main functionality of your Destination Connector.
     }
     ```
 
-- **`Open()`**: Prepares the connector to start producing records based on the last known successful position. If needed, the connector should open connections in this function.
+### `Open()`
+
+Prepares the connector to start producing records based on the last known successful position. If needed, the connector should open connections in this function.
 
     ```go
     func (d *Destination) Open(ctx context.Context) error {
@@ -75,7 +90,9 @@ This file provides the main functionality of your Destination Connector.
     }
     ```
 
-- **`Write()`**: Writes len(records) from a slice of `sdk.Record` objects received from the Conduit pipeline to the destination right away without caching.  It should return the number of records written from the slice and any error encountered that caused the write to stop early.
+### `Write()`
+
+Writes len(records) from a slice of `sdk.Record` objects received from the Conduit pipeline to the destination right away without caching.  It should return the number of records written from the slice and any error encountered that caused the write to stop early.
 
     ```go
     func (d *Destination) Write(ctx context.Context, recs []sdk.Record) (int, error) {
@@ -97,7 +114,9 @@ This file provides the main functionality of your Destination Connector.
     }
     ```
 
-- **`Ack()`**: Ack signals to the implementation that the record with the supplied position was successfully processed.
+### `Ack()`
+
+Ack signals to the implementation that the record with the supplied position was successfully processed.
 
     ```go
     func (d *Destination) Ack(ctx context.Context, position sdk.Position) error {
@@ -106,7 +125,9 @@ This file provides the main functionality of your Destination Connector.
     }
     ```
 
-- **`Teardown()`**: Teardown signals to the connector that there will be no more calls to any other function. Any connections that were created in the `Open()` function should be closed here.
+### `Teardown()`
+
+Teardown signals to the connector that there will be no more calls to any other function. Any connections that were created in the `Open()` function should be closed here.
 
     ```go
     func (d *Destination) Teardown(ctx context.Context) error {

diff --git a/docs/connectors/building-connectors/developing-source-connectors.mdx b/docs/connectors/building-connectors/developing-source-connectors.mdx
@@ -5,15 +5,11 @@ sidebar_position: 3
 
 A Source is responsible for continuously reading data from a third party system and returning it in the form of an [SDK Record](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#Record).
 
-You need to implement the functions required by the Source interface and provide your own implementations. Information about individual functions are listed below.
+You need to implement the functions required by the Source interface and provide your own implementations. Information about individual functions are listed below. The **`source.go`** file is the main file where the functionality of your Source Connector is implemented.
 
-## `source.go`
+## `Source struct`
 
-This file provides the main functionality of your Source Connector.
-
-### Source Connector Lifecycle Functions
-
-- **`Source`** Struct: Every Source implementation needs to include an [UnimplementedSource](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#UnimplementedSource) to satisfy the interface. This allows us to potentially change the interface in the future while remaining backward compatible with existing Source implementations. This struct can be modified to add additional fields that can be accessed throughout the lifecycle of the Connector
+Every Source implementation needs to include an [UnimplementedSource](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#UnimplementedSource) to satisfy the interface. This allows us to potentially change the interface in the future while remaining backward compatible with existing Source implementations. This struct can be modified to add additional fields that can be accessed throughout the lifecycle of the Connector
 
     ```go
     type Source struct {
@@ -27,38 +23,72 @@ This file provides the main functionality of your Source Connector.
     }
     ```
 
-- **`NewSource()`**: A constructor function for your Source struct. Note that this is the same function that should be set as the value of `Connector.NewSource`. The constructor should be used to wrap your Source in the default middleware. You can add additional middleware, but unless you have a very good reason, you should always include the default middleware.
+## Source Connector Lifecycle Functions
 
-    ```go
-    func NewSource() sdk.Source {
-        // Create Source and wrap it in the default middleware.
-        return sdk.SourceWithMiddleware(&Source{}, sdk.DefaultSourceMiddleware()...)
-    }
-    ```
+### `NewSource()`
 
-- **`Parameters()`**: A map of named Parameters that describe how to configure the connector. This map is typically generates using [`paramgen`](https://github.com/ConduitIO/conduit-commons/tree/main/paramgen).
+A constructor function for your Source struct. Note that this is the same function that should be set as the value of `Connector.NewSource`. The constructor should be used to wrap your Source in the default `DefaultSourceMiddleware`. 
 
-    ```go
-    func (s *Source) Parameters() map[string]sdk.Parameter {
-        return s.config.Parameters()
-    }
-    ```
+```go
+func NewSource() sdk.Source {
+    // Create Source and wrap it in the default middleware.
+    return sdk.SourceWithMiddleware(
+      &Source{}, 
+      sdk.DefaultSourceMiddleware()...,
+    )
+}
+```
 
-- **`Configure()`**: Validates and stores configuration data for the connector. Any complex validation logic should be implemented here.
+**Additional options via `SourceMiddlewareOption`**:
 
-    ```go
-    func (s *Source) Configure(ctx context.Context, cfg map[string]string) error {
-    sdk.Logger(ctx).Info().Msg("Configuring Source...")
-    err := sdk.Util.ParseConfig(cfg, &s.config)
-    if err != nil {
-        return fmt.Errorf("invalid config: %w", err)
-    }
-    // add custom validations here
-    return nil
-    }
-    ```
+In case you need to add additional middleware options, you can do so by passing it to the `sdk.SourceWithMiddleware` function via `sdk.DefaultSourceMiddleware(opts ...SourceMiddlewareOption)`. Currently, the available source middleware options can be cound [here](https://github.com/ConduitIO/conduit-connector-sdk/blob/1cbe778fabc8f903e075872560e6a91049d2e978/source_middleware.go#L42-L46).
 
-- **`Open()`**: Prepares the connector to start producing records based on the last known successful position. If needed, the connector should open connections in this function.
+:::note
+If you're using a source connector that's not generating structured data (i.e. produces raw data), you might want to disable schema extraction by default by overwriting the `sdk.SourceWithSchemaExtractionConfig` options:
+
+```go
+sdk.SourceWithMiddleware(
+		&Source{},
+		sdk.DefaultSourceMiddleware(
+			// disable schema extraction by default, because the source produces raw data
+			sdk.SourceWithSchemaExtractionConfig{
+				PayloadEnabled: lang.Ptr(false),
+				KeyEnabled:     lang.Ptr(false),
+			},
+		)...,
+	)
+```
+:::
+
+### `Parameters()`
+
+A map of named Parameters that describe how to configure the connector. This map is typically generates using [`paramgen`](https://github.com/ConduitIO/conduit-commons/tree/main/paramgen).
+
+```go
+func (s *Source) Parameters() map[string]sdk.Parameter {
+    return s.config.Parameters()
+}
+```
+
+### `Configure()` 
+
+Validates and stores configuration data for the connector. Any complex validation logic should be implemented here.
+
+  ```go
+  func (s *Source) Configure(ctx context.Context, cfg map[string]string) error {
+  sdk.Logger(ctx).Info().Msg("Configuring Source...")
+  err := sdk.Util.ParseConfig(cfg, &s.config)
+  if err != nil {
+      return fmt.Errorf("invalid config: %w", err)
+  }
+  // add custom validations here
+  return nil
+  }
+  ```
+
+### `Open()`
+
+Prepares the connector to start producing records based on the last known successful position. If needed, the connector should open connections in this function.
 
     ```go
     func (s *Source) Open(ctx context.Context, pos sdk.Position) error {
@@ -83,7 +113,9 @@ This file provides the main functionality of your Source Connector.
     }
     ```
 
-- **`Read()`**: Gathers data from the configured data source and formats it into a `sdk.Record` that is returned from the function. The returned `sdk.Record` is queued into the pipeline to be consumed by a Destination connector.
+### `Read()`
+
+Gathers data from the configured data source and formats it into a `sdk.Record` that is returned from the function. The returned `sdk.Record` is queued into the pipeline to be consumed by a Destination connector.
 
     ```go
     func (s *Source) Read(ctx context.Context) (sdk.Record, error) {
@@ -136,7 +168,9 @@ This file provides the main functionality of your Source Connector.
     }
     ```
 
-- **`Ack()`**: Ack signals to the implementation that the record with the supplied position was successfully processed.
+### `Ack()`
+
+Ack signals to the implementation that the record with the supplied position was successfully processed.
 
     ```go
     func (s *Source) Ack(ctx context.Context, position sdk.Position) error {
@@ -145,7 +179,9 @@ This file provides the main functionality of your Source Connector.
     }
     ```
 
-- **`Teardown()`**: Teardown signals to the connector that there will be no more calls to any other function. Any connections that were created in the `Open()` function should be closed here.
+### `Teardown()`
+
+Teardown signals to the connector that there will be no more calls to any other function. Any connections that were created in the `Open()` function should be closed here.
 
     ```go
     func (s *Source) Teardown(ctx context.Context) error {

diff --git a/docs/connectors/configuration-parameters/_category_.json b/docs/connectors/configuration-parameters/_category_.json
@@ -0,0 +1,4 @@
+{
+  "label": "Configuration parameters",
+  "position": 3
+}