Schema support

[hariso]

Feature description

Adds support for schemas in Conduit. The work is split as in the lists below. It would be good if we start with #1561 and the conduit-commons issues (minus schema auto-generation) so we can parallelize the rest of the work.

Conduit

• [Schemas] In-memory schema service #1561
• [Schemas] Ability to use Confluent's Schema Registry as the backend implementation #1562
• [Schemas] Limit schema service usage to connectors #1563
• [Schemas] Use configured schema service in Avro processors #1653
• [Schemas] Handle name conflicts #1660
• [Schemas] Decode raw data in records with schemas before passing them to processors #1707
• [Schemas] Replace pkg/foundation/database with database package in conduit-commons #1719

conduit-commons

• [Schemas] Add metadata conduit-commons#50
• [Schemas] Utility functions to create Avro schemas conduit-commons#52
• [Schemas] Encode/decode an object using a schema conduit-commons#53
• [Schemas] Implement schema auto-generation conduit-commons#51

Connector SDK

• [Schemas] Create and get a schema from Conduit conduit-connector-sdk#132
• [Schemas] Add source middleware conduit-connector-sdk#137
• [Schemas] Add destination middleware conduit-connector-sdk#138
• [Schemas] Add caching to the schema.Get() function conduit-connector-sdk#158

Processor SDK

• [Schemas] Create and get a schema from Conduit conduit-processor-sdk#45

Schema registry

• [Schemas] Persist schemas conduit-schema-registry#3

[lovromazgon]

A lot, if not all of the code, needed for auto-generating Avro schemas can be found here.

The utilities mentioned in both SDKs can be placed in conduit-commons, so we can re-use it.

[lovromazgon]

Shipped in https://github.com/ConduitIO/conduit/releases/tag/v0.11.0