Use connector protocol v2 #1622

lovromazgon · 2024-05-28T12:54:39Z

Description

This PR includes code changes to use opencdc.Record everywhere. It removes the old record package entirely. It also updates the connector protocol and SDK to the latest version, so we use the same record across the board. The code is updated according to the changes that were already done in the protocol.

Most of the code changes are straightforward, it's just that there are a lot. I posted a few comments below which call out the important bits, I strongly recommend reading them through before reviewing this PR.

The next PR will remove a bunch of code in pkg/foundation and use the packages in conduit-commons instead.

Closes #1433.

Quick checks:

I have followed the Code Guidelines.
There is no other pull request for the same update/change.
I have written unit tests.
I have made sure that the PR is of reasonable size and can be easily reviewed.

lovromazgon

Here's some pointers to make the review easier.

pkg/plugin/connector/mock/destination.go

pkg/plugin/connector/builtin/stream.go

pkg/plugin/connector/builtin/source.go

pkg/plugin/connector/acceptance_testing.go

pkg/pipeline/lifecycle_test.go

pkg/connector/destination.go

pkg/pipeline/stream/destination.go

pkg/pipeline/stream/base.go

pkg/plugin/connector/builtin/registry.go

pkg/foundation/database/database.go

pkg/connector/destination.go

pkg/connector/source.go

pkg/pipeline/stream/destination_acker.go

pkg/plugin/connector/builtin/destination.go

raulb

@lovromazgon Looks great. Only left some nits

pkg/connector/destination.go

pkg/plugin/connector/builtin/destination.go

pkg/connector/destination.go

pkg/pipeline/dlq.go

pkg/plugin/connector/builtin/stream.go

* remove obsolete foundation packages * replace removed packages with alternatives in conduit-commons * user errors.Join instead of multierror * go mod tidy

lovromazgon added 18 commits May 28, 2024 14:50

update standalone plugin to use connector protocol v2

07f8215

update destination acceptance tests

473407f

update builtin plugin adapter

6896424

move builtin/v1 to builtin

3fa19f6

update connector package, still missing source tests

23a5fea

source tests

5bed7c2

update pkg/pipeline

1ed5f7e

fix pkg/orchestrator

a3cd351

conduit compiles again

9ee5c94

generate typed mocks

deadf56

update protocol and sdk, rename cplugin to pconnector

ead8878

fix linter warning

0e2ea93

Merge branch 'main' into lovro/cpluginv2

3ef82e1

test getting stream without calling run

a3c83ce

make destination plugin mock nicer

2ffbec2

remove redundant mock options

3fe54db

organize imports

60d27d7

clone response in builtin plugins

457de2d

lovromazgon commented Jun 20, 2024

View reviewed changes

pkg/foundation/database/database.go Show resolved Hide resolved

lovromazgon marked this pull request as ready for review June 20, 2024 12:24

lovromazgon requested a review from a team as a code owner June 20, 2024 12:24

lovromazgon added 4 commits June 20, 2024 14:24

Merge branch 'main' into lovro/cpluginv2

76969a7

add builtin generator connector

52c6ac4

include builtin connectors (only postgres missing now)

5fedd71

add comment about cloning requests and responses

b708694

lovromazgon mentioned this pull request Jun 21, 2024

Migrate to Conduit Commons utils #1673

Merged

4 tasks