Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor + cpluginv2 #90

Merged
merged 14 commits into from
May 22, 2024
Merged

Refactor + cpluginv2 #90

merged 14 commits into from
May 22, 2024

Conversation

lovromazgon
Copy link
Member

@lovromazgon lovromazgon commented May 21, 2024

Description

This PR moves package cpluginv1 to cplugin/v1 and deprecates it. Instead, it introduces cplugin/v2. The functionality stays the same, however:

  • v2 is using config/v1/parameter for the parameter types
  • v2 adds support for batching records in the source and destination stream

The protocol provides packages cplugin/client (which was previously placed in the Conduit repo) and cplugin/server. Both ensure backwards compatibility for connectors, so new connectors should work with older Conduit instances and old connectors should work with newer Conduit instances.

Having the package cplugin/client in the protocol opens a way for us to write other tools that can run connector plugins (for instance a standalone acceptance test tool or connector validation utility).

I also added tests for the non-trivial from/to proto transformations. The tests make it clear we need to add schema support, as we lose types in structured data during the transformation.

Related to ConduitIO/conduit#1433.

Quick checks:

  • I have followed the Code Guidelines.
  • There is no other pull request for the same update/change.
  • I have written unit tests.
  • I have made sure that the PR is of reasonable size and can be easily reviewed.

@lovromazgon lovromazgon requested a review from a team as a code owner May 21, 2024 14:30
@lovromazgon lovromazgon mentioned this pull request May 21, 2024
4 tasks
@lovromazgon lovromazgon marked this pull request as draft May 21, 2024 14:31
@lovromazgon lovromazgon marked this pull request as ready for review May 21, 2024 16:39
proto/connector/v2/connector.proto Show resolved Hide resolved
proto/connector/v2/connector.proto Show resolved Hide resolved
cplugin/source.go Show resolved Hide resolved
cplugin/server/serve.go Show resolved Hide resolved
cplugin/client/client.go Show resolved Hide resolved
Copy link
Contributor

@hariso hariso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, Lovro! 🚀 Left a couple of non-blocker comment only.:)

cplugin/client/client.go Outdated Show resolved Hide resolved
cplugin/v2/fromproto/source_test.go Outdated Show resolved Hide resolved
Copy link
Contributor

@hariso hariso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💥

@lovromazgon lovromazgon merged commit 94685ac into main May 22, 2024
5 checks passed
@lovromazgon lovromazgon deleted the lovro/cpluginv2-refactor branch May 22, 2024 16:03
hariso added a commit that referenced this pull request May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants