Configurable output format parameters

[lovromazgon]

We want to make sure the user can configure what the destination connector produces (relevant issue: #531). Since we introduced SDK middleware this is the best place to add this so all destination connectors can benefit from it.

I already started working on this (relevant PR: ConduitIO/conduit-connector-sdk#48) but have a hard time deciding on the config parameters and their names. Here's what I want to achieve with these parameters:

• Configure if the destination outputs an OpenCDC record or Kafka Connect record
  • Configure options like filtering only a specific field (i.e. output only payload, or payload and key)
• Configure if the destination outputs JSON, YAML, Avro, Parquet,...
  • Configure options like discarding nil fields

The first parameter chooses the internal structure, the second chooses the encoding into bytes.

I identified a couple of options (currently the PR uses option 1 option 4):

Option 1 - type and options separate

sdk.record.converter.type: opencdc
sdk.record.converter.options: exclusion=metadata
sdk.record.encoder.type: json
sdk.record.encoder.options: omitempty=true

Option 2 - options nested

sdk.record.type: opencdc
sdk.record.type.options: exclusion=metadata  # exclude metadata field
sdk.record.encoding: json
sdk.record.encoding.options: omitempty=true  # don't marshal empty fields

Option 3 - type and options together in one parameter

In this example options are separated from the value itself with a space.

sdk.record.type: opencdc exclusion=metadata
sdk.record.encoding: json omitempty=true

Option 4 - both types together (MIME type)

Here we essentially use a MIME type with the structure like type/subtype.

sdk.record.format: opencdc/json
sdk.record.format.options: exclusion=metadata,omitempty=true

---

I'm interested in your opinions and ideas!

[maha-hajja]

I personally prefer options 2 and 4, option 2 for the consistency between the names, and option 4 for the simplicity.

[lovromazgon]

I'm also leaning towards option 4, I like that there's only one options field, otherwise it might get confusing for the user.

[lovromazgon]

I'm just thinking if we should change the first parameter to sdk.record.format.type, we have something similar in the generator connector and we're using that approach there (link).

So essentially it could look like this:

sdk.record.format.type: opencdc/json
sdk.record.format.options: exclusion=metadata,omitempty=true

[samirketema]

@lovromazgon @maha-hajja sounds great, reasonable and intuitive to me

[lovromazgon]

This has been implemented and shipped in Conduit 0.6 🎉 Read more about the feature: https://conduit.io/docs/connectors/output-formats