A single base for all python connectors

[alafanechere]

Tech spec

[alafanechere]

Weekly project update: A single base image for all python connectors

What was done:

• Tech spec is ratified. We picked the following final direction:
  • Our base images will be declared in code.
  • Certified connectors won’t have Dockerfile anymore and will use this base image
  • A system of pre/post build hooks will allow customization of connector build.
  • Community connectors can still have Dockerfiles.
• We ran a thorough ad hoc analysis of our existing certified connectors Dockerfile to assess what should be bundled in the base image and how the connectors should be built.
• We added a new connectorBuildOptions.baseImage metadata field. It’ll be used by our build process to define what base image a connector will use.

What’s we want to ship this week:

• Implement a framework that will allow us to manage and version base images. And declare the first base image version.
• Refactor our connector build process in airbyte-ci/connectors/pipeline to use the first base image version.
• Make two connectors use the base image.

What’s next:
Feel free to take a look at the issues linked to this epic.

[alafanechere]

Weekly project update: A base image for all python connectors

What was done:

• We added a connectorBuildOptions.baseImage metadata to make connectors declare their base image.
• We declared a new python package that will manage the declaration, versioning and release of our connector base images.
• We refactored our python connector build process to use the base image declared in their metadata.
• We implemented a pre/post build hook logic for allowing connectors to customize their build.
• We made python connectors unit/integration tests run within their container, and not into an artificial one we provisioned before.
• We made airbyte-ci support building and testing connector with a local CDK version
• We introduced new airbyte-ci commands to help with the migration of connectors to the use of the base image.
• We added a CI check to make tests fail on certified connectors if they’re not using the base image.

What we want to ship this week:

• A good portion of the work mentioned above is still on stacked PR. We’d love it to hit master before the end of the week. Reviews and additional testing required.
• Start migrating certified python connectors to make them use our first base image and delete their dockerfiles.

What got in the way

Healthy but lengthy design discussions on this PR slowed down the delivery. I think it’s for the best: solid bases will prevent a growing tech debt on this project.