[airbyte-ci] test connectors inside their built container #30474

alafanechere · 2023-09-15T13:49:16Z

What

Our original implementation of python connector testing was the following:

Create a python container
Mount the connector code to it
Install some testing dependencies
Run pytest

The problem with this approach is that we're running connector tests inside their real environment: their docker image.
So far our test environment had to keep up with the system dependencies of the connectors.
These test environment could also install python dependencies that are not declare in the original connector python package.

How

Remove the ConnectorPackageInstall step
Make the built connector container the input of the test step
Install dev/test dependencies on top of the original built image
Mount to the container the whole connector directory, to a /test_environment directory (the connector directory is not fully available on the built image).

The state of this "prepared" container is not kept for other steps like CAT or publish. It's just used in the python Unit / integration test step (subclasses of the PytestStep).

alafanechere · 2023-09-15T13:49:35Z

Current dependencies on/for this PR:

master
- PR [airbyte-ci] build python connectors from base images #30456

This comment was auto-generated by Graphite.

airbyte-oss-build-runner · 2023-09-21T11:21:00Z

destination-duckdb test report (commit `e19c6498ab`) - ❌

⏲️ Total pipeline duration: 01mn27s

Step	Result
Build destination-duckdb docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	❌
Validate airbyte-integrations/connectors/destination-duckdb/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-duckdb test

airbyte-oss-build-runner · 2023-09-21T11:21:01Z

source-faker test report (commit `e19c6498ab`) - ✅

⏲️ Total pipeline duration: 01mn19s

Step	Result
Build source-faker docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅
Code format checks	✅
Validate airbyte-integrations/connectors/source-faker/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-faker test

alafanechere · 2023-09-21T11:22:44Z

The test results above show that we did not break the test run:

duckdb poetry tests are correctly executed
faker tests still pass

erohmensing

Not much to say, but I have 👀 this PR

erohmensing · 2023-09-21T14:36:14Z

airbyte-ci/connectors/pipelines/README.md

@@ -404,6 +404,7 @@ This command runs the Python tests for a airbyte-ci poetry package.
 ## Changelog
 | Version | PR                                                        | Description                                                                                               |
 |---------| --------------------------------------------------------- |-----------------------------------------------------------------------------------------------------------|
+| 1.6.0   | [#30474](https://github.com/airbytehq/airbyte/pull/30474) | Test connector insider their containers.                                                                    |


airbyte-ci/connectors/pipelines/pipelines/actions/environments.py

…30474)

alafanechere mentioned this pull request Sep 15, 2023

source-file/faker: make them use our base image + delete dockerfile #30468

Closed

alafanechere force-pushed the 09-14-change_our_python_connector_build_process_to_use_the_base_images branch from 41ce5f5 to b387477 Compare September 15, 2023 15:08

alafanechere force-pushed the augustin/09-15-_airbyte-ci_test_connector_inside_their_built_container branch from f9bea4b to b608d76 Compare September 15, 2023 15:08

alafanechere marked this pull request as ready for review September 15, 2023 15:33

alafanechere requested a review from a team September 15, 2023 15:33

This was referenced Sep 15, 2023

Airbyte-CI: Break up environemnts #30462

Closed

[airbyte-ci] introduce an autocommit command to update base image version to the latest one #30508

Closed

alafanechere force-pushed the 09-14-change_our_python_connector_build_process_to_use_the_base_images branch from b387477 to 0113f6a Compare September 17, 2023 15:01

alafanechere force-pushed the augustin/09-15-_airbyte-ci_test_connector_inside_their_built_container branch from b608d76 to 10448cc Compare September 17, 2023 15:16

alafanechere mentioned this pull request Sep 18, 2023

[airbyte-ci] Implement pre/post build hooks #30513

Closed

alafanechere force-pushed the 09-14-change_our_python_connector_build_process_to_use_the_base_images branch from 157ca30 to 894e0b9 Compare September 18, 2023 10:39

alafanechere force-pushed the augustin/09-15-_airbyte-ci_test_connector_inside_their_built_container branch from 10448cc to 72b1de6 Compare September 18, 2023 10:40

alafanechere changed the base branch from 09-14-change_our_python_connector_build_process_to_use_the_base_images to augustin/09-18-_airbyte-ci_Implement_pre/post_build_hooks September 18, 2023 10:45

alafanechere force-pushed the augustin/09-15-_airbyte-ci_test_connector_inside_their_built_container branch from 72b1de6 to 3fffee5 Compare September 18, 2023 10:45

alafanechere mentioned this pull request Sep 18, 2023

[airbyte-ci] new commands for migration to base image #30520

Merged

alafanechere force-pushed the augustin/09-18-_airbyte-ci_Implement_pre/post_build_hooks branch from db634ba to 8e3804a Compare September 18, 2023 14:16

alafanechere force-pushed the augustin/09-15-_airbyte-ci_test_connector_inside_their_built_container branch from 3fffee5 to 0a60854 Compare September 18, 2023 14:16

alafanechere mentioned this pull request Sep 18, 2023

[airbyte-ci] implement pre/post build hooks #30526

Merged

alafanechere force-pushed the augustin/09-18-_airbyte-ci_Implement_pre/post_build_hooks branch from 8e3804a to 82971ce Compare September 18, 2023 14:46

alafanechere force-pushed the augustin/09-15-_airbyte-ci_test_connector_inside_their_built_container branch from 0a60854 to 94f6f82 Compare September 18, 2023 14:46

This was referenced Sep 18, 2023

[airbyte-ci]: new check on python certified connector to validate their base image use #30527

Merged

Make certified python connectors use our base image #30528

Closed

alafanechere force-pushed the augustin/09-15-_airbyte-ci_test_connector_inside_their_built_container branch from 94f6f82 to 0a1d09c Compare September 18, 2023 16:01

alafanechere force-pushed the 09-14-change_our_python_connector_build_process_to_use_the_base_images branch from 7086a9d to 7cc99f4 Compare September 20, 2023 21:10

alafanechere force-pushed the augustin/09-15-_airbyte-ci_test_connector_inside_their_built_container branch 3 times, most recently from eaf9fbc to d6d0a87 Compare September 21, 2023 11:05

octavia-squidington-iii added area/connectors Connector related issues connectors/destination/duckdb connectors/source/faker labels Sep 21, 2023

octavia-squidington-iii removed the area/connectors Connector related issues label Sep 21, 2023

alafanechere force-pushed the augustin/09-15-_airbyte-ci_test_connector_inside_their_built_container branch from db45b8e to 28a886b Compare September 21, 2023 11:38

erohmensing reviewed Sep 21, 2023

View reviewed changes

bnchrch force-pushed the 09-14-change_our_python_connector_build_process_to_use_the_base_images branch from 96068a5 to 18693fc Compare September 28, 2023 15:16

alafanechere mentioned this pull request Oct 9, 2023

🐛 source-file: fix integration tests #31152

Closed

Base automatically changed from 09-14-change_our_python_connector_build_process_to_use_the_base_images to master October 10, 2023 17:16

alafanechere force-pushed the augustin/09-15-_airbyte-ci_test_connector_inside_their_built_container branch 2 times, most recently from b14764c to 02361d2 Compare October 11, 2023 09:42

[airbyte-ci] test connector inside their built container

310ec11

alafanechere force-pushed the augustin/09-15-_airbyte-ci_test_connector_inside_their_built_container branch from 02361d2 to 310ec11 Compare October 11, 2023 11:55

alafanechere changed the title ~~[airbyte-ci] test connector inside their built container~~ [airbyte-ci] test connectors inside their built container Oct 11, 2023

alafanechere enabled auto-merge (squash) October 11, 2023 12:03

alafanechere disabled auto-merge October 11, 2023 12:03

alafanechere enabled auto-merge (squash) October 11, 2023 12:06

alafanechere merged commit 9fae594 into master Oct 11, 2023
14 checks passed

alafanechere deleted the augustin/09-15-_airbyte-ci_test_connector_inside_their_built_container branch October 11, 2023 12:06

alafanechere mentioned this pull request Oct 12, 2023

airbyte-ci: bind integration tests to dockerhost (again) #31343

Merged

ariesgun pushed a commit to ariesgun/airbyte that referenced this pull request Oct 23, 2023

[airbyte-ci] test connectors inside their built container (airbytehq#…

d59ba0a

…30474)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[airbyte-ci] test connectors inside their built container #30474

[airbyte-ci] test connectors inside their built container #30474

alafanechere commented Sep 15, 2023 •

edited

Loading

alafanechere commented Sep 15, 2023 •

edited by bnchrch

Loading

airbyte-oss-build-runner commented Sep 21, 2023

airbyte-oss-build-runner commented Sep 21, 2023

alafanechere commented Sep 21, 2023

erohmensing left a comment

erohmensing Sep 21, 2023

	\| 1.6.0 \| [#30474](https://github.com/airbytehq/airbyte/pull/30474) \| Test connector insider their containers. \|
	\| 1.6.0 \| [#30474](https://github.com/airbytehq/airbyte/pull/30474) \| Test connectors inside their containers. \|

[airbyte-ci] test connectors inside their built container #30474

[airbyte-ci] test connectors inside their built container #30474

Conversation

alafanechere commented Sep 15, 2023 • edited Loading

What

How

alafanechere commented Sep 15, 2023 • edited by bnchrch Loading

airbyte-oss-build-runner commented Sep 21, 2023

destination-duckdb test report (commit e19c6498ab) - ❌

airbyte-oss-build-runner commented Sep 21, 2023

source-faker test report (commit e19c6498ab) - ✅

alafanechere commented Sep 21, 2023

erohmensing left a comment

Choose a reason for hiding this comment

erohmensing Sep 21, 2023

Choose a reason for hiding this comment

alafanechere commented Sep 15, 2023 •

edited

Loading

alafanechere commented Sep 15, 2023 •

edited by bnchrch

Loading

destination-duckdb test report (commit `e19c6498ab`) - ❌

source-faker test report (commit `e19c6498ab`) - ✅