Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

airbyte-ci: concurrent java tests #31426

Merged
merged 4 commits into from
Oct 16, 2023
Merged

airbyte-ci: concurrent java tests #31426

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
728a28e
airbyte-ci: concurrent java tests
alafanechere Oct 15, 2023
25c2a1e
bump version
alafanechere Oct 16, 2023
685872d
increased concurrency
alafanechere Oct 16, 2023
e107b13
increased concurrency
alafanechere Oct 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions airbyte-ci/connectors/pipelines/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -425,6 +425,7 @@ This command runs the Python tests for a airbyte-ci poetry package.
## Changelog
| Version | PR | Description |
| ------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| 1.9.2 | [#31426](https://github.com/airbytehq/airbyte/pull/31426) | Concurrent execution of java connectors tests. |
| 1.9.1 | [#31455](https://github.com/airbytehq/airbyte/pull/31455) | Fix `None` docker credentials on publish. |
| 1.9.0 | [#30520](https://github.com/airbytehq/airbyte/pull/30520) | New commands: `bump-version`, `upgrade-base-image`, `migrate-to-base-image`. |
| 1.8.0 | [#30520](https://github.com/airbytehq/airbyte/pull/30520) | New commands: `bump-version`, `upgrade-base-image`, `migrate-to-base-image`. |
Expand Down
20 changes: 11 additions & 9 deletions airbyte-ci/connectors/pipelines/pipelines/tests/java_connectors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from typing import List, Optional

import anyio
import asyncer
from dagger import File, QueryError
from pipelines.actions import environments, secrets
from pipelines.bases import StepResult, StepStatus
Expand Down Expand Up @@ -87,11 +88,6 @@ async def run_all_tests(context: ConnectorContext) -> List[StepResult]:
if build_connector_image_results.status is StepStatus.FAILURE:
return step_results

unit_tests_results = await UnitTests(context).run()
step_results.append(unit_tests_results)
if context.fail_fast and unit_tests_results.status is StepStatus.FAILURE:
return step_results

if context.connector.supports_normalization:
normalization_image = f"{context.connector.normalization_repository}:dev"
context.logger.info(f"This connector supports normalization: will build {normalization_image}.")
Expand All @@ -107,9 +103,15 @@ async def run_all_tests(context: ConnectorContext) -> List[StepResult]:
connector_container = build_connector_image_results.output_artifact[LOCAL_BUILD_PLATFORM]
connector_image_tar_file, _ = await export_container_to_tarball(context, connector_container)

integration_tests_results = await IntegrationTests(context).run(connector_image_tar_file, normalization_tar_file)
step_results.append(integration_tests_results)
async with asyncer.create_task_group() as test_task_group:
soon_unit_tests_results = test_task_group.soonify(UnitTests(context).run)()
Copy link
Contributor

@postamar postamar Oct 16, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UnitTests doesn't depend on building and loading the connector image. You might get even better performance (due to gradle caching) if you kicked off this task as soon as the distTar is built. Possibly sooner.

Perhaps this is out of scope for this work but it would be nice to take a more concurrent approach here: kick off each of these steps ASAP and have each of them wait on their dependencies being fulfilled.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this suggestion, trying this out right now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It speeds things up by 1mn!

soon_integration_tests_results = test_task_group.soonify(IntegrationTests(context).run)(
connector_tar_file=connector_image_tar_file, normalization_tar_file=normalization_tar_file
)
soon_acceptance_tests_results = test_task_group.soonify(AcceptanceTests(context).run)(
connector_under_test_image_tar=connector_image_tar_file
)

step_results += [soon_acceptance_tests_results.value, soon_integration_tests_results.value, soon_unit_tests_results.value]

acceptance_tests_results = await AcceptanceTests(context).run(connector_image_tar_file)
step_results.append(acceptance_tests_results)
return step_results
2 changes: 1 addition & 1 deletion airbyte-ci/connectors/pipelines/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"

[tool.poetry]
name = "pipelines"
version = "1.9.1"
version = "1.9.2"
description = "Packaged maintained by the connector operations team to perform CI for connectors' pipelines"
authors = ["Airbyte <[email protected]>"]

Expand Down
Loading