[CI] Reduce the amount taking to run tests in the CI from 5h to 11min #1297
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.
|
tatiana
force-pushed
the
fix-ci-for-tests.py3.8-2.6
branch
from
c0b17c4
to
830bde5
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1297 +/- ##
==========================================
+ Coverage 95.73% 95.85% +0.11%
==========================================
Files 67 67
Lines 3967 3976 +9
==========================================
+ Hits 3798 3811 +13
+ Misses 169 165 -4 ☔ View full report in Codecov by Sentry. |
dosubot
bot
added
size:XS
tatiana
force-pushed
the
fix-ci-for-tests.py3.8-2.6
branch
from
43338c3
to
45aceec
Compare
tatiana
added a commit
that referenced
this pull request
Nov 6, 2024
…1307) Previously, users were unable to run a DbtDag or DbtTask group where the following was used: - `RenderMode.DBT_LS` - `ExecutionConfig(dbt_project_path)` - `RenderConfig(dbt_project_path)` This scenario can be helpful when using ExecutionMode.KUBERNETES or other similar ones and was found out during: #1297
Co-authored-by: Pankaj Koti <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
Closes: #1299
Some CI jobs took an outrageous amount of time to run.
One of our CI test pipeline jobs was taking over five hours to run:
https://github.com/astronomer/astronomer-cosmos/actions/runs/11505558596
More recently, this same job started taking over 6 hours to run and started timing out in the CI, making Cosmos' main branch red for both unit and integration tests for Airflow 2.6. The underlying reason is that it took a long time to resolve dependencies. This seems to have happened since October 29, as seen on the commit 84c5fbd to main.
Example: https://github.com/astronomer/astronomer-cosmos/actions/runs/11594745046/job/32330864858
About this change
This PR solves the original issue by changing where and how we install Airflow dependencies, simplifying the previous setup. We manage Airflow test dependencies in the
pre-install-airflow.shfile, not inpyproject.toml, othersh, or the Github action. We are also being strict as we can dependent on the Airflow version. Where possible, we use constraints. Where different providers' dependencies conflict with previous versions of Airflow, we just ensure the expected version remains being used after the installation.Example of a successful run:
https://github.com/astronomer/astronomer-cosmos/actions/runs/11685652312
Bonus
I realised with this change that users who use K8s and want to define different paths for their dbt projects in Airflow and K8s were facing an issue. This problem was evident when running the k8s example DAG. I've fixed the problem as part of this PR.
Follow-up actions
Since this has been taking a long time to solve and our main branch is still red, I commented out two tasks that were failing tests - and I've logged a follow-up issue for us to address this:
#1304