Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Experimental BQ support to run dbt models with ExecutionMode.AIRFLOW_ASYNC #1230

Merged
merged 51 commits into from
Oct 3, 2024
Merged

Experimental BQ support to run dbt models with ExecutionMode.AIRFLOW_ASYNC #1230

merged 51 commits into from
Oct 3, 2024

Commits on Sep 29, 2024

  1. Draft: dbt compile task

    @pankajkoti
    pankajkoti committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    851564f View commit details
    Browse the repository at this point in the history

  2. Put compiled files under dag_id folder & refactor few snippets

    @pankajkoti
    pankajkoti committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    9dc2c9c View commit details
    Browse the repository at this point in the history

  3. Add tests & minor refactorings

    @pankajkoti
    pankajkoti committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    0ce662e View commit details
    Browse the repository at this point in the history

  4. Apply suggestions from code review

    @pankajkoti
    pankajkoti authored Sep 29, 2024
    Configuration menu
    Copy the full SHA
    1b6f57e View commit details
    Browse the repository at this point in the history

  5. Install deps for the newly added example DAG

    @pankajkoti
    pankajkoti committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    cc48161 View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Add docs

    @pankajkoti
    pankajkoti committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    1068025 View commit details
    Browse the repository at this point in the history

  2. Add async run operator 

    Remove print stmt

Fix query
Fix query

Remove oss execute method code
    @pankajkoti @pankajastro
    pankajkoti authored and pankajastro committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    faa706d View commit details
    Browse the repository at this point in the history

  3. Fix remote sql path and async args

    @pankajastro
    pankajastro committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    0e155e4 View commit details
    Browse the repository at this point in the history

  4. Fix query

    @pankajastro
    pankajastro committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    5f1ecaa View commit details
    Browse the repository at this point in the history

  5. 🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

    @pre-commit-ci
    pre-commit-ci[bot] committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    1278847 View commit details
    Browse the repository at this point in the history

  6. Use dbt node's filepath to construct remote path to fetch compiled SQ… 

    …L from
    @pankajkoti
    pankajkoti committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    b3d6cf3 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into execute-async-task

    @tatiana
    tatiana authored Sep 30, 2024
    Configuration menu
    Copy the full SHA
    78bc069 View commit details
    Browse the repository at this point in the history

  8. 🎨 [pre-commit.ci] Auto format from pre-commit.com hooks

    @pre-commit-ci
    pre-commit-ci[bot] committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    9ca5e85 View commit details
    Browse the repository at this point in the history

  9. Fix unittests

    @tatiana
    tatiana committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    99bf7c0 View commit details
    Browse the repository at this point in the history

  10. Improve code

    @tatiana
    tatiana committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    3aaaf9e View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Working with deferrable=False, not working with deferrable=True

    @tatiana
    tatiana committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    43158be View commit details
    Browse the repository at this point in the history

  2. Working with deferrable=False, not working with deferrable=True

    @tatiana
    tatiana committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    83b1010 View commit details
    Browse the repository at this point in the history

  3. Fix issue when using BQ deferrable operator - it requires location 

    Otherwise, it raises:

    return func(self, *args, **kwargs)
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/operators/airflow_async.py", line 110, in execute
    self.drop_table_sql()
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/operators/airflow_async.py", line 104, in drop_table_sql
    hook.insert_job(configuration=self.configuration, location=self.location, project_id=self.gcp_project)
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-210/lib/python3.10/site-packages/airflow/providers/google/common/hooks/base_google.py", line 528, in inner_wrapper
    return func(self, *args, **kwargs)
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-210/lib/python3.10/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 1676, in insert_job
    job_api_repr.result(timeout=timeout, retry=retry)
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-210/lib/python3.10/site-packages/google/cloud/bigquery/job/query.py", line 1590, in result
    do_get_result()
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-210/lib/python3.10/site-packages/google/cloud/bigquery/job/query.py", line 1579, in do_get_result
    super(QueryJob, self).result(retry=retry, timeout=timeout)
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-210/lib/python3.10/site-packages/google/cloud/bigquery/job/base.py", line 971, in result
    return super(_AsyncJob, self).result(timeout=timeout, **kwargs)
  File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-210/lib/python3.10/site-packages/google/api_core/future/polling.py", line 261, in result
    raise self._exception
google.api_core.exceptions.NotFound: 404 Not found: Dataset astronomer-dag-authoring:release_17 was not found in location US; reason: notFound, message: Not found: Dataset astronomer-dag-authoring:release_17 was not found in location US

Location: US
Job ID: airflow_1727777155412089_041416781bf829a03b1f9b8fbc79980d
    @tatiana
    tatiana committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    bd6657a View commit details
    Browse the repository at this point in the history

  4. Add limitation in docs

    @pankajastro
    pankajastro committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    1195955 View commit details
    Browse the repository at this point in the history

  5. Add full_refresh as templated field

    @pankajastro
    pankajastro committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    2bdd9bb View commit details
    Browse the repository at this point in the history

  6. Add more template fields

    @pankajastro
    pankajastro committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    4a44603 View commit details
    Browse the repository at this point in the history

  7. Construct & relay 'dbt dag-task group' identifier to upload & downloa… 

    …d compile SQL
    @pankajkoti
    pankajkoti committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    c3c51cb View commit details
    Browse the repository at this point in the history

  8. Fix model_name retrieval; get from dbt_node_config

    @pankajkoti
    pankajkoti committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    72c6164 View commit details
    Browse the repository at this point in the history

  9. Fix unit tests

    @pankajkoti
    pankajkoti committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    e67098e View commit details
    Browse the repository at this point in the history

  10. Fix subsequent failing unit tests

    @pankajkoti
    pankajkoti committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    3e550bf View commit details
    Browse the repository at this point in the history

  11. Fix type check failures

    @pankajkoti
    pankajkoti committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    0730d0f View commit details
    Browse the repository at this point in the history

  12. Add back the deleted sources.yml from jaffle_shop as it has dependenc… 

    …ies in models
    @pankajkoti
    pankajkoti committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    745768e View commit details
    Browse the repository at this point in the history

  13. Install dbt bigquery adapter for running simple_dag_async

    @pankajkoti
    pankajkoti committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    43d62ea View commit details
    Browse the repository at this point in the history

  14. Install dbt bigquery adapter in our CI setup scripts

    @pankajkoti
    pankajkoti committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    9656248 View commit details
    Browse the repository at this point in the history

  15. Update gcp conn in dev/dags/simple_dag_async.py

    @pankajkoti
    pankajkoti authored Oct 1, 2024
    Configuration menu
    Copy the full SHA
    a654f49 View commit details
    Browse the repository at this point in the history

  16. Refactor args in DbtRunAirflowAsyncOperator

    @tatiana
    tatiana committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    e60ace2 View commit details
    Browse the repository at this point in the history

  17. Use GoogleCloudServiceAccountDictProfileMapping in profilemapping

    @pankajkoti
    pankajkoti committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    7f055bc View commit details
    Browse the repository at this point in the history

  18. set should_upload_compiled_sql to True

    @pankajkoti
    pankajkoti authored Oct 1, 2024
    Configuration menu
    Copy the full SHA
    ad057c8 View commit details
    Browse the repository at this point in the history

  19. Remove async_op_args

    @tatiana
    tatiana committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    a70ca46 View commit details
    Browse the repository at this point in the history

  20. remove install_deps from DAG

    @pankajkoti
    pankajkoti authored Oct 1, 2024
    Configuration menu
    Copy the full SHA
    7c6a1b2 View commit details
    Browse the repository at this point in the history

  21. Merge branch 'main' into execute-async-task

    @tatiana
    tatiana authored Oct 1, 2024
    Configuration menu
    Copy the full SHA
    64a31d0 View commit details
    Browse the repository at this point in the history

  22. Fix test_build_airflow_graph_with_dbt_compile_task by passing needed … 

    …location key
    @pankajkoti
    pankajkoti committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    c1aeff0 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Specify required project id in the GoogleCloudServiceAccountDictProfi… 

    …leMapping in the DAG
    @pankajkoti
    pankajkoti committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    02f7985 View commit details
    Browse the repository at this point in the history

  2. Pass gcp_conn_id to super class init, otherwise it is lost & uses the… 

    … default google_cloud_default otherwise
    @pankajkoti
    pankajkoti committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    af454a9 View commit details
    Browse the repository at this point in the history

  3. Adapt manifest DAG to use & adapt to the newer GCP conn secret that i… 

    …s to be updated soon
    @pankajkoti
    pankajkoti committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    9081e6a View commit details
    Browse the repository at this point in the history

  4. Release 1.7.0a1

    @tatiana
    tatiana committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    2dccf84 View commit details
    Browse the repository at this point in the history

  5. Retrigger GH actions

    @tatiana
    tatiana committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    7adeb99 View commit details
    Browse the repository at this point in the history

  6. temporarily move out simple_dag_async.py

    @tatiana
    tatiana committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    7e6de30 View commit details
    Browse the repository at this point in the history

  7. Fix CI issue

    @tatiana
    tatiana committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    16a87ea View commit details
    Browse the repository at this point in the history

  8. Fix dbt-compile dependency by using Airflow tasks instead of dbt nodes

    @pankajkoti
    pankajkoti committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    05db6a0 View commit details
    Browse the repository at this point in the history

  9. Apply suggestions from code review 

    Co-authored-by: Tatiana Al-Chueyr <[email protected]>
    @pankajkoti @tatiana
    pankajkoti and tatiana authored Oct 2, 2024
    Configuration menu
    Copy the full SHA
    8fc4ae2 View commit details
    Browse the repository at this point in the history

  10. Apply suggestions from code review

    @pankajkoti
    pankajkoti authored Oct 2, 2024
    Configuration menu
    Copy the full SHA
    ea5816b View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. Add install instruction

    @pankajastro
    pankajastro committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    85f86a4 View commit details
    Browse the repository at this point in the history

  2. Add min airflow version in limitation

    @pankajastro
    pankajastro committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    402f823 View commit details
    Browse the repository at this point in the history

  3. Ignore Async DAG for dbt <=1.5

    @pankajastro
    pankajastro committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    621a4de View commit details
    Browse the repository at this point in the history

  4. Ignore Async DAG for dbt <=1.5

    @pankajastro
    pankajastro committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    a0cb147 View commit details
    Browse the repository at this point in the history