Skip to content

Commit

Permalink
Introduce sleep in tests to avoid bigquery rate limits (#2110)
Browse files Browse the repository at this point in the history
We're encountering difficulties achieving success with the 
integration test `test_aql_replace_existing_table` in our CI
environment.
The issue stems from `Google BigQuery` imposing rate limits on table 
operations—specifically, restricting them to `5` per `10` seconds per
table.
As a consequence, our CI pipeline fails, thereby hindering the release
of
version `1.18.0.`
The rate limits are documented here: 
https://cloud.google.com/bigquery/quotas#standard_tables

Initially, I contemplated skipping this test in CI. However,
implementing
a `10s` sleep for the problematic `BigQuery` test appears to 
circumvent the rate limit error, resulting in a fully successful CI
build.

I will cherrypick this PR in the release branch.
  • Loading branch information
pankajkoti committed Jan 24, 2024
1 parent 0436fb9 commit e9127df
Show file tree
Hide file tree
Showing 3 changed files with 17 additions and 3 deletions.
2 changes: 1 addition & 1 deletion python-sdk/src/astro/sql/operators/raw_sql.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

try:
from airflow.decorators.base import TaskDecorator
except ImportError:
except ImportError: # pragma: no cover
from airflow.decorators import _TaskDecorator as TaskDecorator # type: ignore[attr-defined]

import airflow
Expand Down
2 changes: 1 addition & 1 deletion python-sdk/src/astro/sql/operators/transform.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

try:
from airflow.decorators.base import TaskDecorator
except ImportError:
except ImportError: # pragma: no cover
from airflow.decorators import _TaskDecorator as TaskDecorator # type: ignore[attr-defined]

from airflow.decorators.base import get_unique_task_id, task_decorator_factory
Expand Down
16 changes: 15 additions & 1 deletion python-sdk/tests_integration/sql/operators/test_load_file.py
Original file line number Diff line number Diff line change
Expand Up @@ -207,12 +207,26 @@ def test_aql_load_remote_file_to_dbs(sample_dag, database_table_fixture, remote_
indirect=True,
ids=["snowflake", "bigquery", "postgresql", "sqlite", "redshift", "mssql", "mysql", "duckdb"],
)
def test_aql_replace_existing_table(sample_dag, database_table_fixture):
def test_aql_replace_existing_table(sample_dag, database_table_fixture, request):
import time

test_id = request.node.callspec.id

db, test_table = database_table_fixture
data_path_1 = str(CWD) + "/../../data/homes.csv"
data_path_2 = str(CWD) + "/../../data/homes2.csv"
with sample_dag:
# Bigquery rate limits the number of tables operations per 10s to 5 table operations.
# See more here: https://cloud.google.com/bigquery/quotas#standard_tables
# Hence, introduce a sleep for 10s to avoid rate limit errors.
if test_id == "bigquery":
time.sleep(10)

task_1 = load_file(input_file=File(data_path_1), output_table=test_table)

if test_id == "bigquery":
time.sleep(10)

task_2 = load_file(input_file=File(data_path_2), output_table=test_table)
task_1 >> task_2 # skipcq: PYL-W0104
test_utils.run_dag(sample_dag)
Expand Down

0 comments on commit e9127df

Please sign in to comment.