Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[airflow] Avoid implicit DAG schedule (AIR301) #14581

Merged

Conversation

uranusjr
Copy link
Contributor

Summary

Airflow 3.0 changes the default of the 'schedule' argument. This causes incompatibility if a DAG previously does not specify a schedule explicitly. This is against best practice anyway---you should always prefer to set a schedule explicitly.

Due to backward compatibility, Airflow 2 also possesses multiple arguments to set the schedule. These are all being combined into 'schedule' in 3.0. Usages of those old arguments are also detected.

Test Plan

A test fixture has been included for the rule.

Airflow 3.0 changes the default of the 'schedule' argument. This causes
incompatibility if a DAG previously does not specify a schedule
explicitly. This is against best practice anyway---you should always
prefer to set a schedule explicitly.

Due to backward compatibility, Airflow 2 also possesses multiple
arguments to set the schedule. These are all being combined into
'schedule' in 3.0. Usages of those old arguments are also detected.
@uranusjr uranusjr changed the title [airflow] Avoid implicit DAG schedule (AIR002) [airflow] Avoid implicit DAG schedule (AIR301) Nov 25, 2024
Copy link
Contributor

github-actions bot commented Nov 25, 2024

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

ℹ️ ecosystem check detected linter changes. (+35 -0 violations, +0 -0 fixes in 1 projects; 54 projects unchanged)

apache/airflow (+35 -0 violations, +0 -0 fixes)

ruff check --no-cache --exit-zero --ignore RUF9 --output-format concise --preview --select ALL

+ airflow/models/dag.py:296:16: AIR301 DAG should have an explicit `schedule` argument
+ performance/src/performance_dags/performance_dag/performance_dag.py:230:11: AIR301 DAG should have an explicit `schedule` argument
+ providers/src/airflow/providers/arangodb/example_dags/example_arangodb.py:25:7: AIR301 DAG should have an explicit `schedule` argument
+ providers/src/airflow/providers/google/cloud/example_dags/example_cloud_task.py:41:6: AIR301 DAG should have an explicit `schedule` argument
+ providers/src/airflow/providers/google/cloud/example_dags/example_facebook_ads_to_gcs.py:61:6: AIR301 DAG should have an explicit `schedule` argument
+ providers/src/airflow/providers/google/cloud/example_dags/example_looker.py:31:6: AIR301 DAG should have an explicit `schedule` argument
+ providers/src/airflow/providers/google/cloud/example_dags/example_presto_to_gcs.py:52:6: AIR301 DAG should have an explicit `schedule` argument
+ providers/src/airflow/providers/google/cloud/example_dags/example_salesforce_to_gcs.py:47:6: AIR301 DAG should have an explicit `schedule` argument
+ providers/src/airflow/providers/google/marketing_platform/example_dags/example_display_video.py:124:6: AIR301 DAG should have an explicit `schedule` argument
+ providers/src/airflow/providers/google/marketing_platform/example_dags/example_display_video.py:172:6: AIR301 DAG should have an explicit `schedule` argument
+ providers/src/airflow/providers/google/marketing_platform/example_dags/example_display_video.py:91:6: AIR301 DAG should have an explicit `schedule` argument
+ providers/src/airflow/providers/oracle/example_dags/example_oracle.py:25:6: AIR301 DAG should have an explicit `schedule` argument
+ providers/tests/google/cloud/operators/test_looker.py:47:19: AIR301 DAG should have an explicit `schedule` argument
+ providers/tests/google/cloud/utils/airflow_util.py:54:15: AIR301 DAG should have an explicit `schedule` argument
+ providers/tests/google/cloud/utils/test_mlengine_operator_utils.py:64:12: AIR301 DAG should have an explicit `schedule` argument
+ providers/tests/integration/mongo/sensors/test_mongo.py:49:20: AIR301 DAG should have an explicit `schedule` argument
+ providers/tests/integration/qdrant/operators/test_qdrant_ingest.py:38:20: AIR301 DAG should have an explicit `schedule` argument
+ providers/tests/integration/redis/sensors/test_redis_key.py:35:20: AIR301 DAG should have an explicit `schedule` argument
+ providers/tests/integration/redis/sensors/test_redis_pub_sub.py:38:20: AIR301 DAG should have an explicit `schedule` argument
+ providers/tests/integration/ydb/operators/test_ydb.py:51:20: AIR301 DAG should have an explicit `schedule` argument
+ providers/tests/system/apache/kafka/example_dag_event_listener.py:85:6: AIR301 DAG should have an explicit `schedule` argument
+ providers/tests/system/asana/example_asana.py:49:6: AIR301 DAG should have an explicit `schedule` argument
+ providers/tests/system/google/cloud/cloud_functions/example_functions.py:81:6: AIR301 DAG should have an explicit `schedule` argument
+ task_sdk/tests/dags/super_basic.py:24:2: AIR301 DAG should have an explicit `schedule` argument
+ task_sdk/tests/dags/super_basic_run.py:30:2: AIR301 DAG should have an explicit `schedule` argument
+ task_sdk/tests/defintions/test_baseoperator.py:283:23: AIR301 DAG should have an explicit `schedule` argument
+ task_sdk/tests/defintions/test_baseoperator.py:352:10: AIR301 DAG should have an explicit `schedule` argument
+ task_sdk/tests/defintions/test_baseoperator.py:359:10: AIR301 DAG should have an explicit `schedule` argument
+ task_sdk/tests/defintions/test_dag.py:296:9: AIR301 DAG should have an explicit `schedule` argument
+ task_sdk/tests/defintions/test_dag.py:339:14: AIR301 DAG should have an explicit `schedule` argument
+ task_sdk/tests/defintions/test_dag.py:345:16: AIR301 DAG should have an explicit `schedule` argument
+ tests/jobs/test_scheduler_job.py:6434:16: AIR301 DAG should have an explicit `schedule` argument
+ tests/models/test_dag.py:2855:10: AIR301 DAG should have an explicit `schedule` argument
+ tests/serialization/test_serialized_objects.py:218:47: AIR301 DAG should have an explicit `schedule` argument
+ tests/utils/test_log_handlers.py:273:15: AIR301 DAG should have an explicit `schedule` argument

Changes by rule (1 rules affected)

code total + violation - violation + fix - fix
AIR301 35 35 0 0 0

Copy link
Member

@MichaReiser MichaReiser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This overall looks good to me, but I think we have to figure out the right scoping for all the Airflow3 deprecation rules.

Comment on lines +15 to +16
/// almost never what a user is looking for. Airflow 3 changes this the default
/// to *None*, and would break existing DAGs using the implicit default.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we tell a bit more about how it breaks existing DAGS that use the implicit default. Will it fail with an exception or is it just that the default is different?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just the default is different—your DAGs will simply stop doing anything silently. I’ll add a paragraph to call this out.

@MichaReiser MichaReiser added rule Implementing or modifying a lint rule preview Related to preview mode features labels Nov 25, 2024
@uranusjr uranusjr force-pushed the airflow-upgrade-dag-schedule-argument branch from 4e33f61 to 7745a16 Compare November 26, 2024 10:35
Copy link
Member

@MichaReiser MichaReiser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great. Thank you :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
preview Related to preview mode features rule Implementing or modifying a lint rule
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants