Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Airflow 2 to 3 auto migration rules - ruff #44556

Open
Tracked by #41641
Lee-W opened this issue Dec 2, 2024 · 1 comment
Open
Tracked by #41641

Airflow 2 to 3 auto migration rules - ruff #44556

Lee-W opened this issue Dec 2, 2024 · 1 comment
Assignees
@Lee-W
Labels
area:dependencies Issues related to dependencies problems area:dev-tools area:upgrade Facilitating migration to a newer version of Airflow kind:feature Feature Requests

Comments

@Lee-W
Copy link
Member

Lee-W commented Dec 2, 2024 •
edited
Loading

Description

Ruff

AIR302

function

args

method

attribute

context key

asset related change (need reorganized)

#41348

  • module airflow.datasets → airflow.sdk.definitions.asset
    • class
      • DatasetAlias → AssetAlias
      • DatasetAll → AssetAll
      • DatasetAny → AssetAny
    • function
      • expand_alias_to_datasets → expand_alias_to_assets
    • class DatasetAliasEvent → AssetAliasEvent
      • attribute dest_dataset_uri → BaseAsset
    • class
      • BaseDataset → BaseAsset
      • Dataset → Asset
      • method
        • iter_datasets → iter_assets
        • iter_dataset_aliases → iter_asset_aliases
  • module airflow.datasets.manager → airflow.assets.manager
    • variable dataset_manager → asset_manager
    • function resolve_dataset_manager → resolve_asset_manager
    • class DatasetManager → AssetManager
      • method
        • register_dataset_change → register_asset_change
        • create_datasets → create_assets
        • register_dataset_change → notify_asset_created
        • notify_dataset_changed → notify_asset_changed
        • notify_dataset_alias_created → notify_asset_alias_created
  • module airflow.listeners.spec.dataset → airflow.listeners.spec.asset
    • function
      • on_dataset_created → on_asset_created
      • on_dataset_changed → on_asset_changed
  • module airflow.timetables.datasets → airflow.timetables.assets
    • class DatasetOrTimeSchedule → AssetOrTimeSchedule
  • class airflow.lineage.hook.DatasetLineageInfo → airflow.lineage.hook.AssetLineageInfo
    • attribute dataset → asset
  • package airflow.providers.amazon.aws.datasets → airflow.providers.amazon.aws.assets
    • in module s3
      • method create_dataset → create_asset
      • method convert_dataset_to_openlineage → convert_asset_to_openlineage
  • package airflow.providers.common.io.datasets → airflow.providers.common.io.assets
    • in module file
      • method create_dataset → create_asset
      • method convert_dataset_to_openlineage → convert_asset_to_openlineage
  • package
    • airflow.providers.postgres.datasets → airflow.providers.postgres.assets
    • airflow.providers.mysql.datasets → airflow.providers.mysql.assets
    • airflow.providers.trino.datasets → airflow.providers.trino.assets
  • module
    • airflow.datasets.metadata → airflow.sdk.definitions.asset.metadata
  • class
    • airflow.timetables.datasets.DatasetOrTimeSchedule → airflow.timetables.assets.AssetOrTimeSchedule
    • airflow.auth.managers.models.resource_details.DatasetDetails → airflow.auth.managers.models.resource_details.AssetDetails
    • airflow.timetables.simple.DatasetTriggeredTimetable → airflow.timetables.simple.AssetTriggeredTimetable
    • airflow.providers.openlineage.utils.utils.DatasetInfo → airflow.providers.openlineage.utils.utils.AssetInfo
  • method
    • airflow.providers.amazon.auth_manager.aws_auth_manager.AwsAuthManager.is_authorized_dataset → airflow.providers.amazon.auth_manager.aws_auth_manager.AwsAuthManager.is_authorized_asset
    • airflow.lineage.hook.HookLineageCollector.create_dataset → airflow.lineage.hook.HookLineageCollector.create_asset
    • airflow.lineage.hook.HookLineageCollector.add_input_dataset → airflow.lineage.hook.HookLineageCollector.add_input_asset
    • airflow.lineage.hook.HookLineageCollector.add_output_dataset → airflow.lineage.hook.HookLineageCollector.dd_output_asset
    • airflow.lineage.hook.HookLineageCollector.collected_datasets → airflow.lineage.hook.HookLineageCollector.collected_assets
    • airflow.providers_manager.ProvidersManager.initialize_providers_dataset_uri_resources → airflow.providers_manager.ProvidersManager.initialize_providers_asset_uri_resources
  • function
    • airflow.api_connexion.security.requires_access_dataset → airflow.api_connexion.security.requires_access_dataset.requires_access_asset
    • airflow.auth.managers.base_auth_manager.is_authorized_dataset → airflow.auth.managers.base_auth_manager.is_authorized_asset
    • airflow.www.auth.has_access_dataset → airflow.www.auth.has_access_dataset.has_access_asset
    • airflow.providers.fab.auth_manager.fab_auth_manager.is_authorized_dataset → airflow.providers.fab.auth_manager.fab_auth_manager.is_authorized_asset
    • airflow.providers.openlineage.utils.utils.translate_airflow_dataset → airflow.providers.openlineage.utils.utils.translate_airflow_asset
  • property
    • airflow.providers_manager.ProvidersManager.dataset_factories → airflow.providers_manager.ProvidersManager.asset_factories
    • airflow.providers_manager.ProvidersManager.dataset_uri_handlers → airflow.providers_manager.ProvidersManager.asset_uri_handlers
    • airflow.providers_manager.ProvidersManager.dataset_to_openlineage_converters → airflow.providers_manager.ProvidersManager.asset_to_openlineage_converters
  • constant / variable
    • airflow.security.permissions.RESOURCE_DATASET → airflow.security.permissions.RESOURCE_ASSET
    • airflow.providers.amazon.auth_manager.avp.entities.AvpEntities.DATASET → airflow.providers.amazon.auth_manager.avp.entities.AvpEntities.ASSET
  • context key
    • triggering_dataset_events → triggering_asset_events
  • resource key
    • dataset-uris → asset-uris (for providers amazon, common.io, mysql, fab, postgres, trino)

AIR303: moved to provider

AIR303 (TODO) Some of the paths are not correct, but can be easily fixed by reading the origin PR

package

module

class

function

constant / variable

AIR310: models related changes (AIP-72) not going to do it

Rules to blocked by AIP-72

In review rules

AIR303

Merged rules

AIR302

args

names

Use case/motivation

No response

Related issues

#41641

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@Lee-W Lee-W added area:dev-tools kind:feature Feature Requests labels Dec 2, 2024
@Lee-W Lee-W self-assigned this Dec 2, 2024
@dosubot dosubot bot added area:dependencies Issues related to dependencies problems area:upgrade Facilitating migration to a newer version of Airflow labels Dec 2, 2024
This was referenced Dec 2, 2024
Open
Open
Closed
Merged
@Lee-W
Copy link
Member Author

Lee-W commented Dec 3, 2024 •
edited
Loading

Current progress

Updated: 12/11

This was referenced Dec 4, 2024
Merged
Merged
MichaReiser pushed a commit to astral-sh/ruff that referenced this issue Dec 6, 2024
@Lee-W
[airflow]: extend removed names (AIR302) (#14804)
39623f8 
## Summary

Airflow 3.0 removes various deprecated functions, members, modules, and
other values. They have been deprecated in 2.x, but the removal causes
incompatibilities that we want to detect. This PR deprecates the
following names.
The full list of rules we will extend
apache/airflow#44556


#### package
* `airflow.contrib.*` 

#### module
* `airflow.operators.subdag.*` 

#### class
* `airflow.sensors.external_task.ExternalTaskSensorLink` →
`airflow.sensors.external_task.ExternalDagLin`
* `airflow.operators.bash_operator.BashOperator` →
`airflow.operators.bash.BashOperator`
* `airflow.operators.branch_operator.BaseBranchOperator` →
`airflow.operators.branch.BaseBranchOperator`
* `airflow.operators.dummy.EmptyOperator` →
`airflow.operators.empty.EmptyOperator`
* `airflow.operators.dummy.DummyOperator` →
`airflow.operators.empty.EmptyOperator`
* `airflow.operators.dummy_operator.EmptyOperator` →
`airflow.operators.empty.EmptyOperator`
* `airflow.operators.dummy_operator.DummyOperator` →
`airflow.operators.empty.EmptyOperator`
* `airflow.operators.email_operator.EmailOperator` →
`airflow.operators.email.EmailOperator`
* `airflow.sensors.base_sensor_operator.BaseSensorOperator` →
`airflow.sensors.base.BaseSensorOperator`
* `airflow.sensors.date_time_sensor.DateTimeSensor` →
`airflow.sensors.date_time.DateTimeSensor`
* `airflow.sensors.external_task_sensor.ExternalTaskMarker` →
`airflow.sensors.external_task.ExternalTaskMarker`
* `airflow.sensors.external_task_sensor.ExternalTaskSensor` →
`airflow.sensors.external_task.ExternalTaskSensor`
* `airflow.sensors.external_task_sensor.ExternalTaskSensorLink` →
`airflow.sensors.external_task.ExternalTaskSensorLink`
* `airflow.sensors.time_delta_sensor.TimeDeltaSensor` →
`airflow.sensors.time_delta.TimeDeltaSensor`

#### function
* `airflow.utils.decorators.apply_defaults`
* `airflow.www.utils.get_sensitive_variables_fields` →
`airflow.utils.log.secrets_masker.get_sensitive_variables_fields`
* `airflow.www.utils.should_hide_value_for_key` →
`airflow.utils.log.secrets_masker.should_hide_value_for_key`
* `airflow.configuration.get` → `airflow.configuration.conf.get` 
* `airflow.configuration.getboolean` →
`airflow.configuration.conf.getboolean`
* `airflow.configuration.getfloat` →
`airflow.configuration.conf.getfloat`
* `airflow.configuration.getint` → `airflow.configuration.conf.getint` 
* `airflow.configuration.has_option` →
`airflow.configuration.conf.has_option`
* `airflow.configuration.remove_option` →
`airflow.configuration.conf.remove_option`
* `airflow.configuration.as_dict` → `airflow.configuration.conf.as_dict`
* `airflow.configuration.set` → `airflow.configuration.conf.set` 
* `airflow.secrets.local_filesystem.load_connections` →
`airflow.secrets.local_filesystem.load_connections_dict`
* `airflow.secrets.local_filesystem.get_connection` →
`airflow.secrets.local_filesystem.load_connections_dict`
* `airflow.utils.helpers.chain` → `airflow.models.baseoperator.chain` 
* `airflow.utils.helpers.cross_downstream` →
`airflow.models.baseoperator.cross_downstream`

#### attribute
* in `airflow.utils.trigger_rule.TriggerRule`
    * `DUMMY` 
    * `NONE_FAILED_OR_SKIPPED` 

#### constant / variable
* `airflow.PY\d\d`
MichaReiser pushed a commit to astral-sh/ruff that referenced this issue Dec 6, 2024
@Lee-W
[airflow]: extend removed args (AIR302) (#14765)
3ea14d7 
## Summary

Airflow 3.0 removes various deprecated functions, members, modules, and
other values. They have been deprecated in 2.x, but the removal causes
incompatibilities that we want to detect. This PR deprecates the
following names.

* in `DAG`
    * `sla_miss_callback` was removed
* in `airflow.operators.trigger_dagrun.TriggerDagRunOperator`
    * `execution_date` was removed
* in `airflow.operators.weekday.DayOfWeekSensor`,
`airflow.operators.datetime.BranchDateTimeOperator` and
`airflow.operators.weekday.BranchDayOfWeekOperator`
* `use_task_execution_day` was removed in favor of
`use_task_logical_date`

The full list of rules we will extend
apache/airflow#44556

## Test Plan

<!-- How was it tested? -->
A test fixture is included in the PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Lee-W Lee-W

Labels
area:dependencies Issues related to dependencies problems area:dev-tools area:upgrade Facilitating migration to a newer version of Airflow kind:feature Feature Requests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Lee-W