- Change deprecation status to Yes/No instead of icons (#1572)
- Show deprecation status of operators and sensors on the docs operator list page (#1568)
- Fix
SnowflakeSensorTrigger
storing callable as string (#1530)
- Pass
parameters
to parent class inSnowflakeSensorAsync
(#1519)
- Add ExternalDeploymentSensor (#1472)
- Deprecate the following operators and sensors and proxy them to their upstream Apache Airflow providers' deferrable counterparts
- EmrContainerSensorAsync
- EmrStepSensorAsync
- EmrJobFlowSensorAsync
- RedshiftClusterSensorAsync
- BatchSensorAsync
- S3KeySensorAsync
- S3KeysUnchangedSensorAsync
- EmrContainerOperatorAsync
- RedshiftDeleteClusterOperatorAsync
- RedshiftResumeClusterOperatorAsync
- RedshiftPauseClusterOperatorAsync
- BatchOperatorAsync
- SageMakerProcessingOperatorAsync
- SageMakerTransformOperatorAsync
- SageMakerTrainingOperatorAsync
- RedshiftDataOperatorAsync
- GCSObjectExistenceSensorAsync
- GCSObjectsWithPrefixExistenceSensorAsync
- GCSUploadSessionCompleteSensorAsync
- GCSObjectUpdateSensorAsync
- GKEStartPodOperatorAsync
- BigQueryInsertJobOperatorAsync
- BigQueryCheckOperatorAsync
- BigQueryGetDataOperatorAsync
- BigQueryIntervalCheckOperatorAsync
- BigQueryValueCheckOperatorAsync
- DataprocCreateClusterOperatorAsync
- DataprocDeleteClusterOperatorAsync
- DataprocSubmitJobOperatorAsync
- DataprocUpdateClusterOperatorAsync
- BigQueryTableExistenceSensorAsync
- AzureDataFactoryPipelineRunStatusSensorAsync
- WasbBlobSensorAsync
- WasbPrefixSensorAsync
- AzureDataFactoryRunPipelineOperatorAsync
- FileSensorAsync
- ExternalTaskSensorAsync
- SnowflakeSqlApiOperatorAsync
- DatabricksSubmitRunOperatorAsync
- DatabricksRunNowOperatorAsync
- KubernetesPodOperatorAsync
- SFTPSensorAsync
- HttpSensorAsync
- DbtCloudJobRunSensorAsync
- DbtCloudRunJobOperatorAsync
- LivyOperatorAsync
- Add deprecation notice docs for astronomer-providers (#1468)
- Increase min airflow version to 2.6 (#1438)
- Bump mininium python to 3.8 (#1442)
- DatabricksHookAsync: include 403 as retryable error (#1376)
- Revert "Pin max versions for CNCF Kubernetes and Google Airflow providers (#1359)" (#1368)
- Pin max versions for CNCF Kubernetes and Google Airflow providers (#1359)
- Pin
connexion==2.14.2
due to issue with latest release version3.0.0 (#1354)
- Handle
ClientConnectorError
to allow for retries (#1338) - Make
resource_group_name
andfactory_name
required in Azure DataFactory hook and trigger (#1341)
- Add
keep_response
parameter toHttpHookAsync
(#1330) - Add AioRefreshableCredentials to AwsBaseHookAsync (#1304)
- Fix ExternalDeploymentTaskSensorAsync by making simple http requests instead of aiohttp (#1306)
- Fix impersonation failure on GKE start pod operator async (#1274)
- Bring back hive provider for Python3.11 (#1317)
- Raise
AirflowSkipException
for sensors whensoft_fail
is set to True(#1276) S3KeyTrigger
: Call S3 sensor check_fn only with file size attribute (#1278)HttpSensorAsync
: Fix incorrect warning when poke_interval is passed (#1281)
hive
provider is not supported on Python 3.11 (#1237)
RedshiftDataHook
: remove snowflake dependencies and use DbApiHook.split_sql_string for parsing sqlDatabricksRunNowOperatorAsync
: get job_id through Databricks API if job_name is passed
- Revert "feat(kubernetes): check state before deferring KubernetesPodOperatorAsync (#1104)" (#1209)
- Fixing the issue that
check_fn
was ignored inS3KeySensorAsync
(#1171)
- Enhance
SFTPSensorAsync
by adding below features (#1072):- Remove the need for prefixing
ssh-
to host keys that don't have such prefix e.g. ecdsa type keys - Support validating host keys using a known_hosts file
- Accept string values for
newer_than
field. e.g. passed via Jinja template - Use
readdir
for listing files instead ofstat
on each file to avoid throttling caused by multiple roundtrips to the server for each file
- Remove the need for prefixing
- Mark
DbtCloudRunJobOperatorAsync
failed if cancelled by raisingAirflowFailException
(#1082)
- Support host key verification for
SFTPSensorAsync
(#963) - Make BigQuery & Google Cloud Storage async operators & sensors compatible with
apache-airflow-providers-google>=10.0.0
(#981) - Make
SageMakerProcessingOperatorAsync
compatible withapache-airflow-providers-amazon>=8.0.0
(#979) - Make
BatchOperatorAsync
compatible withapache-airflow-providers-amazon>=8.0.0
- Fix backward compatibility issue with BigQuery Async Operators (#967)
- Allow and prefer non-prefixed extra fields for
KubernetesHookAsync
(#944) - Make
GKEStartPodOperatorAsync
compatible withapache-airflow-providers-google>=9.0.0
. (#954) - BigQuery Async Operators accepts poll_interval as argument to override trigger's default poll_interval. (#953)
- Fix S3 and GCS custom XCOM backend json loading issue (#961)
- Pinning snowflake-sqlalchemy to greater than or equal to 1.4.4 (#962)
- Fix
S3KeyTrigger
to return only one trigger event whencheck_fn
is set to none. (#925) - Handle unclosed connection errors in
HttpTrigger
. (#927)
- Allow and prefer non-prefixed extra fields for
AzureDataFactoryHookAsync
. (#899) - Fix
HttpSensorAsync
to use the correct connection id instead of a default connection id (#896) - Fix
SFTPSensorAsync
to succeed when there is at least one file newer than the provided date and setSFTPHookAsync
default_known_hosts parameter to None. (#905) - Make
SageMakerTransformOperatorAsync
andSageMakerTrainingOperatorAsync
compatible withapache-airflow-providers-amazon>=7.3.0
. (#912) - Fix
BigQueryInsertJobOperatorAsync
to send location parameter to hook. (#866) - Mitigate race condition on deferrable sensors
ExternalTaskSensorAsync
andSnowflakeSensorAsync
immediately getting successful, by making the changes to these sensors to check on the worker first, then defer if the condition is not met. (#908)
This release adds a new async sensor SnowflakeSensorAsync
.
Sensor Class | Import Path | Example DAG |
---|---|---|
SnowflakeSensorAsync |
from astronomer.providers.snowflake.sensors.snowflake import SnowflakeSensorAsync |
Example DAG |
- Support tags param in
EmrContainerOperatorAsync
. (#836) - Enhance
DbtCloudHookAsync
to accept tenant domain name while making connection. (#855)
- Ensure
DataprocLink
is visible for tracking the dataproc jobs status inDataprocSubmitJobOperatorAsync
. (#835) - Handle
SnowflakeHookAsync
when a empty sql list is passed`. (#838) - Use
timeout
instead ofexecution_timeout
withExternalTaskSensor
when sensing task_id`. (#858)
- Add custom XCom backend for S3. (#820)
- Add poke_interval to
ExternalTaskSensorAsync
. (#823) - Support getting logs of pod, pod cleanup and reflect status of the pod in
GKEStartPodOperatorAsync
(#824)
- Enhance
AwsBaseHookAsync
to support assume rolerole_arn
passed viaextra_config
in the airflow connection. (#804) - Support private key authentication for
SFTPHookAsync
via SFTP connection. (#749)
- Fix
KubernetesPodOperatorAsync
to consider kubernetes connection id in the trigger. (#815)
- Add custom XCom backend for GCS. (#681)
- Change return value of
SnowflakeOperatorAsync
to be same as that ofSnowflakeOperator
. (#781) - Add poke_interval to
S3KeySensorAsync
. (#782) - Use
AwsConnectionWrapper
to get connection details to create AWS async client. (#758) - Fix ADF Sensor broken docs. (#779)
- Log warning message when
response_check
attribute is passed inHttpSensorAsync
. (#780)
- Handle
SFTPSensorAsync
failure when file pattern is not passed. (#744) - Fix
RedshiftDataHook
to accept AWS access and secret keys from the connection object. (#746) - Make
BigQueryHookAsync
compatible withapache-airflow-providers-google>=8.5.0
. (#751) - Make
RedshiftSQLOperatorAsync
compatible withapache-airflow-providers-amazon>=6.1.0
. (#762) - Make
SnowflakeOperatorAsync
andSnowflakeSqlApiOperatorAsync
compatible withapache-airflow-providers-snowflake>=4.0.0
. (#767)
- Fix
AwsBaseHookAsync
to accept the AWS session token passed viaextra_config
in the airflow connection (#730) - Change return value of
SageMakerTrainingOperatorAsync
andSageMakerTransformOperatorAsync
to be same as that of corresponding Sync operators (#737)
This release adds the following 3 new async operators:
Operator Class | Import Path | Example DAG |
---|---|---|
SageMakerProcessingOperatorAsync |
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerProcessingOperatorAsync |
Example DAG |
SageMakerTrainingOperatorAsync |
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerTrainingOperatorAsync |
Example DAG |
SageMakerTransformOperatorAsync |
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerTransformOperatorAsync |
Example DAG |
This release adds the following 2 new async sensors/operators:
Operator/Sensor Class | Import Path | Example DAG |
---|---|---|
ExternalDeploymentTaskSensorAsync |
from astronomer.providers.core.sensors.external_task import ExternalDeploymentTaskSensorAsync |
Example DAG |
SFTPSensorAsync |
from astronomer.providers.sftp.sensors.sftp import SFTPSensorAsync |
Example DAG |
- Make Dataproc operator compatible with
apache-airflow-providers-google>=8.4.0
(#680) - Make EMR EKS operator compatible with
apache-airflow-providers-amazon>=6.0.0
(#682)
- Deprecate
poll_interval
and usepoke_interval
for all async sensors (#640)
This release adds the following 2 new async sensors/operators:
Operator/Sensor Class | Import Path | Example DAG |
---|---|---|
DbtCloudJobRunSensorAsync |
from astronomer.providers.dbt.cloud.sensors.dbt import DbtCloudJobRunSensorAsync |
Example DAG |
DbtCloudRunJobOperatorAsync |
from astronomer.providers.dbt.cloud.operators.dbt import DbtCloudRunJobOperatorAsync |
Example DAG |
- Include
astronomer-providers
in the Providers view within the Airflow UI (#626)
- Implement OpenLineage custom extractor for Redshift Async Operators (#561)
- Fix timeout errors on
AzureDataFactoryRunPipelineOperatorAsync
(#602) - Remove
werkzeug
dep & limitprotobuf
to3.20.0
(#615) - Raise exception in case of user error in async Databricks Operator (#612)
- Add poll interval to
HttpSensorAsync
(#554) - Replace execution_timeout with timeout in all the async sensors (#555)
- Get default 'resource_group_name' and 'factory_name' for AzureDataFactoryPipelineRunStatusSensorAsync (#589)
- Add elaborate documentation and use cases for
SnowflakeOperatorAsync
(#556) - Improve telemetry for Async Databricks Operators (#582)
- Enhance
S3KeySensorAsync
to accept multiple keys and deprecateS3PrefixSensorAsync
andS3KeySizeSensorAsync
(#577)
- Bump up
MarkupSafe
version as per Airflow 2.3.3 constraints (#542) - Downgrade
Werkzeug
version below 2.2.0 as it causesImportError: cannot import name 'parse_rule' from 'werkzeug.routing'
(#551)
This release adds the following 4 new async sensors/operators:
Operator/Sensor Class | Import Path | Example DAG |
---|---|---|
BatchSensorAsync |
from astronomer.providers.amazon.aws.sensors.batch import BatchSensorAsync |
Example DAG |
SnowflakeSqlApiOperatorAsync |
from astronomer.providers.snowflake.operators.snowflake import SnowflakeSqlApiOperatorAsync |
Example DAG |
WasbBlobSensorAsync |
from astronomer.providers.microsoft.azure.sensors.wasb import WasbBlobSensorAsync |
Example DAG |
WasbPrefixSensorAsync |
from astronomer.providers.microsoft.azure.sensors.wasb import WasbPrefixSensorAsync |
Example DAG |
- Add copy button to code blocks in docs (#505)
- Add custom Sphinx extension to list available operators & sensors (#504)
- Add pre-commit hook to check for dead links in markdown files (#524)
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class | Import Path | Example DAG |
---|---|---|
DataprocCreateClusterOperatorAsync |
from astronomer.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperatorAsync |
Example DAG |
DataprocDeleteClusterOperatorAsync |
from astronomer.providers.google.cloud.operators.dataproc import DataprocDeleteClusterOperatorAsync |
Example DAG |
DataprocUpdateClusterOperatorAsync |
from astronomer.providers.google.cloud.operators.dataproc import DataprocUpdateClusterOperatorAsync |
Example DAG |
RedshiftDataOperatorAsync |
from astronomer.providers.amazon.aws.operators.redshift_data import RedshiftDataOperatorAsync |
Example DAG |
RedshiftDeleteClusterOperatorAsync |
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftDeleteClusterOperatorAsync |
Example DAG |
- Implement OpenLineage custom extractor for BigQuery Async Operators (#429)
- Add session specific query tag and OpenLineage Extractor for Snowflake Async operator (#437)
- Handle
DataprocCreateClusterOperatorAsync
errors gracefully and add additional functionality withuse_if_exists
anddelete_on_error
parameters (#448)
- Fix
BigQueryInsertJobOperatorAsync
failure after Google provider upgrade to 8.1.0 (#471)
This release adds the following 2 new async sensors/operators:
Operator/Sensor Class | Import Path | Example DAG |
---|---|---|
BatchOperatorAsync |
from astronomer.providers.amazon.aws.operators.batch import BatchOperatorAsync |
Example DAG |
GKEStartPodOperatorAsync |
from astronomer.providers.google.cloud.operators.kubernetes_engine import GKEStartPodOperatorAsync |
Example DAG |
- Enhance KubernetesPodOperatorAsync to periodically resume the sync portion of the task to fetch and emit the latest logs before deferring again. (#139)
- Fix a bug on the KubernetesPodOperatorAsync to not fail with
ERROR - Unclosed client session
(#394)
- Enable Kerberos Authentication in
HivePartitionSensorAsync
andNamedHivePartitionSensorAsync
(#357)
- Fix example Redshift DAGs to catch appropriate exception during cluster deletion (#348)
- Move
xcom_push
call toexecute
method for all async operators (#371)
- Correct module name for
DagStateTrigger
which prevented use ofExternalTaskSensorAsync
whenexternal_task_id
was not passed (#361) - Add
template_fields
toS3KeySensorAsync
(#373)
- Add missing Extras in
README.rst
and automate it (#329)
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class | Import Path | Example DAG |
---|---|---|
AzureDataFactoryRunPipelineOperatorAsync |
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperatorAsync |
Example DAG |
AzureDataFactoryPipelineRunStatusSensorAsync |
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryPipelineRunStatusSensorAsync |
Example DAG |
EmrContainerOperatorAsync |
from astronomer.providers.amazon.aws.operators.emr import EmrContainerOperatorAsync |
Example DAG |
HivePartitionSensorAsync |
from astronomer.providers.apache.hive.sensors.hive_partition import HivePartitionSensorAsync |
Example DAG |
NamedHivePartitionSensorAsync |
from astronomer.providers.apache.hive.sensors.named_hive_partition import NamedHivePartitionSensorAsync |
Example DAG |
- Improved example DAGs so that minimal resources are created during integration tests
- Fixes a bug on the DatabricksRunNowOperatorAsync to check event status correctly (#251)
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class | Import Path | Example DAG |
---|---|---|
DataprocSubmitJobOperatorAsync |
from astronomer.providers.google.cloud.operators.dataproc import DataprocSubmitJobOperatorAsync |
Example DAG |
EmrContainerSensorAsync |
from astronomer.providers.amazon.aws.sensors.emr import EmrContainerSensorAsync |
Example DAG |
EmrStepSensorAsync |
from astronomer.providers.amazon.aws.sensors.emr import EmrStepSensorAsync |
Example DAG |
EmrJobFlowSensorAsync |
from astronomer.providers.amazon.aws.sensors.emr import EmrJobFlowSensorAsync |
Example DAG |
LivyOperatorAsync |
from astronomer.providers.apache.livy.operators.livy import LivyOperatorAsync |
Example DAG |
- Improved example DAGs so that resource creation and clean up is handled during system tests rather than doing it manually
- Enhanced the Async Databricks Operator to persist
run_id
andrun_page_url
inXCom
(#175)
This release adds the following 7 new async sensors/operators:
Operator/Sensor Class | Import Path | Example DAG |
---|---|---|
S3KeySizeSensorAsync |
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySizeSensorAsync |
Example DAG |
S3KeysUnchangedSensorAsync |
from astronomer.providers.amazon.aws.sensors.s3 import S3KeysUnchangedSensorAsync |
Example DAG |
S3PrefixSensorAsync |
from astronomer.providers.amazon.aws.sensors.s3 import S3PrefixSensorAsync |
Example DAG |
GCSObjectsWithPrefixExistenceSensorAsync |
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectsWithPrefixExistenceSensorAsync |
Example DAG |
GCSObjectUpdateSensorAsync |
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectUpdateSensorAsync |
Example DAG |
GCSUploadSessionCompleteSensorAsync |
from astronomer.providers.google.cloud.sensors.gcs import GCSUploadSessionCompleteSensorAsync |
Example DAG |
BigQueryTableExistenceSensorAsync |
from astronomer.providers.google.cloud.sensors.bigquery import BigQueryTableExistenceSensorAsync |
Example DAG |
The dependencies for installing this repo are now split into multiple extras as follows (#113)
Extra Name | Installation Command | Dependencies |
---|---|---|
all |
pip install 'astronomer-providers[all]' |
All providers |
amazon |
pip install 'astronomer-providers[amazon]' |
Amazon |
cncf.kubernetes |
pip install 'astronomer-providers[cncf.kubernetes]' |
Kubernetes |
databricks |
pip install 'astronomer-providers[databricks]' |
Databricks |
google |
pip install 'astronomer-providers[google]' |
Google Cloud |
http |
pip install 'astronomer-providers[http]' |
HTTP |
snowflake |
pip install 'astronomer-providers[snowflake]' |
Snowflake |
This will allow users to just install dependencies of a single provider. For example, if a user
wants to just use KubernetesPodOperatorAsync
, they should not need to install GCP, AWS or
Snowflake dependencies by running pip install 'astronomer-providers[cncf.kubernetes]'
.
- Fixes a bug on the Async Databricks Triggerer failing due to malformed authentication header along with improved exception handling to send the Triggerer errors back to the worker to understand why a particular job execution has failed. (#147)
- Initial release, with the following 18 Async Operators/Sensors:
Operator/Sensor Class | Import Path | Example DAG |
---|---|---|
RedshiftSQLOperatorAsync |
from astronomer.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperatorAsync |
Example DAG |
RedshiftPauseClusterOperatorAsync |
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftPauseClusterOperatorAsync |
Example DAG |
RedshiftResumeClusterOperatorAsync |
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftResumeClusterOperatorAsync |
Example DAG |
RedshiftClusterSensorAsync |
from astronomer.providers.amazon.aws.sensors.redshift_cluster import RedshiftClusterSensorAsync |
Example DAG |
S3KeySensorAsync |
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync |
Example DAG |
KubernetesPodOperatorAsync |
from astronomer.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperatorAsync |
Example DAG |
ExternalTaskSensorAsync |
from astronomer.providers.core.sensors.external_task import ExternalTaskSensorAsync |
Example DAG |
FileSensorAsync |
from astronomer.providers.core.sensors.filesystem import FileSensorAsync |
Example DAG |
DatabricksRunNowOperatorAsync |
from astronomer.providers.databricks.operators.databricks import DatabricksRunNowOperatorAsync |
Example DAG |
DatabricksSubmitRunOperatorAsync |
from astronomer.providers.databricks.operators.databricks import DatabricksSubmitRunOperatorAsync |
Example DAG |
BigQueryCheckOperatorAsync |
from astronomer.providers.google.cloud.operators.bigquery import BigQueryCheckOperatorAsync |
Example DAG |
BigQueryGetDataOperatorAsync |
from astronomer.providers.google.cloud.operators.bigquery import BigQueryGetDataOperatorAsync |
Example DAG |
BigQueryInsertJobOperatorAsync |
from astronomer.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperatorAsync |
Example DAG |
BigQueryIntervalCheckOperatorAsync |
from astronomer.providers.google.cloud.operators.bigquery import BigQueryIntervalCheckOperatorAsync |
Example DAG |
BigQueryValueCheckOperatorAsync |
from astronomer.providers.google.cloud.operators.bigquery import BigQueryValueCheckOperatorAsync |
Example DAG |
GCSObjectExistenceSensorAsync |
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectExistenceSensorAsync |
Example DAG |
HttpSensorAsync |
from astronomer.providers.http.sensors.http import HttpSensorAsync |
Example DAG |
SnowflakeOperatorAsync |
from astronomer.providers.snowflake.operators.snowflake import SnowflakeOperatorAsync |
Example DAG |