Skip to content

Latest commit

 

History

History
315 lines (228 loc) · 14.1 KB

CHANGELOG.rst

File metadata and controls

315 lines (228 loc) · 14.1 KB

Changelog

1.3.0 (2022-05-09)

New Operators

This release adds the following 5 new async sensors/operators:

Operator/Sensor Class Import Path Example DAG
AzureDataFactoryRunPipelineOperatorAsync 
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperatorAsync
 Example DAG
AzureDataFactoryPipelineRunStatusSensorAsync 
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryPipelineRunStatusSensorAsync
 Example DAG
EmrContainerOperatorAsync 
from astronomer.providers.amazon.aws.operators.emr import EmrContainerOperatorAsync
 Example DAG
HivePartitionSensorAsync 
from astronomer.providers.apache.hive.sensors.hive_partition import HivePartitionSensorAsync
 Example DAG
NamedHivePartitionSensorAsync 
from astronomer.providers.apache.hive.sensors.named_hive_partition import NamedHivePartitionSensorAsync
 Example DAG

Improvements

  • Improved example DAGs so that minimal resources are created during integration tests
  • Fixes a bug on the DatabricksRunNowOperatorAsync to check event status correctly (#251)

1.2.0 (2022-04-12)

New Operators

This release adds the following 5 new async sensors/operators:

Operator/Sensor Class Import Path Example DAG
DataprocSubmitJobOperatorAsync 
from astronomer.providers.google.cloud.operators.dataproc import DataprocSubmitJobOperatorAsync
 Example DAG
EmrContainerSensorAsync 
from astronomer.providers.amazon.aws.sensors.emr import EmrContainerSensorAsync
 Example DAG
EmrStepSensorAsync 
from astronomer.providers.amazon.aws.sensors.emr import EmrStepSensorAsync
 Example DAG
EmrJobFlowSensorAsync 
from astronomer.providers.amazon.aws.sensors.emr import EmrJobFlowSensorAsync
 Example DAG
LivyOperatorAsync 
from astronomer.providers.apache.livy.operators.livy import LivyOperatorAsync
 Example DAG

Improvements

  • Improved example DAGs so that resource creation and clean up is handled during system tests rather than doing it manually
  • Enhanced the Async Databricks Operator to persist run_id and run_page_url in XCom (#175)

1.1.0 (2022-03-23)

New Operators

This release adds the following 7 new async sensors/operators:

Operator/Sensor Class Import Path Example DAG
S3KeySizeSensorAsync 
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySizeSensorAsync
 Example DAG
S3KeysUnchangedSensorAsync 
from astronomer.providers.amazon.aws.sensors.s3 import S3KeysUnchangedSensorAsync
 Example DAG
S3PrefixSensorAsync 
from astronomer.providers.amazon.aws.sensors.s3 import S3PrefixSensorAsync
 Example DAG
GCSObjectsWithPrefixExistenceSensorAsync 
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectsWithPrefixExistenceSensorAsync
 Example DAG
GCSObjectUpdateSensorAsync 
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectUpdateSensorAsync
 Example DAG
GCSUploadSessionCompleteSensorAsync 
from astronomer.providers.google.cloud.sensors.gcs import GCSUploadSessionCompleteSensorAsync
 Example DAG
BigQueryTableExistenceSensorAsync 
from astronomer.providers.google.cloud.sensors.bigquery import BigQueryTableExistenceSensorAsync
 Example DAG

Improvements

The dependencies for installing this repo are now split into multiple extras as follows (#113)

Extra Name Installation Command Dependencies
all pip install 'astronomer-providers[all]' All providers
amazon pip install 'astronomer-providers[amazon]' Amazon
cncf.kubernetes pip install 'astronomer-providers[cncf.kubernetes]' Kubernetes
databricks pip install 'astronomer-providers[databricks]' Databricks
google pip install 'astronomer-providers[google]' Google Cloud
http pip install 'astronomer-providers[http]' HTTP
snowflake pip install 'astronomer-providers[snowflake]' Snowflake

This will allow users to just install dependencies of a single provider. For example, if a user wants to just use KubernetesPodOperatorAsync, they should not need to install GCP, AWS or Snowflake dependencies by running pip install 'astronomer-providers[cncf.kubernetes]'.

Bug Fixes

  • Fixes a bug on the Async Databricks Triggerer failing due to malformed authentication header along with improved exception handling to send the Triggerer errors back to the worker to understand why a particular job execution has failed. (#147)

1.0.0 (2022-03-01)

  • Initial release, with the following 18 Async Operators/Sensors:
Operator/Sensor Class Import Path Example DAG
RedshiftSQLOperatorAsync 
from astronomer.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperatorAsync
 Example DAG
RedshiftPauseClusterOperatorAsync 
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftPauseClusterOperatorAsync
 Example DAG
RedshiftResumeClusterOperatorAsync 
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftResumeClusterOperatorAsync
 Example DAG
RedshiftClusterSensorAsync 
from astronomer.providers.amazon.aws.sensors.redshift_cluster import RedshiftClusterSensorAsync
 Example DAG
S3KeySensorAsync 
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync
 Example DAG
KubernetesPodOperatorAsync 
from astronomer.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperatorAsync
 Example DAG
ExternalTaskSensorAsync 
from astronomer.providers.core.sensors.external_task import ExternalTaskSensorAsync
 Example DAG
FileSensorAsync 
from astronomer.providers.core.sensors.filesystem import FileSensorAsync
 Example DAG
DatabricksRunNowOperatorAsync 
from astronomer.providers.databricks.operators.databricks import DatabricksRunNowOperatorAsync
 Example DAG
DatabricksSubmitRunOperatorAsync 
from astronomer.providers.databricks.operators.databricks import DatabricksSubmitRunOperatorAsync
 Example DAG
BigQueryCheckOperatorAsync 
from astronomer.providers.google.cloud.operators.bigquery import BigQueryCheckOperatorAsync
 Example DAG
BigQueryGetDataOperatorAsync 
from astronomer.providers.google.cloud.operators.bigquery import BigQueryGetDataOperatorAsync
 Example DAG
BigQueryInsertJobOperatorAsync 
from astronomer.providers.google.cloud.operators.bigquery import  BigQueryInsertJobOperatorAsync
 Example DAG
BigQueryIntervalCheckOperatorAsync 
from astronomer.providers.google.cloud.operators.bigquery import BigQueryIntervalCheckOperatorAsync
 Example DAG
BigQueryValueCheckOperatorAsync 
from astronomer.providers.google.cloud.operators.bigquery import BigQueryValueCheckOperatorAsync
 Example DAG
GCSObjectExistenceSensorAsync 
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectExistenceSensorAsync
 Example DAG
HttpSensorAsync 
from astronomer.providers.http.sensors.http import HttpSensorAsync
 Example DAG
SnowflakeOperatorAsync 
from astronomer.providers.snowflake.operators.snowflake import SnowflakeOperatorAsync
 Example DAG