1.1.0b1
Pre-release
Pre-release
Features
-
Support for Datasets introduced in Airflow 2.4 (#786, #808)
-
inlets
andoutlets
will be automatically set for all the operators. -
Users can now schedule DAGs on
File
andTable
objects. Example:input_file = File( path="https://raw.githubusercontent.com/astronomer/astro-sdk/main/tests/data/imdb_v2.csv" ) imdb_movies_table = Table(name="imdb_movies", conn_id="sqlite_default") top_animations_table = Table(name="top_animation", conn_id="sqlite_default") START_DATE = datetime(2022, 9, 1) @aql.transform() def get_top_five_animations(input_table: Table): return """ SELECT title, rating FROM {{input_table}} WHERE genre1='Animation' ORDER BY rating desc LIMIT 5; """ with DAG( dag_id="example_dataset_producer", schedule=None, start_date=START_DATE, catchup=False, ) as load_dag: imdb_movies = aql.load_file( input_file=input_file, task_id="load_csv", output_table=imdb_movies_table, ) with DAG( dag_id="example_dataset_consumer", schedule=[imdb_movies_table], start_date=START_DATE, catchup=False, ) as transform_dag: top_five_animations = get_top_five_animations( input_table=imdb_movies_table, output_table=top_animations_table, )
-
-
Dynamic Task Templates: Tasks that can be used with Dynamic Task Mapping (Airflow 2.3+)
-
Create upstream_tasks parameter for dependencies independent of data transfers (#585)
Bug fixes
- Add response_size to run_raw_sql and warn about db thrashing (#815)