1.0.0

Summary

Features

• Improved the performance of aql.load_file by supporting database-specific (native) load methods.
  This is now the default behaviour. Previously, the Astro SDK Python would always use Pandas to load files to
  SQL databases which passed the data to worker node which slowed the performance. (#557, #481)

  Introduced new arguments to aql.load_file:

  • use_native_support for data transfer if available on the destination (defaults to use_native_support=True)
  • native_support_kwargs is a keyword argument to be used by method involved in native support flow.
  • enable_native_fallback can be used to fall back to default transfer(defaults to enable_native_fallback=True).

  Now, there are three modes:

  • Native: Default, uses Bigquery Load Job in the
    case of BigQuery and Snowflake COPY INTO
    using external stage in the case of Snowflake.
  • Pandas: This is how datasets were previously loaded. To enable this mode, use the argument
    use_native_support=False in aql.load_file.
  • Hybrid: This attempts to use the native strategy to load a file to the database and if native strategy(i)
    fails , fallback to Pandas (ii) with relevant log warnings. #557

• Allow users to specify the table schema (column types) in which a file is being loaded by using table.columns.
  If this table attribute is not set, the Astro SDK still tries to infer the schema by using Pandas
  (which is previous behaviour).#532

• Add Example DAG for Dynamic Map Task with Astro-SDK.
  #377,airflow-2.3.0

Breaking Change

• The aql.dataframe argument identifiers_as_lower (which was boolean, with default set to False)
  was replaced by the argument columns_names_capitalization (string within possible values
  ["upper", "lower", "original"], default is lower).#564
• The aql.load_file before would change the capitalization of all column titles to be uppercase, by default,
  now it makes them lowercase, by default. The old behaviour can be achieved by using the argument
  columns_names_capitalization="upper". #564
• aql.load_file attempts to load files to BigQuery and Snowflake by using native methods, which may have
  pre-requirements to work. To disable this mode, use the argument use_native_support=False in aql.load_file.
  #557, #481
• aql.dataframe will raise an exception if the default Airflow XCom backend is being used.
  To solve this, either use an external XCom backend, such as S3 or GCS
  or set the configuration AIRFLOW__ASTRO_SDK__DATAFRAME_ALLOW_UNSAFE_STORAGE=True. #444
• Change the declaration for the default Astro SDK temporary schema from using AIRFLOW__ASTRO__SQL_SCHEMA
  to AIRFLOW__ASTRO_SDK__SQL_SCHEMA #503
• Renamed aql.truncate to aql.drop_table #554

Bug fixes

• Fix missing airflow's task terminal states to CleanupOperator #525
• Allow chaining aql.drop_table (previously truncate) tasks using the Task Flow API syntax. #554, #515

Enhancements

• Improved the performance of aql.load_file for files for below:
  • From AWS S3 to Google BigQuery up to 94%. #429, #568
  • From Google Cloud Storage to Google BigQuery up to 93%. #429, #562
  • From AWS S3/Google Cloud Storage to Snowflake up to 76%. #430, #544
  • From GCS to Postgres in K8s up to 93%. #428, #531
• Get configurations via Airflow Configuration manager. #503
• Change catching ValueError and AttributeError to DatabaseCustomError #595
• Unpin pandas upperbound dependency #620
• Remove markupsafe from dependencies #623
• Added extend_existing to Sqla Table object #626
• Move config to store DF in XCom to settings file #537
• Make the operator names consistent #634
• Use exc_info for exception logging #643
• Update query for getting bigquery table schema #661
• Use lazy evaluated Type Annotations from PEP 563 #650
• Provide Google Cloud Credentials env var for bigquery #679
• Handle breaking changes for Snowflake provide version 3.2.0 and 3.1.0 #686

What's Changed (Full Changelog)

• Get configs via Airflow Configuration manager by @utkarsharma2 in #505
• Load files from GCS to Bigquery using BigqueryHook by @utkarsharma2 in #489
• Fix benchmark permissions by @tatiana in #513
• Fix benchmark to work without table metadata by @tatiana in #514
• Add performance result to load_file to snowflake using Python SDK 0.11.0 by @sunank200 in #480
• Fix docs sidebar by @dimberman in #517
• Benchmark postgres by @dimberman in #510
• Rename BaseSQLOperator class by @dimberman in #518
• Read default config if airflow's isn't defined by @utkarsharma2 in #520
• Add Snowflake stage methods by @tatiana in #523
• Fix broken link to tutorial in README by @jlaneve in #526
• Simplify debugging issues when building docs by @tatiana in #527
• create a possible solution to users passing large dataframes between … by @dimberman in #522
• Use native path for S3 to Bigquery in load_file operator by @utkarsharma2 in #519
• Update links in Contribution Guidelines section in README by @josh-fell in #536
• Add CI job to check for dead links by @kaxil in #528
• Adjust storage integration so it is consistent for AWS and GCP by @tatiana in #539
• improve docs to @kaxil and @tatiana's comments by @dimberman in #521
• Refactor table creation in load_file by @tatiana in #538
• Add missing task terminal states to CleanUp Operator by @utkarsharma2 in #540
• Optimize postgres performance by @dimberman in #531
• Benchmark reporting to expose the information from GCS to Markdown by @sunank200 in #547
• Refactor db.load_file_to_table, make json config optional by @tatiana in #549
• Allow running tests on PRs from forks + label by @kaxil in #546
• Add native path from local to bigquery by @utkarsharma2 in #535
• Fix MyPY issue of 'path' and 'conn_id' property of Class File by @utkarsharma2 in #545
• Improve (Sphinx) gitignore by @tatiana in #548
• Handle nrows for export_to_dataframe() by @utkarsharma2 in #559
• Add benchmarking result from GCS to Bigquery after optimization by @sunank200 in #563
• Optimize Snowflake load_file using native COPY INTO by @tatiana in #544
• Fix DistutilsOptionError #570 by @tatiana in #571
• Add benchmarking results for S3 to Bigquery transfer by @utkarsharma2 in #568
• Rename aql.truncate to aql.drop_table and fix dependency issue by @dimberman in #556
• Implement fallback mechanism incase native support fails to default option by @sunank200 in #560
• Add 'columns_names_capitalization' param to control the cols case by @utkarsharma2 in #567
• Example Dynamic Map Task with Astro-SDK by @pankajastro in #575
• Update the benchmark results for PostGres on result.md by @sunank200 in #578
• Release 1.0.0b1 by @sunank200 in #576

New Contributors

• @josh-fell made their first contribution in #536

Full Changelog: 0.11.1...1.0.0