Python SDK 1.5.3

Bug fix:

• Support using SQL operators (run_raw_sql, transform, dataframe)
  to convert a Pandas dataframe into a table when using a DuckDB in-memory
  database. #1831

Thanks to @pgzmnk for reporting the issue!

1.5.2

Improvements

• Restore pandas load option classes - PandasCsvLoadOptions, PandasJsonLoadOptions, PandasNdjsonLoadOptions and PandasParquetLoadOptions #1795

1.5.1

Improvements

• Add Openlineage facets for Microsoft SQL server. #1752

Bug fixes

• Use use_native_support param in load_file operator for table creation. #1756
• Resolved pandas-gbq dependency issue. #1768
• Fix Minio support for Snowflake. #1767
• Add handler param in database.run_sql() method 1773

1.5.0

Feature:

• Add support for Microsoft SQL server. #1538
• Add support for DuckDB. #1695
• Add result_format and fail_on_empty params to run_raw_sql operator #1584
• Add support validation_mode as part of the COPY INTO command for snowflake. #1689
• Add support for native transfers for Azure Blob Storage to Snowflake in LoadFileOperator. #1675

Improvements

• Use cache to reduce redundant database calls #1488
• Remove default copy_options as part of SnowflakeLoadOptions. All copy_options are now supported as part of SnowflakeLoadOptions as per documentation. #1689
• Remove load_options from File object. #1721
• Consolidated PandasCsvLoadOptions, PandasJsonLoadOptions, PandasNdjsonLoadOptions and PandasParquetLoadOptions to single PandasLoadOptions. #1722
• Render SQL code with parameters in BaseSQLDecoratedOperator. #897

Bug fixes

• Fix handling of multiple dataframes in the run_raw_sql operator. #1700

Docs

• Add documentation around Microsoft SQL support with example DAG. #1538
• Add documentation around DuckDB support with example DAG. #1695
• Add documentation for validation_mode as part of the COPY INTO command for snowflake. #1689
• Add documentation and example DAGs for snowflake SnowflakeLoadOptions for various available options around copy_options and file_options. #1689
• Fix the documentation to run the quickstart example described in the Python SDK README. #1716

Misc

• Add cleanup DAG to clean snowflake tables created as part of CI when the runners fail as part of GitHub actions. #1663
• Run example DAGs on astro-cloud and collect the results. #1499

1.5.0.a1

Feature:

• Add load_options param to accept pandasOptions in load_file operator #1466
• Add result_format and fail_on_empty params to run_raw_sql operator #1584

Improvements:

• Use cache to reduce redundant database calls #1488

1.4.1

Feature:

• Implement check_table Operator to validate data quality at table level #1239
• Add check_column Operator to validate data quality for columns in a table/dataframe #1239

Bug fixes

• Support "s3" conn type for S3Location #1647

Docs

• Add the documentation and example DAG for Azure blob storage #1598
• Fix dead link in documentation #1596
• Update README with newly supported location and database #1596
• Update configuration reference for XCom #1646
• Add step to generate constraints in Python SDK release process #1474
• Add document to showcase the use of check_table and check_column operators #1631

Misc

• Install google-cloud-sdk-gke-gcloud-auth-plugin in benchmark CI job #1557
• Pin sphinx-autoapi==2.0.0 version for docs build #1609

1.4

Feature:

• Support SFTP as file location docs #1481
• Support FTP as file location docs #1482
• Add support for Azure Blob Storage (only non-native implementation) #1275, #1542
• Add databricks delta table support docs #1352, #1397, #1452, #1476, #1480, #1555
• Add sourceCode facet to aql.dataframe() and aql.transform() as part of OpenLineage integration #1537
• Enhance LoadFileOperator so that users can send pandas attributes through PandasLoadOptions docs #1466
• Enhance LoadFileOperator so that users can send Snowflake specific load attributes through SnowflakeLoadOptions docs #1516
• Expose get_file_list_func to users so that it returns iterable File list from given destination file storage #1380

Improvements

• Deprecate export_table_to_file in favor of export_to_file (ExportTableToFileOperator and export_table_to_file operator would be removed in astro-python-sdk 1.5.0) #1503

Bug fixes

• LoadFileOperator operator checks for conn_type and conn_id provided to File #1471
• Generate constraints on releases and pushes (not PRs) #1472

Docs

• Change export_file to export_table_to_file in the documentation #1477
• Enhance documentation to describe the new Xcom requirements from Astro SDK 1.3.3 and airflow 2.5 #1483
• Add documentation around LoadOptions with example DAGs #1567

Misc

• Refactor snowflake merge function for easier maintenance #1493

1.3.3

Bug fixes

• Disable Custom serialization for Back-compat #1453
• Use different approach to get location for Bigquery tables #1449

Full Changelog: 1.3.2...1.3.3

1.3.3b2

Bug fixes

• Disable Custom serialization for Back-compat #1453
• Use different approach to get location for Bigquery tables #1449

1.3.3b1

Bug fixes

• Disable Custom serialization for Back-compat #1453

Full Changelog: 1.3.2...1.3.3b1