v6.3.0

6.3.0 (2024-10-24)

Features

• Add support for -comp-fields=*, which also adds support for --exclude-columns (#1291) (f181377)
• Added ORC and PARQUET support among source tests (#1298) (e102d1e)
• Partition Custom Queries (#1289) (7e947d5)
• Support row validation of boolean columns (#1284) (c1de5f3)

Bug Fixes

• Add proper handling for GCPSecretManager exceptions (#1285) (8028836)
• Combiner did not take boolean datatypes into account (#1290) (84824db)
• Correctly cater for Oracle FLOAT in custom-query validations (#1304) (fbe9d23)

v6.2.0

6.2.0 (2024-09-27)

Features

• Enhance --tables-list to support schema.* shorthand for find-tables command (#1265) (cbd9bbb)

Bug Fixes

• big query run id message (#1263) (09659e0)
• cast count to bigint when generating partitions (#1281) (a2861ca)
• Explicit set the parameter for line terminator on converting datafram… (#1269) (b20b2b5), closes #1248
• Oracle validations of tables with hyphen in name (#1245) (4adaa79)
• Set RAND() as default for DB2's random function (#1246) (d01a40b)
• Update Teradata rtrim() to account for tabs and other whitespace (#1273) (db7e9d3)
• Using datetime constants in filters (#1259) (44863bd)

Documentation

• Add comments on both verbose and log-level parameters, also alter the… (#1257) (35caf98)

v6.1.0

6.1.0 (2024-08-27)

Features

• Add Oracle TIMESTAMP WITH LOCAL TIME ZONE support (#1238) (1e9f458)
• Add TIME data type support for Teradata, BigQuery and SQL Server (#1229) (ab7007b)
• Adding exclude_columns flag for row validation hash (#1243) (a1fd616)
• Auto split row concat/hash validations when many columns (#1233) (ae9b72d)
• BigQuery result handler logs textual output at DEBUG instead of INFO (#1240) (a9aafa2)

Bug Fixes

• Adds automatic RTRIM for custom query row validations for Teradata string comparison fields (#1230) (5c1a2be)
• Bug for generate partitions with dates with different column names for PKs (#1231) (5f51653)
• Fixing multiple function call, to get schema in custom-query validation for Hive (#1180) (a584e5b)
• support generate partitions for one row per partition (#1241) (4099a29)
• TD to BQ - Support hash row validation with Latin and Unicode Characters in Strings (#1226) (e1b24ef)
• TD to BQ generate partitions with date PKs (#1220) (a03f3b0)

Documentation

• Update Cloud Functions sample to support Cloud Scheduler job triggering (#1228) (3a7c164)

v6.0.0

6.0.0 (2024-08-06)

⚠ BREAKING CHANGES

• Failing validations of strings with extended ascii characters (#1184)

Features

• Add TLS options to PostgreSQL connections add (#1199) (5506680)
• Enhance PostgreSQL connections add documentation (#1200) (f179f5a)
• generate-table-partitions write multiple partitions to a single yaml file (#1178) (59a9a18)

Bug Fixes

• Apply RTRIM on string column when generating partitions with -tsp (#1182) (9dcaad1)
• Close source and target connections after executing a validation (#1197) (3dc9fa7)
• Ensure BigQuery queries are executed in UTC (#1174) (a77cdd9)
• Ensure Teradata OutofBound dates don't affect other date columns (#1219) (e02609d)
• Failing validations of strings with extended ascii characters (#1184) (c8ac146)
• Issue with GoogleSQL regarding dates before 1000CE (#1186) (6107d9c)
• More robust exit on DataValidation() (#1206) (13ec3ee)
• Oracle cast(decimal to string) caters for non-zero values <1 (#1204) (fa44466)
• Oracle INTERVAL exception in validate schema (#1215) (f133f73)
• Support char comparison fields in Teradata (#1203) (dc19580)
• Support DATE '0001-01-01' comparison fields (#1208) (cca0c4f)

Documentation

• Fix some typos in Cloud Run readme (#1188) (c8cd0de)
• Tweaks to Cloud Functions sample (#1176) (7a49a95)

v5.1.1

5.1.1 (2024-06-12)

Documentation

• Update generate-partitions flags (#1168) (5a747c2)

v5.1.0

5.1.0 (2024-06-11)

Features

• Add a workaround for a Snowflake IN list limitation (#1152) (16b979e)
• Support --trim-string-pks flag for padded string semantics (#1166) (a81f396)
• Support GCS custom query files (#1155) (e3fe3d1)

Bug Fixes

• Fixes bug in get_max_in_list_size (#1158) (973e6b6)
• Removing t0 alias from column name, while getting schema from query. Adding Integration test for Hive Custom-Query (#1164) (74a14af)
• Support PKs with different casing for generate-partitions (#1142) (021ce75)
• Update to support up to 10K partitions (#1139) (210c352)

v5.0.0

5.0.0 (2024-05-21)

⚠ BREAKING CHANGES

• Support for GCS config paths decoupled from environment variables (#1129)
• Filters not working correctly in Snowflake (#1126)

Features

• Add support for random row sampling on binary id columns (#1135) (c3d2155)
• Control Teradata decimal format when cast to string (#1138) (e68e2a6)
• Support for GCS config paths decoupled from environment variables (#1129) (72e41b7)

Bug Fixes

• Filters not working correctly in Snowflake (#1126) (9845643)
• Fix casting from binary to string on Snowflake & BigQuery (#1113) (4f5ae81)
• Issue 1127 configs dir fails with more than 40 files (#1130) (15c81cf)
• Teradata's ValueError after large timestamp epoch second handling (#1121) (ee8d6da)

Documentation

• Add custom-query code snippet for Cloud Run sample documentation (#1124) (93bb64f)
• Distributed DVT Cloud Run Jobs sample (#1133) (f51f327)

v4.5.0

4.5.0 (2024-03-18)

Features

• Support GCS files in configs list command (#1108) (b49e1c3)

Bug Fixes

• Add table names to report results when source and/or target dataframes are empty (#1104) (812ed62)
• Fixes issue casting Snowflake decimal with scale>0 to string (#1110) (34446a4)
• force cast for aggregates (#1114) (44b60cf)
• Teradata large timestamp handling (#1117) (842d8b7)

v4.4.0

4.4.0 (2024-02-22)

Features

• Add --url to Oracle connections add options (#1083) (2f078c2)
• Add PostgreSQL OID support (#1076) (58f8fcb)
• Add support to generate a JSON config file only for applications purposes (#1089) (d463038)
• set default oracle sql alchemy arraysize to 500 (#1088) (1672ac5)
• Support for Kubernetes (#1058) (fdbdbe0)

Bug Fixes

• Add support for cx_Oracle's DB_TYPE_LONG_RAW (#1095) (90547ef)
• Better casts to string for binary floats/doubles (#1078) (15bfc4c)
• case-insensitive comparison field support (#1103) (d28786f)
• Fix merge issue for Teradata empty dataframes (#1100) (cc91fa2)
• increase upper limit on recursion columns (#1090) (c599ebf)
• Remove DDL automatically issued by Ibis for Postgres connections (#1067) (c2b660b)
• Row validation primary key columns >64bit int/float are cast to string (#1080) (9e70e9e)
• Spanner generate-partition to use BQ dialect (#1066) (f3cc565)
• spanner hash function to return string instead of bytes (#1062) (722dff9)

Documentation

• Add Airflow Kubernetes pod operator samples (#1087) (7d5ea91)
• Updates on nested column limitations, contributing guide examples and incorrect example (#1082) (cc0f60a)

v4.3.0

4.3.0 (2023-11-28)

Features

• Adding Exclude columns flag for aggregations in column validations (#961) (faa32dc)
• support query parameter for MSSQL connection (#1026) (48b0355)

Bug Fixes

• --dry-run for SQLAlchemy clients with valid raw SQL (#1047) (c1e0e34)
• Add Spanner RawSQL operation to enable filtering (#1054) (3a01503)
• Adding credentials as parameter for Spanner (#1031) (367658e)
• Adjust find-tables to properly get Oracle and Postgres schemas (#1034) (45fb40a)
• Cast should treat nullable and non-nullables as the same (#1037) (5e5c5eb)
• Fix --grouped-columns issue for Oracle validation (#1050) (3473a27)
• Fix decimal separator to "." (dot) on Oracle (#1042) (14cc7ef)
• Teradata SSLMODE issue fix (#1014) (e7aab6b)

Documentation

• Add CLOB to Oracle BLOB validation document (#1029) (8c76c1b)
• Update connections.md to add supported version of DB2 (#1030) (44b4be7)