v0.4.0-beta

0.4.0-beta (2023-08-07)

Features

• 756 postgre sql to big query (#805) (121bf93)
• add bq_dataset_region parameter (a9647ad)
• Add ORACLE_SCHEMA to parameterize scripts (969e314)
• added init file (3273fe7)
• Added Paramterized script file (2951e3a)
• HiveToBiqquery parameterize script (2eec146)
• include logging (ca1f9be)
• include logging (2714b7f)
• include logging (cf0a81c)
• service account as env variable (9c93710)
• service account as env variable (57292f9)
• update nb constants & script name (cb07530)
• update nb with IS_PARAMETERIZED flag (b4dcd35)
• update notebook constants (5c9bdc2)
• update working dir (ea6a52b)
• update working dir for parameterization (5840c88)
• Updated notebook to handle parameters (08dd1c4)
• updated readme for parametrized script (82ff6ad)
• updated readme for parametrized script (afac2a6)
• updated run_notebook (c84883f)

Bug Fixes

• add log_output (09dc88a)
• black formatting (fcab888)
• correct pip install cmd (f9c17c5)
• correct project id variable (5c61b94)
• correct sqlachemy execute (14cb3d3)
• Deal with upgraded google_cloud_pipeline_components.experimental vs v1 (4551fab)
• Deal with upgraded google_cloud_pipeline_components.experimental vs v1 (2304179)
• deletes gcs folder created after all tests run (66e3000)
• except statement (b88758a)
• Fix merge conflict (deb3a43)
• Fix merge from main (5a75ae0)
• Fix merge from main (6ad8fe6)
• Fix merge from main (2dd4be5)
• Fix merge from main (fcc8ca6)
• Fix merge from main (90a65ec)
• ignored some tests in general template (ae74321)
• Implement de-duplication (b1c9f8a)
• implement de-duplication changes (1aeef92)
• implement de-duplication changes (02c2508)
• implement get_common_args function (6525dc5)
• implement get_env_vars in base script (0734fb7)
• integration test - change custom container image (#810) (8943508)
• make subnet optional variable (64c0e68)
• make subnet optional variable (294eea5)
• modified except block (8871b25)
• move max_parallelism from common_args (1f88912)
• Notebook MYSQLTABLE_LIST to MYSQL_TABLE_LIST (5abe159)
• Notebook MYSQLTABLE_LIST to MYSQL_TABLE_LIST (e22fb72)
• oracle table list parsing corrected (5f33fb4)
• oracle table list parsing corrected (7f1e913)
• raise exception on job failure (e5b1a99)
• remove common args from script (fc125f2)
• remove duplicate get_env_var (73bad48)
• remove log_level from nb_parameters (634d926)
• remove log_level from nb_parameters (fe49799)
• remove project id constant (e9173bb)
• removed duplicate arugment (e0c3262)
• removed pubsub output project parameter (24a9d18)
• removed pubsub output project parameter (838c23a)
• removed unused outputProjectID (5f106c4)
• removing version parameter while running with cluster (d9faf34)
• renamed init file (aa60431)
• Resolve conflicting changes (8ab7e0d)
• small fixes in notebook (155c5c8)
• update project id variable (97e58f1)
• update PROJECT_ID to PROJECT (1a06e52)
• update scipt_name with oracletopostgres (5dac5d4)
• update spark to 3.3.2 (60757dd)
• updated indentation (306e4e5)
• updated log_output (0b2a510)
• updated notebook with tag parameters (df007b4)
• updated notebook_constants (c4aaaa7)
• updated pom.xml to avoid commons-cli conflict in cluster mode (40cc8f7)
• updated postgres arg in notebook_constants (5900066)
• updated sql connection (8c5680d)
• updated sql query error for boolean value (a71e7f1)
• updated to UUID GCS staging location (a6fef9e)

Documentation

• added Oracle to Postgres notebook blogpost link to README files (281bde5)
• Change Dataflow to Dataproc (#815) (74cee01)
• Corrected typos in Snowflake README (78e9d42)
• Corrected typos in Snowflake README (ab60e0e)
• removed PubSubLiteToGCS from README files (a5d4c06)