[Databricks E2E] Rzarka/882 synapse pause resume #1060
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
raafatzarka
added
documentation
raafatzarka
requested review from
bsherwin,
elenaterenzi,
chuckheinzelman,
ydaponte and
LiliamLeme
e2e_samples/parking_sensors/utilities/README_pause_resume_synapse.md
Outdated
Show resolved
Hide resolved
e2e_samples/parking_sensors/utilities/README_pause_resume_synapse.md
Outdated
Show resolved
Hide resolved
…atic module installation
ydaponte
requested changes
Jan 28, 2025
e2e_samples/parking_sensors/utilities/README_pause_resume_synapse.md
Outdated
Show resolved
Hide resolved
e2e_samples/parking_sensors/utilities/README_pause_resume_synapse.md
Outdated
Show resolved
Hide resolved
e2e_samples/parking_sensors/utilities/README_pause_resume_synapse.md
Outdated
Show resolved
Hide resolved
…e and improved logging for state transitions
ydaponte
requested changes
Jan 30, 2025
…esourceGroups parameter
ydaponte
approved these changes
Jan 31, 2025
raafatzarka
merged commit dbbf64b
into
e2e/databricks/parking-sensors-V1
3 checks passed
elenaterenzi
added a commit
that referenced
this pull request
Feb 6, 2025
* [E2E Databricks] - Automate region vm + dbx type cluster in the region (#988) * automate region vm + dbx type cluster in the region * Update e2e_samples/parking_sensors/scripts/configure_databricks.sh Co-authored-by: Elena Terenzi <[email protected]> * filtering photon support --------- Co-authored-by: Elena Terenzi <[email protected]> * [Databricks E2E Sample] - bug fix dbx compute (#1006) * fix breaking change from dbx cli output * fix hyperlink check * Cluster config update (#1012) * Update clean_up.sh Adding the Cleanup for federal credentials which follows a certain order: Get the SP Objc ID of the Service Connection Delete federated credentials from the SP Object ID Delete the service connection * Update deploy_azdo_service_connections_azure.sh Service Connections are using the Workload Identity which comes from a configuration file. The cleanup in case of re-start of this script also follows a peculiar order: Get SP object if according to the Service Connection Get the federate credential according to the SP Delete federated credentials Delete the Service Connection * Update README.md changing the docs with the required permission * Update README.md rephrase the permissions needed for azdevops project * Update e2e_samples/parking_sensors/scripts/clean_up.sh Co-authored-by: Yennifer Santos <[email protected]> * Update e2e_samples/parking_sensors/scripts/deploy_azdo_service_connections_azure.sh Co-authored-by: Yennifer Santos <[email protected]> * Update README.md I adjust the order of the permissions * Update README.md adding know issues how to delete federate credentials * Update README.md identation was not correct * Fix The property "options" Warnings in Dashboard.bicep * Update deploy_azdo_service_connections_azure.sh adding the following changes: 1) Function for cleanup 2)Function for sleep 3)Remove log file * Update clean_up.sh Implement function for cleanup and function for sleep * Update common.sh Added cleanup_federate_credentials and wait_for_credentials used in the clean_up.sh and deploy_azdo_service_connection.sh * Update clean_up.sh rename wait_for_cleanup to wait_for_process * Update deploy_azdo_service_connections_azure.sh renamed wait_for_cleanup to wait_for_process * Fix typo in clean_up.sh log message typo: comtain to contain fxed * Update e2e_samples/parking_sensors/README.md added Elena's suggestion of rewrite Co-authored-by: Elena Terenzi <[email protected]> * Update common.sh remove the log for SP * Update deploy_azdo_service_connections_azure.sh Clean up of the log suppress update pipelines message remove response from logging * Update common.sh changing the log for federate credentials removal * Update common.sh adding the SP in the log message * Update deploy_azdo_service_connections_azure.sh checking if the file exist before removal. * Update e2e_samples/parking_sensors/scripts/common.sh Co-authored-by: Elena Terenzi <[email protected]> * Update e2e_samples/parking_sensors/scripts/common.sh Co-authored-by: Elena Terenzi <[email protected]> * Update e2e_samples/parking_sensors/scripts/common.sh Co-authored-by: Elena Terenzi <[email protected]> * Update e2e_samples/parking_sensors/scripts/deploy_azdo_service_connections_azure.sh Co-authored-by: Elena Terenzi <[email protected]> * Update e2e_samples/parking_sensors/scripts/deploy_azdo_service_connections_azure.sh Co-authored-by: Elena Terenzi <[email protected]> * Update e2e_samples/parking_sensors/scripts/clean_up.sh Co-authored-by: Elena Terenzi <[email protected]> * Update e2e_samples/parking_sensors/scripts/common.sh Co-authored-by: Elena Terenzi <[email protected]> * Update e2e_samples/parking_sensors/scripts/common.sh Co-authored-by: Elena Terenzi <[email protected]> * Remove unnesissary dependsOn in main.bicep * Remove redundant comment and variable role unecessary * Checking if I can solve the conflicts * conflict * conflict * The new file after tryto solve the conflict * Trying again to solve the conflict * Suppress output for service connection deletion adding -0 none to resolve conflict * add code to delete app registrations (#1019) * [Databricks E2E] [BUG] cleanup fix bug with federated credential removal (#1047) * remove federated credentials only when needed * fix typo * [Databricks E2E]: Remove AzDO Service Connection Principal (#1057) * remove SPN for AzDO service connection * minor edits * minor edits * minor typo * add readme change for manual deletion * Deploy.sh will have the option to choose deployment type (#1049) * Deploy.sh will have the option to deploy as: 1) dev 2) Dev and Stage 3) Dev, Stage and Prod Common.sh will heold the function for this changes envtemplate the new environmnent variable for the deployment options deploy_azdo_pipelines.sh will deploy the release pipeline if option 2 or 3 are choosen which implies the deploy of t least 2 environmnents * adjusting the readme according to the latest version in v1 * Seems this session was not veru well documented - **Workaround - Manually Deleting Federated Credentials:** * to* typo * Update e2e_samples/parking_sensors/README.md Co-authored-by: Copilot <[email protected]> * Remove duplicate deployment instruction in README fixing copilot fix * Adding if for deploy.sh respect configuration in the enviromnent variable Initialize env variable in the init_environment * adjust the readme for env. varaible configuration * Clarify deployment options and environment variable usage Adjusting explanations about env variable in the readme and envtemplate * Update workaround for deleting Azure DevOps Service Connection fixing conflict * Refactor cleanup function to delete service principal fix the conflict --------- Co-authored-by: Copilot <[email protected]> * [Databricks E2E] Redesign CI CD Process diagrams with drawio (#1056) * Recreates the CI CD process diagrams in drawio * Remove the ADF publish step from CI CD process diagrams * Update the README.md to reflect the changes in the process sequence and diagram files --------- Co-authored-by: Jose Perales <[email protected]> * update SQL Pool definition to avoid resizing on redeployment (#1066) * [Databricks E2E] Fix devcontainer for wheel library builds (#1040) * Fix devcontainer for wheel library builds * env file required for devcontainer * Update e2e_samples/parking_sensors/src/ddo_transform/.devcontainer/Dockerfile * [E2E Databricks] - fix clusterId and release folders (#1064) * fix: release paths and cluster id * fix: adding azdo file * fix: removing unnecessary config * update: readMe, optimize code flow * update: upgrade DB CLI and fix release pipeline * fix: pr comments * fix: py extension on adf pipeline (#1072) * [Databricks E2E] Rzarka/882 synapse pause resume (#1060) * Add PowerShell and Bash scripts for Synapse Pause/Resume * Support Manual and Automated Synapse Pause PowerShell Script with Additional features * Add README file for Synapse Pause script * Add References section to Synapse Pause README file * Add InstallModules parameter to pause_resume_synapse script for automatic module installation * Enhance README and script usage documentation for pause_resume_synapse and improved logging for state transitions * Update README_pause_resume_synapse to include example values for parameters * Update README_pause_resume_synapse to provide specific examples for ResourceGroups parameter * small fix --------- Co-authored-by: Yennifer Santos <[email protected]> Co-authored-by: Ayman El-Ghazali <[email protected]> Co-authored-by: LiliamLeme <[email protected]> Co-authored-by: Raafat Zarka <[email protected]> Co-authored-by: Liliam Leme <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Jose Perales <[email protected]> Co-authored-by: Jose Perales <[email protected]> Co-authored-by: Brian Sherwin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Type of PR
Purpose
This PR introduces the addition of a PowerShell script (pause_resume_synapse.ps1) and its accompanying README file. These files provide the functionality and documentation for pausing or resuming Azure Synapse SQL Pools and Azure SQL Data Warehouse. It supports both Azure Automation (Managed Identity) and local manual execution.
Does this introduce a breaking change? If yes, details on what can break
Author pre-publish checklist
Validation steps
Issues Closed or Referenced