[Databricks E2E] Changes to zip REST API and Integrate with ADF.

[bsherwin]

Type of PR

• Documentation changes
• Code changes
• Test changes

Purpose

This PR is to integrate the REST API into the Azure Data Factory process. We removed the binary ZIP package for deployment and build that as part of the deployment process.

In addition to modifying the Linked Services to use the new REST endpoints, we also modified the Databricks activity to use the ddo_cluster that was set up during the deployment. This allows the cluster to be shared rather than spinning up a cluster for each databricks task to "standardize" and "transform" the data.

Author pre-publish checklist

• No PII in logs
• Made corresponding changes to the documentation

Validation steps

• Run a deployment from the devcontainer
• In ADF, navigate to the Linked Services and validate that the Databricks linked service correctly references the ddo_cluster
• Validate that the ADF pipeline "Copy Data" activities correctly point to the correct REST Endpoints
• Validate that the ADF pipeline "Databricks" activities correctly point to the correct binaries and are correctly set to a Wheel file.

Issues Closed or Referenced

• Closes Integrate Data generation into the MDW solution - Batch #829
• Closes Create Alternative to public REST API #764

[bsherwin]

@ydaponte pointed out that since we are using the existing cluster, we don't need the wheel as an appended library in the ADF Pipeline. I removed it as unnecessary.

[ydaponte]

Suggesting some changes - there are 2 things that won't work. The shebang is missing on your app service bash script and the CI pipeline deploy-adf-job is going to fail because is referring to an old field of the databricks linked service definition. Thanks for the great work so far!

[elenaterenzi]

in order to approve I need to finish testing, but I have some comments and figured it was worth sharing now in case you have time.
Some are not that important or are just questions, others would be changes that I would request you implement in order for me to approve.

[elenaterenzi]

found issues with idempotency

[ydaponte]

Requesting a couple of changes

[ydaponte]

Very minor changes.

[ydaponte]

Great Job! Thanks for the patience and effort!