Automatically provision a Prefect Cloud account with resources for the debug tutorial #2
Conversation
…or the debugging tutorial
|
We'll need to update the debug tutorial to include instructions for starting workers (since this script automatically kills those once the initial runs are complete). |
…ince it's not needed
There was a problem hiding this comment.
@daniel-prefect here are a few high level comments/questions.
| @@ -0,0 +1,32 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
What do you think about converting this script to Python? We can safely assume our users are familiar with Python whereas bash syntax might be new. You can then also use Prefect's built in conveniences for things like getting the active profile.
There was a problem hiding this comment.
I'm down with converting this to Python if it simplifies the script. I started with Terraform, and then wrapped it in a bash script when I realized it didn't do everything I needed.
| @@ -0,0 +1,38 @@ | |||
| terraform { | |||
setup_env.sh
Outdated
| # Check if Python is installed and determine the Python command | ||
| echo "🐍 Checking if Python is installed..." | ||
| if command -v python3 &> /dev/null; then | ||
| PYTHON_CMD="python3" | ||
| elif command -v python &> /dev/null; then | ||
| PYTHON_CMD="python" | ||
| else | ||
| echo "❌ Error: Python is not installed. Please install Python 3.9 or higher and try again." | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Verify Python version is 3.9 or higher | ||
| if ! $PYTHON_CMD -c "import sys; assert sys.version_info >= (3, 9), 'Python 3.9 or higher is required'" &> /dev/null; then | ||
| echo "❌ Error: Python 3.9 or higher is required. Found $($PYTHON_CMD --version)" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "✅ Python $(${PYTHON_CMD} --version) is installed" | ||
|
|
||
| ############################################################################### | ||
| # Set up virtual environment | ||
| ############################################################################### | ||
|
|
||
| # Create and activate virtual environment | ||
| echo "🌟 Setting up Python virtual environment..." | ||
| $PYTHON_CMD -m venv temp_venv | ||
| source temp_venv/bin/activate | ||
|
|
||
| # Install requirements | ||
| echo "📦 Installing Python packages..." | ||
| pip install -r requirements.txt |
| # Create default work pool in staging workspace | ||
| resource "prefect_work_pool" "staging_default" { | ||
| name = "default-work-pool" | ||
| workspace_id = prefect_workspace.staging.id | ||
| type = "docker" | ||
| } | ||
|
|
||
| # Create default work pool in production workspace | ||
| resource "prefect_work_pool" "production_default" { | ||
| name = "default-work-pool" | ||
| workspace_id = prefect_workspace.production.id | ||
| type = "docker" | ||
| } |
There was a problem hiding this comment.
What do you think about picking a Cloud based work pool, for example GCP Cloud Run? There would be more infrastructure but it would also be a more real world example.
| account_id = var.prefect_account_id | ||
| } | ||
|
|
||
| # Create staging workspace |
There was a problem hiding this comment.
If we want to get a bit fancier we should separate these into modules.
| echo "🔑 Reading Prefect API key and account ID..." | ||
|
|
||
| ############################################################################### | ||
| # Provision Prefect Cloud resources |
There was a problem hiding this comment.
In practice a big part of this kind of setup work is creating all the related objects: blocks, variables, automations, etc.
It would be great to show an example of creating a few blocks as well as a simple "notify on failure" automation. I believe that can all be expressed in both Terraform and Python.
(Please suggest better ways to accomplish this, I'm just being a script kiddie :-) )
Run
./setup_envto provision a paid Prefect Cloud account with the following resources:productionandstaging)stagingworkspace)The user can then complete the Debug a data pipeline tutorial without having to first manually perform all tasks in the Set up a platform for data pipelines tutorial.
NOTE: This script doesn't work on personal Prefect accounts because it requires multiple workspaces. There's a check which fails the script if you try to use it with a personal account.