-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
6 changed files
with
151 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
135 changes: 135 additions & 0 deletions
135
docs/content/dagster-plus/deployment/azure/acr-user-code.mdx
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,135 @@ | ||
# Deploying user code in Azure Container Registry (ACR) with Dagster+ | ||
|
||
This quickstart guide will walk you through setting up a new repository for your Dagster code, setting up CI/CD with GitHub Actions backed by Azure Container Registry (ACR), and deploying your code to your Azure Kubernetes Service (AKS) cluster. | ||
|
||
This guide assumes you already have an AKS agent running. You can follow along [here](/dagster-plus/deployment/azure/aks-agent) if you still need to set up an AKS agent. | ||
|
||
## Prerequisites | ||
|
||
This guide will use a Github repository to store the Dagster code, and GitHub Actions to deploy the code to Azure Container Registry - but you should be able to adapt these steps to other version control systems and CI/CD tools. | ||
|
||
- The azure CLI installed on your machine. You can download it [here](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli). | ||
- A GitHub account, and the ability to run GitHub Actions workflows in a repository. | ||
|
||
## Step 1: Creating a repository for Dagster code. | ||
|
||
We'll create a new repository based on the [Dagster+ hybrid quickstart repository](https://github.com/dagster-io/dagster-cloud-hybrid-quickstart). We'll go through these steps using a brand new repository in GitHub, but you should be able to easily adapt these steps to an existing repository or other version control systems. | ||
|
||
First, [create a new repository in GitHub](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-new-repository). Going forward, we'll refer to this repository as `dagster-plus-code`. | ||
|
||
Next, we'll clone both our new repository and the Dagster+ hybrid quickstart repository to our local machine. | ||
|
||
```bash | ||
git clone <your-repo-url> | ||
git clone [email protected]:dagster-io/dagster-cloud-hybrid-quickstart.git | ||
``` | ||
|
||
We'll copy the contents of the `dagster-cloud-hybrid-quickstart` repository into our `dagster-plus-code` repository, and commit the changes. | ||
|
||
```bash | ||
cp -r dagster-cloud-hybrid-quickstart/* dagster-plus-code/ --exclude=".git" | ||
cd dagster-plus-code | ||
git add . | ||
git commit -m "Initial commit" | ||
git push | ||
``` | ||
|
||
### Project structure | ||
|
||
The project has the following structure: | ||
|
||
```plaintext | ||
├── .github | ||
│ └── workflows | ||
│ └── dagster-cloud-deploy.yml # GitHub Actions workflow for re-deploying code location | ||
├── .vscode # Standard VSCode settings for working with a Dagster repository | ||
├── Dockerfile # Dockerfile for building the user code image | ||
├── README.md | ||
├── dagster_cloud.yaml # Configuration file describing all code locations in the repository | ||
├── pyproject.toml # Python project configuration file for the code location | ||
├── quickstart_etl # Python package containing the user code | ||
│ ├── __init__.py | ||
│ ├── assets | ||
│ │ ├── __init__.py | ||
│ │ └── hackernews.py | ||
│ └── definitions.py | ||
├── quickstart_etl_tests # User code tests | ||
│ ├── __init__.py | ||
│ └── test_assets.py | ||
├── setup.cfg | ||
└── setup.py | ||
``` | ||
|
||
## Step 2: Setting up an Azure Container Registry | ||
|
||
Next, we'll set up an Azure Container Registry to store our Docker images. We'll use the Azure CLI to create the registry. | ||
|
||
```bash | ||
az login | ||
az acr create --resource-group <your_resource_group> --name dagstercodelocations --sku Basic | ||
``` | ||
|
||
Then, we'll make images from our ACR available to our AKS cluster. | ||
|
||
```bash | ||
az aks update -n <your-namespace> -g <your_resource_group> --attach-acr dagstercodelocations | ||
``` | ||
|
||
## Step 3: Setting up GitHub Actions | ||
|
||
Now, we'll set up a Github Actions workflow to build and push our Docker image to Azure Container Registry. | ||
|
||
We already have a GitHub Actions workflow in our repository, located at `.github/workflows/dagster-cloud-deploy.yml`. This workflow will build the Docker image, push it to ACR, and update the code location in Dagster+. To get it working with your repository, you'll need to do a few things. | ||
|
||
#### Generate Azure credentials | ||
|
||
First, we'll need to generate a service principal for GitHub Actions to use to authenticate with Azure. We'll use the Azure CLI to create the service principal. | ||
|
||
```bash | ||
az ad sp create-for-rbac --name "github-actions-acr" --role contributor --scopes /subscriptions/<your_azure_subscription_id>/resourceGroups/<your_resource_group>/providers/Microsoft.ContainerRegistry/registries/<your_acr_name> | ||
``` | ||
|
||
This command will output a JSON object with the service principal details. Make sure to save the `appId`, `password`, and `tenant` values - we'll use them in the next step. | ||
|
||
### Add secrets to your repository | ||
|
||
We'll add the service principal details as secrets in our repository. Go to your repository in GitHub, and navigate to `Settings` -> `Secrets`. Add the following secrets: | ||
|
||
- `AZURE_CLIENT_ID`: The `appId` from the service principal JSON object. | ||
- `AZURE_CLIENT_SECRET`: The `password` from the service principal JSON object. | ||
|
||
### Update the workflow | ||
|
||
Finally, we'll update the workflow to use the service principal details. Open `.github/workflows/dagster-cloud-deploy.yml` in your repository, and uncomment the section on Azure Container Registry. It should look like this: | ||
|
||
```yaml | ||
# Azure Container Registry (ACR) | ||
# https://github.com/docker/login-action#azure-container-registry-acr | ||
- name: Login to Azure Container Registry | ||
if: steps.prerun.outputs.result != 'skip' | ||
uses: docker/login-action@v3 | ||
with: | ||
registry: ${{ env.IMAGE_REGISTRY }} | ||
username: ${{ secrets.AZURE_CLIENT_ID }} | ||
password: ${{ secrets.AZURE_CLIENT_SECRET }} | ||
``` | ||
### Push and run the workflow | ||
Now, commit and push the changes to your repository. The GitHub Actions workflow should run automatically. You can check the status of the workflow in the `Actions` tab of your repository. | ||
|
||
<Image | ||
src="/images/dagster-cloud/azure/github-actions-workflow.png" | ||
alt="GitHub Actions workflow for deploying user code to Azure Container Registry" | ||
width={970} | ||
height={794} | ||
/> | ||
|
||
When the workflow completes, you should see the new code location in Dagster+. Navigate to the `Status` page, and click the `Code Locations` tab. You should see your new code location listed. | ||
|
||
<Image | ||
src="/images/dagster-cloud/azure/dagster-cloud-code-locations.png" | ||
alt="Dagster+ code locations page showing the new code location" | ||
width={1152} | ||
height={320} | ||
/> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file added
BIN
+358 KB
docs/next/public/images/dagster-cloud/azure/dagster-cloud-code-locations.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+189 KB
docs/next/public/images/dagster-cloud/azure/github-actions-workflow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.