Skip to content

Commit

Permalink
[azure-docs] Deploying with ACR.
Browse files Browse the repository at this point in the history
  • Loading branch information
dpeng817 committed Nov 27, 2024
1 parent 54b3b79 commit a1892b7
Show file tree
Hide file tree
Showing 6 changed files with 151 additions and 4 deletions.
4 changes: 4 additions & 0 deletions docs/content/_navigation.json
Original file line number Diff line number Diff line change
Expand Up @@ -556,6 +556,10 @@
{
"title": "Part 1: Setting up an AKS agent.",
"path": "/dagster-plus/deployment/azure/aks-agent"
},
{
"title": "Part 2: Deploy user code using ACR.",
"path": "/dagster-plus/deployment/azure/acr-user-code"
}
]
}
Expand Down
135 changes: 135 additions & 0 deletions docs/content/dagster-plus/deployment/azure/acr-user-code.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
# Deploying user code in Azure Container Registry (ACR) with Dagster+

This quickstart guide will walk you through setting up a new repository for your Dagster code, setting up CI/CD with GitHub Actions backed by Azure Container Registry (ACR), and deploying your code to your Azure Kubernetes Service (AKS) cluster.

This guide assumes you already have an AKS agent running. You can follow along [here](/dagster-plus/deployment/azure/aks-agent) if you still need to set up an AKS agent.

## Prerequisites

This guide will use a Github repository to store the Dagster code, and GitHub Actions to deploy the code to Azure Container Registry - but you should be able to adapt these steps to other version control systems and CI/CD tools.

- The azure CLI installed on your machine. You can download it [here](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli).
- A GitHub account, and the ability to run GitHub Actions workflows in a repository.

## Step 1: Creating a repository for Dagster code.

We'll create a new repository based on the [Dagster+ hybrid quickstart repository](https://github.com/dagster-io/dagster-cloud-hybrid-quickstart). We'll go through these steps using a brand new repository in GitHub, but you should be able to easily adapt these steps to an existing repository or other version control systems.

First, [create a new repository in GitHub](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-new-repository). Going forward, we'll refer to this repository as `dagster-plus-code`.

Next, we'll clone both our new repository and the Dagster+ hybrid quickstart repository to our local machine.

```bash
git clone <your-repo-url>
git clone [email protected]:dagster-io/dagster-cloud-hybrid-quickstart.git
```

We'll copy the contents of the `dagster-cloud-hybrid-quickstart` repository into our `dagster-plus-code` repository, and commit the changes.

```bash
cp -r dagster-cloud-hybrid-quickstart/* dagster-plus-code/ --exclude=".git"
cd dagster-plus-code
git add .
git commit -m "Initial commit"
git push
```

### Project structure

The project has the following structure:

```plaintext
├── .github
│ └── workflows
│ └── dagster-cloud-deploy.yml # GitHub Actions workflow for re-deploying code location
├── .vscode # Standard VSCode settings for working with a Dagster repository
├── Dockerfile # Dockerfile for building the user code image
├── README.md
├── dagster_cloud.yaml # Configuration file describing all code locations in the repository
├── pyproject.toml # Python project configuration file for the code location
├── quickstart_etl # Python package containing the user code
│ ├── __init__.py
│ ├── assets
│ │ ├── __init__.py
│ │ └── hackernews.py
│ └── definitions.py
├── quickstart_etl_tests # User code tests
│ ├── __init__.py
│ └── test_assets.py
├── setup.cfg
└── setup.py
```

## Step 2: Setting up an Azure Container Registry

Next, we'll set up an Azure Container Registry to store our Docker images. We'll use the Azure CLI to create the registry.

```bash
az login
az acr create --resource-group <your_resource_group> --name dagstercodelocations --sku Basic
```

Then, we'll make images from our ACR available to our AKS cluster.

```bash
az aks update -n <your-namespace> -g <your_resource_group> --attach-acr dagstercodelocations
```

## Step 3: Setting up GitHub Actions

Now, we'll set up a Github Actions workflow to build and push our Docker image to Azure Container Registry.

We already have a GitHub Actions workflow in our repository, located at `.github/workflows/dagster-cloud-deploy.yml`. This workflow will build the Docker image, push it to ACR, and update the code location in Dagster+. To get it working with your repository, you'll need to do a few things.

#### Generate Azure credentials

First, we'll need to generate a service principal for GitHub Actions to use to authenticate with Azure. We'll use the Azure CLI to create the service principal.

```bash
az ad sp create-for-rbac --name "github-actions-acr" --role contributor --scopes /subscriptions/<your_azure_subscription_id>/resourceGroups/<your_resource_group>/providers/Microsoft.ContainerRegistry/registries/<your_acr_name>
```

This command will output a JSON object with the service principal details. Make sure to save the `appId`, `password`, and `tenant` values - we'll use them in the next step.

### Add secrets to your repository

We'll add the service principal details as secrets in our repository. Go to your repository in GitHub, and navigate to `Settings` -> `Secrets`. Add the following secrets:

- `AZURE_CLIENT_ID`: The `appId` from the service principal JSON object.
- `AZURE_CLIENT_SECRET`: The `password` from the service principal JSON object.

### Update the workflow

Finally, we'll update the workflow to use the service principal details. Open `.github/workflows/dagster-cloud-deploy.yml` in your repository, and uncomment the section on Azure Container Registry. It should look like this:

```yaml
# Azure Container Registry (ACR)
# https://github.com/docker/login-action#azure-container-registry-acr
- name: Login to Azure Container Registry
if: steps.prerun.outputs.result != 'skip'
uses: docker/login-action@v3
with:
registry: ${{ env.IMAGE_REGISTRY }}
username: ${{ secrets.AZURE_CLIENT_ID }}
password: ${{ secrets.AZURE_CLIENT_SECRET }}
```
### Push and run the workflow
Now, commit and push the changes to your repository. The GitHub Actions workflow should run automatically. You can check the status of the workflow in the `Actions` tab of your repository.

<Image
src="/images/dagster-cloud/azure/github-actions-workflow.png"
alt="GitHub Actions workflow for deploying user code to Azure Container Registry"
width={970}
height={794}
/>

When the workflow completes, you should see the new code location in Dagster+. Navigate to the `Status` page, and click the `Code Locations` tab. You should see your new code location listed.

<Image
src="/images/dagster-cloud/azure/dagster-cloud-code-locations.png"
alt="Dagster+ code locations page showing the new code location"
width={1152}
height={320}
/>
4 changes: 4 additions & 0 deletions docs/content/dagster-plus/deployment/azure/aks-agent.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,7 @@ kubectl config current-context
## Step 3: Install the Dagster+ agent on the AKS cluster.

Next, we'll install the agent helm chart. You should be able to follow the guide [here](/concepts/deployment/agents/kubernetes/configuring-running-kubernetes-agent) to install the agent on the AKS cluster.

## Next steps

Now that you have an agent running on your AKS cluster, you can start deploying Dagster code to it. You can follow the guide [here](/concepts/deployment/azure/acr-user-code) to deploy user code to your AKS cluster backed by Azure Container Registry (ACR).
12 changes: 8 additions & 4 deletions docs/content/dagster-plus/deployment/azure/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,12 @@
This guide will walk you through deploying Dagster+ on Azure. We'll start from a brand new organization in Dagster+, and finish with a full hybrid deployment of Dagster+ using Azure infrastructure.

<ArticleList>
<ArticleListItem
title="Step 1: Deploy an Azure Kubernetes Service (AKS) agent"
href="/dagster-plus/deployment/azure/aks-agent"
></ArticleListItem>
<ArticleListItem
title="Step 1: Deploy an Azure Kubernetes Service (AKS) agent"
href="/dagster-plus/deployment/azure/aks-agent"
></ArticleListItem>
<ArticleListItem
title="Step 2: Deploy user code in Azure Container Registry (ACR)"
href="/dagster-plus/deployment/azure/acr-user-code"
></ArticleListItem>
</ArticleList>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit a1892b7

Please sign in to comment.