Dagster k8s Deployment

Deployment of Dagster (with a materialized asset) on Kubernetes with exposed UI through Nginx Ingress controller protected by basic-auth

Table of Content

1. Required Tools
2. Local K8s Cluster
3. Nginx Ingress Controller
4. Dagster Deployment
   • 1. Decrypt Secrets
   • 2. Call The Script
5. Code Location
   • gldas_noah Asset
6. CI/CD

Required Tools

Kubectl client version: v1.30.0

Kubernetes server version: v1.30.0

Helm client version: v3.15.2

Local K8s Cluster

You may skip if you already have a k8s cluster running.

kind create cluster --config cluster-config.yaml

Nginx Ingress Controller

Again please skip this also if you have Nginx-Ingress-Controller somewhere in your cluster.

kubectl apply -f nginx-ingress-controller/deploy.yaml

Dagster Deployment

1. Decrypt Secrets

I tried to keep it simple so we don't need to install more dependencies. There's a file dagster/secrets.yaml.enc which is the encoded file of the secret manifest. Please use the Password provided to you to decrypt it first:

openssl enc -d -aes-256-cbc -salt -in dagster/secrets.yaml.enc -out dagster/secrets.yaml -k <password>

2. Call The Script

There is a simple script dagster/upgrade.sh to upgrade/install the dagster chart after creating namespace and the secrets. (you are welcome to skim through it before running it!) It's also helpful for keeping track of which chart version is in production.

./dagster/upgrade.sh <namespace> <hostname>

• Args:
  • <namespace>: namespace to deploy dagster [Default: 'default']
  • <hostname>: hostname for the ingress [Default: 'localhost']

Default usage:

./dagster/upgrade.sh default localhost

The Dagster Webserver UI should be accessible by heading to localhost or the domain given to the script if a DNS is configured for the domain to point to the Ingress Controller Service.

The UI is protected by Nginx Basic Auth and the credentials are encrypted in the secrets YAML file.

Code Location

Each directory in the src represents a user deployment in Dagster with its own assets and pipelines.

gldas_noah Asset

For demonstration purposes it sleeps for 60 seconds so we can check whether the file has been downloaded successfully in the ephemeral disk space. It also outputs the secrets and the S3 configs so we can verify their sanity.

CI/CD

A GitHub Action tests the asset, builds the asset code location, and pushes it to Docker Hub if there are changes in the code location directory. We can use tools like ArgoCD for CD as future improvements.