Skip to content

Commit

Permalink
Change default python version in ubuntu image for newly templated ser…
Browse files Browse the repository at this point in the history
…verless orgs, bring in sync with dagster-cloud-action repo (#33)

Summary:
Now tht python 3.8 is EOL, change the default python version to 3.10 and the default ubuntu version to one that has 3.10 installed by default instead of 3.8.

Also make the quickstart use the same yaml that is templated when you create a new serverless org.
  • Loading branch information
gibsondan authored Oct 29, 2024
1 parent db898fc commit d6af248
Show file tree
Hide file tree
Showing 5 changed files with 119 additions and 163 deletions.
78 changes: 0 additions & 78 deletions .github/workflows/branch_deployments.yml

This file was deleted.

114 changes: 114 additions & 0 deletions .github/workflows/dagster-plus-deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
name: Dagster Cloud Serverless Deployment
on:
push:
branches:
- "main"
- "master"
pull_request:
types: [opened, synchronize, reopened, closed]

concurrency:
# Cancel in-progress deploys to same branch
group: ${{ github.ref }}/deploy
cancel-in-progress: true
env:
DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
ENABLE_FAST_DEPLOYS: 'true'
PYTHON_VERSION: '3.10'
DAGSTER_CLOUD_YAML_PATH: '.'
DAGSTER_CLOUD_FILE: 'dagster_cloud.yaml'
DAGSTER_CLOUD_ORGANIZATION: 'ORGANIZATION_NAME'
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

jobs:
dagster_cloud_default_deploy:
name: Dagster Serverless Deploy
runs-on: ubuntu-22.04

steps:
- name: Prerun Checks
id: prerun
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]

- name: Checkout
uses: actions/checkout@v3
with:
ref: ${{ github.head_ref }}

# Validate dagster_cloud.yaml and the connection to dagster.cloud
- name: Validate configuration
id: ci-validate
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci check --project-dir ${{ env.DAGSTER_CLOUD_YAML_PATH }} --dagster-cloud-yaml-path ${{ env.DAGSTER_CLOUD_FILE }}"

# Parse dagster_cloud.yaml, detect if this is branch deployment and initialize the build session
- name: Initialize build session
id: ci-init
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
project_dir: ${{ env.DAGSTER_CLOUD_YAML_PATH }}
dagster_cloud_yaml_path: ${{ env.DAGSTER_CLOUD_FILE }}
# The full deployment name. If this run is for a PR, this value is ignored and a branch
# deployment is used.
deployment: 'prod'


# If using fast build, build the PEX
# First ensure the correct Python version is installed
- name: Set up Python ${{ env.PYTHON_VERSION }} for target
id: setup-python-version
if: steps.prerun.outputs.result == 'pex-deploy'
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install setuptools
if: steps.prerun.outputs.result == 'pex-deploy'
run: ${{ steps.setup-python-version.outputs.python-path }} -m pip install setuptools
shell: bash

- name: Run PEX build
id: run-pex-build
if: steps.prerun.outputs.result == 'pex-deploy'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci build --build-strategy=python-executable --python-version ${{ env.PYTHON_VERSION }} --pex-deps-cache-from='${{ github.repository }}' --pex-deps-cache-to='${{ github.repository }}'"


# Otherwise, enable buildx for caching and build the Docker image
- name: Set up Docker Buildx
if: steps.prerun.outputs.result == 'docker-deploy'
uses: docker/setup-buildx-action@v2

- name: Run Docker build
id: run-docker-build
if: steps.prerun.outputs.result == 'docker-deploy'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci build --build-strategy=docker --python-version ${{ env.PYTHON_VERSION }}"


# Deploy all code locations in this build session to Dagster Cloud
- name: Deploy to Dagster Cloud
id: ci-deploy
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci deploy"

# Update a PR comment - this runs always() so the comment is updated on success and failure
- name: Update PR comment for branch deployments
id: ci-notify
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci notify --project-dir=${{ env.DAGSTER_CLOUD_YAML_PATH }}"

# Generate a summary that shows up on the Workflow Summary page
- name: Generate a summary
id: ci-summary
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci status --output-format=markdown >> $GITHUB_STEP_SUMMARY"
80 changes: 0 additions & 80 deletions .github/workflows/deploy.yml

This file was deleted.

8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
Welcome to your Dagster Cloud sample code repo. Here, you can find the code that's being deployed to your Dagster Cloud instance. For more in-depth information, check out our [Serverless](https://docs.dagster.io/dagster-cloud/deployment/serverless) docs.

Pushing to production will automatically kick off a [workflow](./.github/workflows/deploy.yml) which will redeploy your code to your `prod` deployment.
Pushing to production will automatically kick off a [workflow](./.github/workflows/dagster-plus-deploy.yml) which will redeploy your code to your `prod` deployment.

Creating a pull request will kick off a [workflow](./.github/workflows/deploy.yml) which will create a new [**Branch Deployment**](https://docs.dagster.io/dagster-cloud/developing-testing/branch-deployments), an ephemeral deployment where you can test your changes.
Creating a pull request will kick off a [workflow](./.github/workflows/dagster-plus-deploy.yml) which will create a new [**Branch Deployment**](https://docs.dagster.io/dagster-cloud/developing-testing/branch-deployments), an ephemeral deployment where you can test your changes.

# Setting up Quickstart Template Manually

Expand All @@ -26,10 +26,10 @@ Set up secrets on your newly created repository by navigating to the `Settings`

## Update workflows

Replace the `ORGANIZATION_NAME` in both `.github/workflows/deploy.yml` and `.github/workflows/branch_deployments.yml` with your Dagster cloud organization name:
Replace the `ORGANIZATION_NAME` in `.github/workflows/dagster-plus-deploy.yml` with your Dagster cloud organization name:

```
DAGSTER_CLOUD_URL: "http://ORGANIZATION_NAME.dagster.cloud"
DAGSTER_CLOUD_URL: "https://ORGANIZATION_NAME.dagster.cloud"
```

## Verify Builds are Successful
Expand Down
2 changes: 1 addition & 1 deletion dagster_cloud.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
locations:
- code_source:
package_name: quickstart_etl
package_name: quickstart_etl.definitions
location_name: my-project

0 comments on commit d6af248

Please sign in to comment.