-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AAQ-583] CI/CD for GCP #264
Merged
Merged
Changes from all commits
Commits
Show all changes
37 commits
Select commit
Hold shift + click to select a range
4ffb991
add gcp github actions workflow
suzinyou 23f9df4
configure deployment env
suzinyou 322c4df
fix provider path and use script
suzinyou daafb2c
add redeploy script
suzinyou ff2a045
hardcode repo path
suzinyou 8fde17a
update OIDC provider and add prune statement to redeploy script
suzinyou 032ff89
remove service account to use workload identity federation?
suzinyou e78e64e
upgrade google-github-actions
suzinyou c8340c0
fix env to vars
suzinyou f9265d2
impersonate service account
suzinyou f7773e4
switch to command in ssh
suzinyou b423e21
remove redeploy script
suzinyou 16db682
add cloudbuild configs
suzinyou 4e693d6
test caddy deployment workflow to GCP
suzinyou 295ee2b
remove old deploy script and fix branch name in gcp workflow
suzinyou 50c5f0e
fix docker build cloudbuild command syntax
suzinyou 0f6536a
add deployment workflow for backend, admin app, and litellm proxy
suzinyou fcb997d
fix typos in substitution var name
suzinyou 44cb9db
fix litellm proxy cloudbuild config path
suzinyou d900544
add restart policy and network
suzinyou db48026
pass custom CMD for config file path in litellm dockerfile
suzinyou 6dfd919
add docker commands to transition to new containers
suzinyou d403158
parse passworld for db connection string
suzinyou f55f380
update restart condition for caddy
suzinyou a6bccf5
use URL for connection strings
suzinyou b7a7eed
fix mypy error on port dtype
suzinyou 5540b9c
completely stop and remove existing container before running
suzinyou b03ec9c
remove this branch from trigger condition
suzinyou 4e86979
check for health after deployment
suzinyou adf9c3d
show stdout and stderr
suzinyou 0576049
fix caddy typo, and show ssh stderr and stdout
suzinyou 609a08b
fix litellm proxy path in make
suzinyou bb22c4c
add production branch trigger condition and parametrize deployment en…
suzinyou 07a28e3
clean up step name
suzinyou 62c64c0
Improve gcp deploy (#274)
suzinyou 77538fd
move config files back to docker-compose dir and undo testing code
suzinyou 57ac896
Test push
markbotterill File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| name: Deploy admin_app to GCP | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| - testing | ||
| - production | ||
| paths: | ||
| - "admin_app/**" | ||
| - ".github/workflows/deploy_gcp_admin_app.yaml" | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| DeployAdminAppToGCP: | ||
| runs-on: ubuntu-latest | ||
|
|
||
| permissions: | ||
| contents: "read" | ||
| id-token: "write" | ||
|
|
||
| # TODO: replace improve-gcp-deploy with main | ||
| environment: gcp-${{ (github.ref_name == 'main' && 'testing') || github.ref_name }} | ||
|
|
||
| env: | ||
| RESOURCE_PREFIX: ${{ secrets.PROJECT_NAME }}-${{ (github.ref_name == 'main' && 'testing') || github.ref_name }} | ||
| REPO: ${{ secrets.DOCKER_REGISTRY_DOMAIN }}/${{ secrets.GCP_PROJECT_ID }}/${{ secrets.PROJECT_NAME }}-${{ (github.ref_name == 'main' && 'testing') || github.ref_name }} | ||
|
|
||
| steps: | ||
| - uses: "actions/checkout@v4" | ||
|
|
||
| - id: "auth" | ||
| name: "Authenticate to Google Cloud" | ||
| uses: "google-github-actions/auth@v2" | ||
| with: | ||
| project_id: ${{ secrets.GCP_PROJECT_ID }} | ||
| workload_identity_provider: projects/${{ secrets.GCP_PROJECT_NUMBER }}/locations/global/workloadIdentityPools/${{ vars.POOL_ID }}/providers/${{ vars.PROVIDER_ID }} | ||
| service_account: ${{ secrets.GCP_SERVICE_ACCOUNT_EMAIL }} | ||
|
|
||
| - name: Retrieve secrets from Secret Manager | ||
| id: "secrets" | ||
| uses: "google-github-actions/get-secretmanager-secrets@v2" | ||
| with: | ||
| min_mask_length: 4 | ||
| secrets: |- | ||
| domain:${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }}-domain | ||
| google_login_client_id:${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }}-google-login-client-id | ||
|
|
||
| - name: Configure Docker to use gcloud as a credential helper | ||
| run: | | ||
| gcloud auth configure-docker ${{ secrets.DOCKER_REGISTRY_DOMAIN}} | ||
|
|
||
| - name: Build and push admin_app image | ||
| working-directory: admin_app | ||
| run: | | ||
| docker build \ | ||
| --build-arg NEXT_PUBLIC_BACKEND_URL="https://${{ steps.secrets.outputs.domain }}/api" \ | ||
| --build-arg NEXT_PUBLIC_GOOGLE_LOGIN_CLIENT_ID="${{ steps.secrets.outputs.google_login_client_id }}" \ | ||
| -t ${{ env.REPO }}/admin_app:latest \ | ||
| -t ${{ env.REPO }}/admin_app:${{ github.sha }} \ | ||
| . | ||
| docker image push --all-tags ${{ env.REPO }}/admin_app | ||
|
|
||
| - name: Deploy admin_app container | ||
| id: "compute-ssh" | ||
| uses: "google-github-actions/ssh-compute@v1" | ||
| env: | ||
| REPO: ${{ secrets.DOCKER_REGISTRY_DOMAIN }}/${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }} | ||
| with: | ||
| instance_name: "${{ secrets.DEPLOYMENT_INSTANCE_NAME }}" | ||
| zone: "${{ secrets.DEPLOYMENT_ZONE }}" | ||
| ssh_private_key: "${{ secrets.GCP_SSH_PRIVATE_KEY }}" | ||
| command: | | ||
| docker-credential-gcr configure-docker \ | ||
| --registries ${{ secrets.DOCKER_REGISTRY_DOMAIN }} | ||
| docker pull \ | ||
| ${{ env.REPO }}/admin_app:latest | ||
| docker stop admin_app | ||
| docker rm admin_app | ||
| docker run -d \ | ||
| --log-driver=gcplogs \ | ||
| --restart always \ | ||
| --network aaq-network \ | ||
| --name admin_app \ | ||
| ${{ env.REPO }}/admin_app:latest | ||
| docker system prune -f | ||
|
|
||
| - name: Show deployment command output | ||
| run: |- | ||
| echo '${{ steps.compute-ssh.outputs.stdout }}' | ||
| echo '${{ steps.compute-ssh.outputs.stderr }}' | ||
|
|
||
| - name: Wait for Application to start | ||
| id: wait-for-app | ||
| run: sleep 1m | ||
| shell: bash | ||
|
|
||
| - name: Check if deployment was successful | ||
| id: check-deployment | ||
| run: | | ||
| curl -f -X 'GET' \ | ||
| 'https://${{ steps.secrets.outputs.domain }}/api/healthcheck' \ | ||
| -H 'accept: application/json' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| name: Deploy Caddy to GCP | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| - testing | ||
| - production | ||
| paths: | ||
| - "deployment/docker-compose/caddy/**" | ||
| - ".github/workflows/deploy_gcp_caddy.yaml" | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| DeployCaddyToGCP: | ||
| runs-on: ubuntu-latest | ||
|
|
||
| permissions: | ||
| contents: "read" | ||
| id-token: "write" | ||
|
|
||
| # TODO: replace improve-gcp-deploy with main | ||
| environment: gcp-${{ (github.ref_name == 'main' && 'testing') || github.ref_name }} | ||
|
|
||
| env: | ||
| RESOURCE_PREFIX: ${{ secrets.PROJECT_NAME }}-${{ (github.ref_name == 'main' && 'testing') || github.ref_name }} | ||
|
|
||
| steps: | ||
| - uses: "actions/checkout@v4" | ||
|
|
||
| - id: "auth" | ||
| name: "Authenticate to Google Cloud" | ||
| uses: "google-github-actions/auth@v2" | ||
| with: | ||
| project_id: ${{ secrets.GCP_PROJECT_ID }} | ||
| workload_identity_provider: projects/${{ secrets.GCP_PROJECT_NUMBER }}/locations/global/workloadIdentityPools/${{ vars.POOL_ID }}/providers/${{ vars.PROVIDER_ID }} | ||
| service_account: ${{ secrets.GCP_SERVICE_ACCOUNT_EMAIL }} | ||
|
|
||
| - name: Retrieve secrets from Secret Manager | ||
| id: "secrets" | ||
| uses: "google-github-actions/get-secretmanager-secrets@v2" | ||
| with: | ||
| secrets: |- | ||
| domain:${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }}-domain | ||
|
|
||
| - name: Copy Caddy deployment files | ||
| working-directory: deployment/docker-compose | ||
| run: | | ||
| gcloud compute scp Caddyfile \ | ||
| ${{ secrets.DEPLOYMENT_INSTANCE_NAME }}:~/Caddyfile \ | ||
| --zone ${{ secrets.DEPLOYMENT_ZONE }} | ||
|
|
||
| - name: Deploy Caddy container | ||
| id: "compute-ssh" | ||
| uses: "google-github-actions/ssh-compute@v1" | ||
| with: | ||
| instance_name: "${{ secrets.DEPLOYMENT_INSTANCE_NAME }}" | ||
| zone: "${{ secrets.DEPLOYMENT_ZONE }}" | ||
| ssh_private_key: "${{ secrets.GCP_SSH_PRIVATE_KEY }}" | ||
| command: | | ||
| docker stop caddy | ||
| docker rm caddy | ||
| docker run -d \ | ||
| -v caddy_data:/data \ | ||
| -v caddy_config:/config \ | ||
| -e DOMAIN=${{ steps.secrets.outputs.domain }} \ | ||
| -p 80:80 \ | ||
| -p 443:443 \ | ||
| -p 443:443/udp \ | ||
| -v ~/Caddyfile:/etc/caddy/Caddyfile \ | ||
| --log-driver=gcplogs \ | ||
| --restart always \ | ||
| --network aaq-network \ | ||
| --name caddy \ | ||
| caddy:2.7.6 | ||
| docker system prune --volumes -f | ||
|
|
||
| - name: Show deployment command output | ||
| run: |- | ||
| echo '${{ steps.compute-ssh.outputs.stdout }}' | ||
| echo '${{ steps.compute-ssh.outputs.stderr }}' | ||
|
|
||
| - name: Wait for Application to start | ||
| id: wait-for-app | ||
| run: sleep 1m | ||
| shell: bash | ||
|
|
||
| - name: Check if deployment was successful | ||
| id: check-deployment | ||
| run: | | ||
| curl -f -X 'GET' \ | ||
| 'https://${{ steps.secrets.outputs.domain }}/api/healthcheck' \ | ||
| -H 'accept: application/json' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| name: Deploy core_backend to GCP | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| - testing | ||
| - production | ||
| paths: | ||
| - "core_backend/**" | ||
| - ".github/workflows/deploy_gcp_core_backend.yaml" | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| DeployCoreBackendToGCP: | ||
| runs-on: ubuntu-latest | ||
|
|
||
| permissions: | ||
| contents: "read" | ||
| id-token: "write" | ||
|
|
||
| # TODO: replace improve-gcp-deploy with main | ||
| environment: gcp-${{ (github.ref_name == 'main' && 'testing') || github.ref_name }} | ||
|
|
||
| env: | ||
| RESOURCE_PREFIX: ${{ secrets.PROJECT_NAME }}-${{ (github.ref_name == 'main' && 'testing') || github.ref_name }} | ||
| REPO: ${{ secrets.DOCKER_REGISTRY_DOMAIN }}/${{ secrets.GCP_PROJECT_ID }}/${{ secrets.PROJECT_NAME }}-${{ (github.ref_name == 'main' && 'testing') || github.ref_name }} | ||
|
|
||
| steps: | ||
| - uses: "actions/checkout@v4" | ||
|
|
||
| - id: "auth" | ||
| name: "Authenticate to Google Cloud" | ||
| uses: "google-github-actions/auth@v2" | ||
| with: | ||
| project_id: ${{ secrets.GCP_PROJECT_ID }} | ||
| workload_identity_provider: projects/${{ secrets.GCP_PROJECT_NUMBER }}/locations/global/workloadIdentityPools/${{ vars.POOL_ID }}/providers/${{ vars.PROVIDER_ID }} | ||
| service_account: ${{ secrets.GCP_SERVICE_ACCOUNT_EMAIL }} | ||
|
|
||
| - name: Retrieve secrets from Secret Manager | ||
| id: "secrets" | ||
| uses: "google-github-actions/get-secretmanager-secrets@v2" | ||
| with: | ||
| secrets: |- | ||
| domain:${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }}-domain | ||
| jwt-secret:${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }}-jwt-secret | ||
| google-login-client-id:${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }}-google-login-client-id | ||
| langfuse-secret-key:${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }}-langfuse-secret-key | ||
| langfuse-public-key:${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }}-langfuse-public-key | ||
| db-host:${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }}-db-host | ||
| db-password:${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }}-db-password | ||
| admin-username:${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }}-admin-username | ||
| admin-password:${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }}-admin-password | ||
| admin-api-key:${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }}-admin-api-key | ||
|
|
||
| - name: Configure Docker to use gcloud as a credential helper | ||
| run: | | ||
| gcloud auth configure-docker ${{ secrets.DOCKER_REGISTRY_DOMAIN}} | ||
|
|
||
| - name: Build and push core_backend image | ||
| working-directory: core_backend | ||
| run: | | ||
| docker build \ | ||
| -t ${{ env.REPO }}/core_backend:latest \ | ||
| -t ${{ env.REPO }}/core_backend:${{ github.sha }} \ | ||
| . | ||
| docker image push --all-tags ${{ env.REPO }}/core_backend | ||
|
|
||
| - name: Deploy core_backend container | ||
| id: "compute-ssh" | ||
| uses: "google-github-actions/ssh-compute@v1" | ||
| env: | ||
| REPO: ${{ secrets.DOCKER_REGISTRY_DOMAIN }}/${{ secrets.GCP_PROJECT_ID }}/${{ env.RESOURCE_PREFIX }} | ||
| with: | ||
| instance_name: "${{ secrets.DEPLOYMENT_INSTANCE_NAME }}" | ||
| zone: "${{ secrets.DEPLOYMENT_ZONE }}" | ||
| ssh_private_key: "${{ secrets.GCP_SSH_PRIVATE_KEY }}" | ||
| command: | | ||
| docker-credential-gcr configure-docker \ | ||
| --registries ${{ secrets.DOCKER_REGISTRY_DOMAIN }} | ||
| docker pull \ | ||
| ${{ env.REPO }}/core_backend:latest | ||
| docker stop core_backend | ||
| docker rm core_backend | ||
| docker run -d \ | ||
| --log-driver=gcplogs \ | ||
| --restart always \ | ||
| --network aaq-network \ | ||
| --name core_backend \ | ||
| -e JWT_SECRET="${{ steps.secrets.outputs.jwt-secret }}" \ | ||
| -e NEXT_PUBLIC_GOOGLE_LOGIN_CLIENT_ID="${{ steps.secrets.outputs.google-login-client-id }}" \ | ||
| -e DOMAIN="${{ steps.secrets.outputs.domain }}" \ | ||
| -e POSTGRES_HOST="${{ steps.secrets.outputs.db-host }}" \ | ||
| -e POSTGRES_PASSWORD="${{ steps.secrets.outputs.db-password }}" \ | ||
| -e ADMIN_USERNAME="${{ steps.secrets.outputs.admin-username }}" \ | ||
| -e ADMIN_PASSWORD="${{ steps.secrets.outputs.admin-password }}" \ | ||
| -e ADMIN_API_KEY="${{ steps.secrets.outputs.admin-api-key }}" \ | ||
| -e PROMETHEUS_MULTIPROC_DIR=/tmp \ | ||
| -e LITELLM_ENDPOINT=http://litellm_proxy:4000 \ | ||
| -e LANGFUSE=True \ | ||
| -e LANGFUSE_SECRET_KEY="${{ steps.secrets.outputs.langfuse-secret-key }}" \ | ||
| -e LANGFUSE_PUBLIC_KEY="${{ steps.secrets.outputs.langfuse-public-key }}" \ | ||
| -e BACKEND_ROOT_PATH=/api \ | ||
| ${{ env.REPO }}/core_backend:latest | ||
| docker system prune -f | ||
|
|
||
| - name: Show deployment command output | ||
| run: |- | ||
| echo '${{ steps.compute-ssh.outputs.stdout }}' | ||
| echo '${{ steps.compute-ssh.outputs.stderr }}' | ||
|
|
||
| - name: Wait for Application to start | ||
| id: wait-for-app | ||
| run: sleep 1m | ||
| shell: bash | ||
|
|
||
| - name: Check if deployment was successful | ||
| id: check-deployment | ||
| run: | | ||
| curl -f -X 'GET' \ | ||
| 'https://${{ steps.secrets.outputs.domain }}/api/healthcheck' \ | ||
| -H 'accept: application/json' |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All deployment scripts follow the same pattern!