[AAQ-583] CI/CD for GCP #264

suzinyou · 2024-06-27T17:40:33Z

Reviewer: @markbotterill
Estimate: 2 hours

Ticket

Fixes: https://idinsight.atlassian.net/browse/AAQ-583

Description

Goal

Deploy to GCP infra on push to main

Changes

Deployment scripts: .github/workflows/deploy_gcp_*.yaml, triggered for each component
1. Fetch GCP secrets
2. Build and push docker image
3. For LiteLLM and Caddy, we SCP the config files to the machine (*instead of baking them into the image)
  - This way, we don't have to keep a copy of aaq-core repo in the VM. Just deploy using the current version of these configs.
4. SSH to VM to deploy docker containers
  - We are using docker run and not docker compose.
    - The Container-Optimized OS doesn't come with docker compose!!
    - We can theoretically SCP docker-compose.yml (similarly to 3. config files above) but didn't want to sink time into installing docker compose to VM
Replace DB connection string with sqlalchemy's URL object
1. Our DB connection would fail because of special characters in the connection string. Using URL object handles special characters automatically.
2. However, we should avoid % in password -- this causes issues for alembic.

Future Tasks (optional)

~~Turn infra into Terraform~~ In progress at [AAQ-613] Terraform for GCP production infra #272
Find a way to deploy with docker-compose.yml?

How has this been tested?

To-do before merge (optional)

Checklist

Fill with x for completed.

My code follows the style guidelines of this project
I have reviewed my own code to ensure good quality
I have tested the functionality of my code to ensure it works as intended
I have resolved merge conflicts

(Delete any items below that are not relevant)

I have updated affected documentation
I have added a blogpost in Latest Updates
I have updated the CI/CD scripts in .github/workflows/
~~I have updated the Terraform code~~ Future task

suzinyou · 2024-07-01T06:47:56Z

.github/workflows/deploy_gcp_admin_app.yaml

+      - "admin_app/**"
+      - "deployment/gcp/cloudbuild_admin_app.yaml"
+      - ".github/workflows/deploy_gcp_admin_app.yaml"


Trigger only if the component code or CI/CD scripts change

.github/workflows/deploy_gcp_admin_app.yaml

suzinyou · 2024-07-01T06:50:19Z

.github/workflows/deploy_gcp_admin_app.yaml

All deployment scripts follow the same pattern!

suzinyou · 2024-07-01T06:52:55Z

deployment/gcp/cloudbuild_admin_app.yaml

+        docker build \
+          --build-arg NEXT_PUBLIC_BACKEND_URL=https://$$DOMAIN/api \
+          --build-arg NEXT_PUBLIC_GOOGLE_LOGIN_CLIENT_ID=$$NEXT_PUBLIC_GOOGLE_LOGIN_CLIENT_ID \
+          -t ${_DOCKER_REGISTRY_DOMAIN}/$PROJECT_ID/${_RESOURCE_PREFIX}/admin_app:latest \
+          .
+    secretEnv: ["NEXT_PUBLIC_GOOGLE_LOGIN_CLIENT_ID", "DOMAIN"]


Secret environment variables are accessed by $$<var>, substituted variables (which we pass in github actions) are accessed by $<var> and must start with _

suzinyou · 2024-07-01T06:53:38Z

deployment/gcp/cloudbuild_admin_app.yaml

+      env: "DOMAIN"
+
+images:
+  - "${_DOCKER_REGISTRY_DOMAIN}/$PROJECT_ID/${_RESOURCE_PREFIX}/admin_app:latest"


$PROJECT_ID is availabe through cloudbuild direclty, rather than from GCP Secret Manager or Github Actions

…v name

* send caddyfile and docker compose via scp * remove ssh key * format multiline commnad * fix docker compose name * use docker run * fix multiline command * build admin_app from github actions * remove cloudbuild file trigger * temporarily add this branch as trigger for admin_app * update litellm cicd to build from github actions * update core backend cicd to build from github actions * fix order of config in litellm * quote env vars * add docker push statements * fix docker push statement * fix secrets typo * fix quotes * fix secrets outputs * delete backend cloudbuild * delete unused docker compose

markbotterill

Approved.

Have discussed V2 improvements w/ @suzinyou which we can flesh out in coming weeks!

suzinyou commented Jul 1, 2024

View reviewed changes

.github/workflows/deploy_gcp_admin_app.yaml Outdated Show resolved Hide resolved

suzinyou commented Jul 1, 2024

View reviewed changes

.github/workflows/deploy_gcp_admin_app.yaml

Copy link

Collaborator Author

suzinyou Jul 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All deployment scripts follow the same pattern!

suzinyou commented Jul 1, 2024

View reviewed changes

suzinyou marked this pull request as ready for review July 1, 2024 06:55

suzinyou requested review from sidravi1, lickem22, amiraliemami, Tanmay-97 and markbotterill as code owners July 1, 2024 06:55

suzinyou added 19 commits July 1, 2024 15:27

add gcp github actions workflow

4ffb991

configure deployment env

23f9df4

fix provider path and use script

322c4df

add redeploy script

daafb2c

hardcode repo path

ff2a045

update OIDC provider and add prune statement to redeploy script

8fde17a

remove service account to use workload identity federation?

032ff89

upgrade google-github-actions

e78e64e

fix env to vars

c8340c0

impersonate service account

f9265d2

switch to command in ssh

f7773e4

remove redeploy script

b423e21

add cloudbuild configs

16db682

test caddy deployment workflow to GCP

4e693d6

remove old deploy script and fix branch name in gcp workflow

295ee2b

fix docker build cloudbuild command syntax

50c5f0e

add deployment workflow for backend, admin app, and litellm proxy

0f6536a

fix typos in substitution var name

fcb997d

fix litellm proxy cloudbuild config path

44cb9db

suzinyou added 12 commits July 1, 2024 15:27

add restart policy and network

d900544

pass custom CMD for config file path in litellm dockerfile

db48026

add docker commands to transition to new containers

6dfd919

parse passworld for db connection string

d403158

update restart condition for caddy

f55f380

use URL for connection strings

a6bccf5

fix mypy error on port dtype

b7a7eed

completely stop and remove existing container before running

5540b9c

remove this branch from trigger condition

b03ec9c

check for health after deployment

4e86979

show stdout and stderr

adf9c3d

fix caddy typo, and show ssh stderr and stdout

0576049

suzinyou force-pushed the deploy-on-gcp-ga branch from 2a0ffec to 0576049 Compare July 1, 2024 09:57

suzinyou mentioned this pull request Jul 2, 2024

[AAQ-614] Use Vertex AI API in GCP deployment #270

Merged

13 tasks

suzinyou added 3 commits July 3, 2024 13:10

fix litellm proxy path in make

609a08b

add production branch trigger condition and parametrize deployment en…

bb22c4c

…v name

clean up step name

07a28e3

suzinyou mentioned this pull request Jul 3, 2024

[AAQ-613] Deploy prod #273

Closed

13 tasks

suzinyou temporarily deployed to gcp-production July 8, 2024 06:45 — with GitHub Actions Inactive

suzinyou added 2 commits July 8, 2024 12:16

move config files back to docker-compose dir and undo testing code

77538fd

suzinyou temporarily deployed to gcp-production July 8, 2024 10:45 — with GitHub Actions Inactive

Test push

57ac896

markbotterill approved these changes Jul 8, 2024

View reviewed changes

suzinyou merged commit 95b93ea into main Jul 8, 2024
1 check passed

suzinyou deleted the deploy-on-gcp-ga branch July 8, 2024 15:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AAQ-583] CI/CD for GCP #264

[AAQ-583] CI/CD for GCP #264

suzinyou commented Jun 27, 2024 •

edited

Loading

suzinyou Jul 1, 2024

suzinyou Jul 1, 2024

suzinyou Jul 1, 2024

suzinyou Jul 1, 2024

markbotterill left a comment

[AAQ-583] CI/CD for GCP #264

[AAQ-583] CI/CD for GCP #264

Conversation

suzinyou commented Jun 27, 2024 • edited Loading

Ticket

Description

Goal

Changes

Future Tasks (optional)

How has this been tested?

To-do before merge (optional)

Checklist

suzinyou Jul 1, 2024

Choose a reason for hiding this comment

suzinyou Jul 1, 2024

Choose a reason for hiding this comment

suzinyou Jul 1, 2024

Choose a reason for hiding this comment

suzinyou Jul 1, 2024

Choose a reason for hiding this comment

markbotterill left a comment

Choose a reason for hiding this comment

suzinyou commented Jun 27, 2024 •

edited

Loading