Merge branch 'main' into upgrade-jupyterhub

.github/workflows/deploy-kubernetes.yml

@@ -57,7 +57,7 @@ jobs:
       # Invoke deployment tasks
       - run: poetry install
         working-directory: ci
-      - run: poetry run invoke secrets -f "./channels/prod.yaml"
+      - run: poetry run invoke secrets -f "./kubernetes-workloads.yaml"
         working-directory: ci
-      - run: poetry run invoke release -f "./channels/prod.yaml"
+      - run: poetry run invoke release -f "./kubernetes-workloads.yaml"
         working-directory: ci

.github/workflows/preview-kubernetes.yml

@@ -57,7 +57,7 @@ jobs:
       # Diff and write back to PR
       - run: poetry install
         working-directory: ci
-      - run: poetry run invoke diff -f "./channels/prod.yaml" --outfile=diff.md
+      - run: poetry run invoke diff -f "./kubernetes-workloads.yaml" --outfile=diff.md
         working-directory: ci
       - uses: peter-evans/find-comment@v2
         id: fc

ci/README.md

@@ -1,7 +1,7 @@
 # CI - deploys via pyinvoke
 
 This folder contains code and YAML that drives the deployments of Kubernetes-based applications and services. For example,
-a deployment named `archiver` is configured in [the prod channel](./channels/prod.yaml) and is ultimatedly deployed
+a deployment named `archiver` is configured in [the prod channel](./kubernetes-workloads.yaml) and is ultimatedly deployed
 by `invoke` (see below) calling `kubectl` commands.
 
 ## invoke (aka pyinvoke)
@@ -11,11 +11,9 @@ The tasks are defined in `tasks.py` and configuration in `invoke.yaml`; config v
 are specific to our defined tasks.
 
 Run `poetry run invoke -l` to list the available commands, and `poetry run invoke -h <command>` to get more detailed help for each individual command.
-Individual release channels/environments are config files that are passed to invoke. For example, to deploy to test:
+Individual release channels/environments are config files that are passed to invoke.
 
-```bash
-poetry run invoke release -f channels/test.yaml
-```
+In general, pyinvoke commands are not intended to be run locally, but rather via GitHub Actions in respose to pull requests (i.e. GitOps)
 
 ## GitOps
 

ci/channels/prod.yaml → ci/kubernetes-workloads.yaml

@@ -110,3 +110,28 @@ calitp:
       namespace: dask
       helm_name: dask
       helm_chart: kubernetes/apps/charts/dask
+
+    # *-test deployments to test namespace
+    - name: archiver-test
+      driver: kustomize
+      kustomize_dir: kubernetes/apps/overlays/gtfs-rt-archiver-v3-test
+      secrets:
+        - gtfs-rt-v3-test_gtfs-feed-secrets
+    - name: metabase-test
+      driver: helm
+      namespace: metabase-test
+      helm_name: metabase-test
+      helm_chart: kubernetes/apps/charts/metabase
+      helm_values:
+        - kubernetes/apps/values/metabase-test.yaml
+    - name: postgresql-backup-metabase-test
+      driver: helm
+      namespace: metabase-test
+      helm_name: postgresql-backup
+      helm_chart: kubernetes/apps/charts/postgresql-backup
+      helm_values:
+        - kubernetes/apps/values/postgresql-backup-metabase.yaml
+        - kubernetes/apps/values/postgresql-backup-metabase-test.yaml
+      secrets:
+        - metabase-test_database-backup
+        - metabase-test_gcs-upload-svcacct

kubernetes/README.md

@@ -77,21 +77,15 @@ Once the old node pool is removed from the array, it can be drained and deleted
 
 Cluster workloads are divided into two classes:
 
-1. Apps are the workloads that users actually care about; this includes deployed "applications" such as the GTFS-RT archiver but also includes "services" like Grafana and Sentry. These workloads are deployed using `invoke` as defined in the [ci](../ci/) folder.
+1. Apps are the workloads that users actually care about; this includes deployed "applications" such as the GTFS-RT archiver but also includes "services" like Grafana and Sentry.
 
 2. System workloads are used to support running applications. This includes items such as an ingress controller, HTTPS certificate manager, etc. The system deploy command is run at cluster create time, but when new system workloads are added it may need to be run again.
 
-   ```bash
-   kubectl apply -k kubernetes/system
-   ```
+Changes to workloads should be deployed by opening a pull request according to the [GitOps](#gitops) section above.
 
 ## JupyterHub
 
-JupyterHub is a good example of an application using a Helm chart that is ultimately exposed to the outside internet for user access. In general, any non-secret changes to the chart can be accomplished by modifying the chart's `values.yaml` and running the `invoke release` specific to JupyterHub.
-
-```
-poetry run invoke release -f channels/prod.yaml --app=jupyterhub
-```
+JupyterHub is a good example of an application using a Helm chart that is ultimately exposed to the outside internet for user access. In general, any non-secret changes to the chart can be accomplished by modifying the chart's `values.yaml`.
 
 ### Secrets