Verify deployments before serving traffic

[theseanything]

Currently, we promote a deployment to an environment (e.g. staging, production), which involves serving live traffic to the new version as soon as pods are available. Then post rollout, we then run the smokey tests to verify the deployment. If the smoke tests fail we send a notification, however do not rollback if there's a failure with the smoke tests. This is okay for integration or staging, as it prevents the automatic promotion of the deployment to production. However doesn't help stop issues, if the failures are only seen in production.

We could use progressive delivery (such as Argo Rollouts) to run verifications (smoke tests) on application before serving live traffic and allowing us to automatically rollback to the previous version.

[theseanything]

As requests are proxied via router from Frontend apps, this makes it difficult to test the preview service. Could use header-based traffic management, however that would require a service mesh.