What would happen if we unpinned the python base image from all the daap- lambdas?

[MatMoore]

Context

The ingestion architecture consists of multiple lambda containers that depend on a shared base image with its own ECR repository.

There was a previous discussion about ways we can reduce the number of deploy steps, but I want to focus this one on one specific suggestion that is potentially easy to implement, providing that we understand the implications on the way we work.

While we've been working on these lambdas, we've found that:

• Lambdas depend heavily on functionality in the base image, and we tend to update both at the same frequency
• We often introduce functionality in a lambda, and then extract code into the base later on
• The base image pulls in most of our core dependencies like boto3

The requirement to pin versions at multiple layers (requirements.txt + lambda dockerfiles) is creating extra work during development, and it may be unnecessary.

Proposal

• Unpin the base image in all of the lambdas
• When a lambda image is built, the latest base image will be used
• CI needs to run tests against every lambda container when the base image changes
• On merge, the base image must be built before the lambda images (if it's not already)

We could mitigate risk by restricting this to one or two lambdas before rolling it out to the rest of them.

Guardrails

This change must not:

• break the integrity of deploys (i.e. the artefacts we deploy do not change as we roll them out from dev->test->preprod->prod)
• introduce any way for untrusted changes to enter our supply chain without a code review from a team member

It should also ensure that the main branch of the monorepo is always in a deployable state; i.e. if we make a small change to the code, the build will not break for unrelated reasons.

Counterproposal - what we will do if we keep pinning the base image

• Dependabot will raise PRs to update each lambda to the latest version when the base is updated - about 12 PRs per base update that will require approval & merging
  • At the moment rolling out the versions also demands an additional PR & deploy of modernisation-platform-environments, but we would look to eliminate this step regardless (future spike)

Consequences of this proposal

1. The base image and lambdas can be changed in one go, instead of creating separate PRs, reducing the cost of refactoring
2. When we update the base image, we are forced to update the code that depends on it straight away, rather than finding out later on that something is broken
3. When we merge dependabots for 3rd party libraries, we can update everything in one PR against the monorepo, instead of doing it in multiple steps
4. Versioning is still applied to the container lambdas themselves. We still have the ability to roll back to earlier versions.

Relation to other work

• We have a spike to investigate the Serverless Application Model (SAM), alongside AWS Power Tools. If we adopt this model, then we would manage the whole API as one artefact. Adopting this would be complementary to the current proposal, as we would be able to work on API endpoint definitions, the lambdas that they call, and the shared code they depend on together as one application.