Can we improve Docker image build time?

[regisb]

Re-building the Open edX Docker images with Tutor is consistently identified as one of the main pain points of platform administrators and developers:

• In the context of continuous integration (CI): https://discuss.openedx.org/t/deploying-open-edx-on-kubernetes-using-helm/8771/14?u=regis
• When deploying new changes to production: https://discuss.openedx.org/t/tutor-images-building-taking-more-than-40-minutes/8998
• When developing locally: https://openedx.slack.com/archives/C04J9GTLHH8/p1676305700731199?thread_ts=1676305358.485799&cid=C04J9GTLHH8

A constructive proposal was made by @ARMBouhali to improve this situation: https://discuss.openedx.org/t/tutor-enhancement-proposal-tep-improve-support-for-build-caching/9036

To address this issue, I think that we should beware of two pitfalls:

a. That improving build-time in one scenario will make the others worse.
b. Implementing ad-hoc solutions that will increase the complexity (and thus: the reliability) of Tutor and Open edX

Before we go any further, we will need to identify/profile the build time required for each step of the building process, in each scenario (development, CI, production).

Then, my intuition is that the best solutions will reside in either:

1. Making better use of the Docker build cache.
2. Making upstream changes to openedx repos (including edx-platform and MFEs).

I am worried that other solutions will cause to stumble into either pitfalls listed above -- though of course we must remain open to alternatives.

Ideas

• Encourage GitHub Actions users to make use of the Docker layer cache (docs). This affects Travis as described here.
• Revert to npm ci instead of npm install (suggested by @bradenmacdonald here)
• Jeff Cohen reduced Open edX image building time in CI from 90 to 15 minutes by moving from GitHub actions to self-hosted runners (see this slack conversation)
• @ghassanmass suggests:
  • relying on a dedicated Docker cache for pip requirements
  • removing useless webpack assets
• @kdmccormick is working on quite a few projects to resolve the problems upstream. Obviously, resolving the problems at the source is the best approach :)

[regisb]

A couple optimizations are in progress here, notably for "cold-start" builds: overhangio/tutor#812

[regisb]

I propose that we prototype solutions and measure their impact in this GitHub project: https://github.com/overhangio/test-docker-build/
For instance, the Docker layer caching in a remote registry is reducing vanilla "cold-start" build-time from 28 minutes to 9 minutes:

• from scratch: https://github.com/overhangio/test-docker-build/actions/runs/4447024271/jobs/7847560759
• with Docker registry cache: https://github.com/overhangio/test-docker-build/actions/runs/4447024271/jobs/7847560948

[kdmccormick]

Fantastic!! I will certainly be adding some tests there later.