Move dependencies needed in CI on prod images to separate requirements file.

[KevinMind]

Fixes: mozilla/addons#15191
Fixes: mozilla/addons#15066

Description

Key Changes

1. Docker Configuration Overhaul

• Consolidation of Docker Compose files
• Removal of docker-compose.override.yml (marked as obsolete)
• Introduction of environment variable-based configuration through .env file
• Improved volume management for better data persistence

2. Build System Improvements

• Enhanced BuildKit and Bake implementation for Docker builds
• Multi-stage Dockerfile optimization for better caching and faster builds
• Separation of production and development dependencies

3. Development Environment Changes

• New standardized approach to environment setup using make commands
• Improved handling of the olympia user and permissions
• Better management of mounted volumes and dependencies

Context

Impacts of this PR are:

1. Setup Process

• Developers must now use make setup to initialize their environment
• The .env file becomes crucial for configuration (automatically managed by make)
• More consistent behavior between direct Docker commands and make commands

2. Development Workflow

• Simplified commands: make up and make down handle most common operations
• Better dependency management with mounted volumes
• Improved caching for faster rebuilds
• Clearer separation between production and development environments

3. Potential Migration Steps

• Existing developers may need to run make down to clean up old configurations
• May need to recreate their .env file using make setup
• Might need to rebuild their Docker images to take advantage of the new optimizations

4. Performance Improvements

• Better caching of dependencies through volume mounts
• More efficient Docker builds with optimized layers
• Improved handling of static assets and locales

1. More consistent development environment across team members
2. Better performance in builds and container operations
3. Clearer separation of concerns between development and production
4. More robust error handling and validation
5. Better documentation and standardized workflows

Testing

TODO:

• test generating locales at runtim
• test data load/dump
• test make up CLEAN
• test docker_compose_run
• test COMPOSE_FILE
• test failing web/worker fails make up

Verify make up fails when web doesn't start.

• Modify (one at a time) the web and worker items in the Procfile
• run make up
• Expect it to fail as web is unhealthy

Verify site-static files are available in "prod" mode only

• run make up DOCKER_TARGET=development
• in a web shell expect /data/olympia/site-static to be empty

Verify CI dependencies available only with flag set

• run make up INSTALL_CI_DEPS=True
• in a web shell run ./scripts/check_pip_packages.sh ci
• If you set the flag to false, it should fail with missing dependencies

First create an env in prod mode

make up DOCKER_TARGET=production DOCKER_VERSION=local

Expect no ci or dev dependencies.

now run:

make update_deps_production INSTALL_CI_DEPS=true

Now expect production and ci deps to be installed

FInally install dev deps

make update_deps_production INSTALL_CI_DEPS=true

You can trigger these deps to be installed automatically if you set the argument on make up

make up DOCKER_TARGET=production INCLUDE_CI_DEPS=true

This will install prod and ci deps in the container.

Checklist

• Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
• Successfully verified the change locally.
• The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
• Add before and after screenshots (Only for changes that impact the UI).
• Add or update relevant docs reflecting the changes made.