Merge branch 'tai'

- Update settings for compatibility with old Django versions and consistency.
  - Use tuple instead of list for template dirs, context processors, loaders.
  - Define only current version settings (e.g. `TEMPLATES`, `MIDDLEWARE`) and then add compatibility for earlier Django versions at the bottom of `ixc_django_docker/settings/__init__.py`.
  - Fix `debug` and `request` context processor paths for Django <1.8.
- `setup.sh` and `setup-tests.sh` no longer automatically run any npm script, even if `SETUP_NPM_RUN` or `SETUP_TESTS_NPM_RUN` is defined.

  Instead, export `SETUP_COMMAND` and `SETUP_TESTS_COMMAND`, which can be any shell command instead of the name of an npm script.

  This allows us to execute any combination of npm scripts in sequence (via `run-s`) or parallel (via `run-p`), without cluttering up `package.json` with near identical wrapper scripts.
- Store nginx and supervisor pid files in a dedicated `run` directory, optionally namespaced by `$CONTEXT`. This allows multiple instances of nginx and supervisor (e.g. gunicorn, twisted) to run with a shared `var` volume.
- Use `.$(uname)` as a filename suffix to avoid conflicts between native (macOS) and Docker (Linux) environments.
- Apply migrations last via `setup.sh`. Because the database is shared state and there might be backwards incompatible changes. This should reduce the duration of any potential downtime before recreating old containers, caused by a slow `SETUP_COMMAND`.
- Use `python -m pip` (best practice) to be more sure that we are installing into the right environment.
- Fix `XViewMiddleware` name in Django <1.8.
- Update datadog env vars. `DATADOG_ENV` -> `DD_ENV`, per current ddtrace convention.
- Delete duplicate/vestigial Django >=1.8 only `MIDDLEWARE` and `TEMPLATES` settings on earlier versions.
- Fix `npm-install.sh` when run interactively inside containers as root.
- Use a unique name for the `django-compressor` and `ixc-whitenoise` manifest files, so we can run new and old versions of a codebase side by side with a shared volume.

CHANGES.md

@@ -1,6 +1,65 @@
 Breaking and notable changes
 ===
 
+6 April 2021
+---
+
+### Breaking
+
+- `django-compressor` and `ixc-whitenoise` will now include a `.$GIT_COMMIT` suffix in their manifest filename, so we can run new and old versions of a codebase side by side with a shared volume.
+
+  When upgrading:
+
+  - To disable this behaviour for local development, where you do not want to re-compress or collect static files for every commit, add the following to your `.env.develop.secret` file:
+
+        # Configure static file manifests. Comment out to save versioned files with a
+        # `$GIT_COMMIT` suffix.
+        export COMPRESS_OFFLINE_MANIFEST='manifest.json'
+        export STATICFILES_MANIFEST='staticfiles.json'
+
+13 November 2020
+---
+
+### Breaking
+
+- Use tuple instead of list for template dirs, context processors, loaders. Check if you are using `+= [..., ]` and change to `(..., )`
+
+17 September 2020
+---
+
+### Breaking
+
+- `setup.sh` and `setup-tests.sh` no longer automatically run any npm script, even if `SETUP_NPM_RUN` or `SETUP_TESTS_NPM_RUN` is defined.
+
+  Instead, export `SETUP_COMMAND` and `SETUP_TESTS_COMMAND`, which can be any shell command instead of the name of an npm script.
+
+  This allows us to execute any combination of npm scripts in sequence (via `run-s`) or parallel (via `run-p`), without cluttering up `package.json` with near identical wrapper scripts.
+
+  We can also execute any shell script or bash command.
+
+  When upgrading:
+
+  - Add `export SETUP_COMMAND='npm run -l build --if-present'` and `export SETUP_TESTS_COMMAND='npm run -l build --if-present'` to your `.env` file to restore the old behaviour.
+
+  - Or add `export SETUP_COMMAND='...'` to your `.env` file, as required.
+
+- The `COMPRESS_OFFLINE` setting is now `True`. Offline compression is required when using WhiteNoise without autorefresh, which is not used in production because it is only intended for development and has had a serious security issue in the past.
+
+  When upgrading:
+
+  - Add `"compress": "manage.py compress --verbosity=0"` to the `package.json` file.
+  - Add `export SETUP_COMMAND='run-s -l compress ...'` to your `.env` file.
+  - Add `run-s -l collectstatic compress` to `Dockerfile`.
+  - Confirm that your `{% compress %}` blocks do not have any dynamic content (changes with context), or set `IXC_COMPRESSOR_REQUEST`, `IXC_COMPRESSOR_GLOBAL_CONTEXT` and `IXC_COMPRESSOR_OPTIONAL_CONTEXT` accordingly.
+
+### Notable
+
+- Supervisor no longer wraps its programs with `entrypoint.sh`. This was a convenience during troubleshooting so we could edit a dotenv file and restart the program, without having to restart supervisor or the container itself.
+
+  Now that `entrypoint.sh` is optional, it is no longer appropriate to always wrap programs with it.
+
+  If you are still using `entrypoint.sh`, in a pinch you can live edit the supervisor template to reinstate the wrapper.
+
 8 September 2020
 ---
 

docs/commands.rst

@@ -26,7 +26,7 @@ Actions:
 
 * Create virtualenv for project
 * (Re)install Python requirements from ``requirements.txt`` when this file
-  changes, as detected via hash file ``requirements.txt.md5``
+  changes, as detected via hash file ``requirements.txt.md5.$(uname)``
 * Source bootstrap environment variables from ``.env.local``
 * Exec ``entrypoint.sh`` with command arguments passed to this script if any,
   otherwise with ``setup-django.sh bash.sh``

docs/how-to-upgrade.md

@@ -185,7 +185,7 @@ Test your changes:
 
 Things to watch out for:
 
-- Run `rm -rf var` and `find . -name "*.md5" -delete` to reset the environment and start over.
+- Run `rm -rf var` and `find . -name "*.md5.$(uname)" -delete` to reset the environment and start over.
 
 - Upgrading Node.js can require that we also upgrade dependencies in `package.json`. If you see any npm errors during setup:
 

ixc_django_docker/bin/bash.sh

@@ -50,7 +50,7 @@ EOF
 
 	# Compare git commit and print reminder if setup has not completed successfully.
 	GIT_COMMIT="$(git-commit.sh)"
-	if [[ "$GIT_COMMIT" != $(cat "$PROJECT_DIR/var/setup-git-commit.txt" 2>&1) ]]; then
+	if [[ "$GIT_COMMIT" != $(cat "$PROJECT_DIR/var/setup-git-commit.txt.$(uname)" 2>&1) ]]; then
 		>&2 cat <<EOF
 WARNING: Setup is not complete for git commit: '$GIT_COMMIT'
          Run 'setup.sh' manually.

ixc_django_docker/bin/bower-install.sh

@@ -20,13 +20,16 @@ if [[ ! -s bower.json ]]; then
 EOF
 fi
 
-touch bower.json.md5
+UNAME="$(uname)"
 
-if [[ ! -s bower.json.md5 ]] || ! md5sum --status -c bower.json.md5 > /dev/null 2>&1; then
+touch "bower.json.md5.$UNAME"
+
+if [[ ! -s "bower.json.md5.$UNAME" ]] || ! md5sum --status -c "bower.json.md5.$UNAME" > /dev/null 2>&1; then
 	echo "Bower components in '$DIR' directory are out of date, 'bower.json' has been updated."
 	if [[ -d bower_components ]]; then
 		rm -rf bower_components
 	fi
 	bower install --allow-root
-	md5sum bower.json > bower.json.md5
+	rm -f bower.json.md5.*  # 'bower_components' is shared, if we rebuild for one platform, other platforms become invalid
+	md5sum bower.json > "bower.json.md5.$UNAME"
 fi

ixc_django_docker/bin/ddtrace.sh

@@ -2,11 +2,12 @@
 
 set -e
 
-export DATADOG_ENV="${DATADOG_ENV:-$PROJECT_NAME-${DATADOG_ENV:-$DOTENV}}"
+export DD_ENV="${DATADOG_ENV:-$PROJECT_NAME-${DATADOG_ENV:-$DOTENV}}"
 export DD_TRACE_ANALYTICS_ENABLED='true'
 
-# For ddtrace < 0.25.
-export DD_ANALYTICS_ENABLED="$DD_TRACE_ANALYTICS_ENABLED"
+# Legacy.
+export DATADOG_ENV="${DD_ENV}"
+export DD_ANALYTICS_ENABLED="${DD_TRACE_ANALYTICS_ENABLED}"
 
 echo 'Run command via Datadog Trace Client.'
 

ixc_django_docker/bin/migrate.sh

@@ -22,7 +22,9 @@ else
 	manage.py showmigrations > "$DIR/migrate.txt" 2> /dev/null
 fi
 
-if [[ ! -s "$DIR/migrate.txt.md5" ]] || ! md5sum --status -c "$DIR/migrate.txt.md5" > /dev/null 2>&1; then
+UNAME="$(uname)"
+
+if [[ ! -s "$DIR/migrate.txt.md5.$UNAME" ]] || ! md5sum --status -c "$DIR/migrate.txt.md5.$UNAME" > /dev/null 2>&1; then
 	echo 'Migrations are out of date.'
 
 	# Skip initial migration if all tables created by the initial migration
@@ -39,5 +41,5 @@ if [[ ! -s "$DIR/migrate.txt.md5" ]] || ! md5sum --status -c "$DIR/migrate.txt.m
 		manage.py showmigrations > "$DIR/migrate.txt" 2> /dev/null
 	fi
 
-	md5sum "$DIR/migrate.txt" > "$DIR/migrate.txt.md5"
+	md5sum "$DIR/migrate.txt" > "$DIR/migrate.txt.md5.$UNAME"
 fi

ixc_django_docker/bin/nginx.sh

@@ -2,19 +2,21 @@
 
 set -e
 
-# Create context specific config directory.
+# Create context specific config and run directories.
 if [[ -z "${CONTEXT+1}" ]]; then
-	DIR="$PROJECT_DIR/var/etc"
+	ETC_DIR="$PROJECT_DIR/var/etc"
+	export RUN_DIR="$PROJECT_DIR/var/run"
 else
-	DIR="$PROJECT_DIR/var/etc/$CONTEXT"
+	ETC_DIR="$PROJECT_DIR/var/etc/$CONTEXT"
+	export RUN_DIR="$PROJECT_DIR/var/run/$CONTEXT"
 fi
-mkdir -p "$DIR"
+mkdir -p "$ETC_DIR" "$RUN_DIR"
 
 # Generate htpasswd file if credentials are set.
 if [[ -n "$NGINX_BASIC_AUTH" ]]; then
 	# Split on `:`.
 	IFS=: read BASIC_AUTH_USERNAME BASIC_AUTH_PASSWORD <<< "$NGINX_BASIC_AUTH"
-	echo "$BASIC_AUTH_PASSWORD" | htpasswd -ci "$DIR/nginx.htpasswd" "$BASIC_AUTH_USERNAME"
+	echo "$BASIC_AUTH_PASSWORD" | htpasswd -ci "$ETC_DIR/nginx.htpasswd" "$BASIC_AUTH_USERNAME"
 fi
 
 # Default environment variables for config template.
@@ -26,8 +28,8 @@ export NGINX_WORKER_PROCESSES="${NGINX_WORKER_PROCESSES:-1}"
 export SUPERVISOR_PROGRAM_NUMPROCS="${SUPERVISOR_PROGRAM_NUMPROCS:-1}"
 
 # Render nginx config template.
-dockerize -template "${NGINX_TMPL_CONF:-$IXC_DJANGO_DOCKER_DIR/etc/nginx.tmpl.conf}:$DIR/nginx.conf"
+dockerize -template "${NGINX_TMPL_CONF:-$IXC_DJANGO_DOCKER_DIR/etc/nginx.tmpl.conf}:$ETC_DIR/nginx.conf"
 
 # Set `error_log` via command line to avoid a permissions error when run as an
 # unprivileged user. See: https://stackoverflow.com/a/24423319
-exec nginx -c "$DIR/nginx.conf" -g "error_log /dev/stderr;" "$@"
+exec nginx -c "$ETC_DIR/nginx.conf" -g "error_log /dev/stderr;" "$@"

ixc_django_docker/bin/npm-install.sh

@@ -20,9 +20,11 @@ if [[ ! -s package.json ]]; then
 EOF
 fi
 
-touch package.json.md5
+UNAME="$(uname)"
 
-if [[ ! -s package.json.md5 ]] || ! md5sum --status -c package.json.md5 > /dev/null 2>&1; then
+touch "package.json.md5.$UNAME"
+
+if [[ ! -s "package.json.md5.$UNAME" ]] || ! md5sum --status -c "package.json.md5.$UNAME" > /dev/null 2>&1; then
 	echo "Node modules in '$DIR' directory are out of date, 'package.json' has been updated."
 	if [[ -d node_modules ]]; then
 		rm -rf node_modules
@@ -34,5 +36,6 @@ if [[ ! -s package.json.md5 ]] || ! md5sum --status -c package.json.md5 > /dev/n
 	else
 		npm install --unsafe-perm
 	fi
-	md5sum package.json > package.json.md5
+	rm -f package.json.md5.*  # 'node_modules' is shared, if we rebuild for one platform, other platforms become invalid
+	md5sum package.json > "package.json.md5.$UNAME"
 fi

ixc_django_docker/bin/pip-install.sh

@@ -9,18 +9,20 @@ DIR="${1:-$PWD}"
 mkdir -p "$DIR"
 cd "$DIR"
 
+UNAME="$(uname)"
+
 if [[ -f requirements.txt ]]; then
-	if [[ ! -s requirements.txt.md5 ]] || ! md5sum --status -c requirements.txt.md5 > /dev/null 2>&1; then
+	if [[ ! -s "requirements.txt.md5.$UNAME" ]] || ! md5sum --status -c "requirements.txt.md5.$UNAME" > /dev/null 2>&1; then
 		echo "Python requirements in '$DIR' directory are out of date, 'requirements.txt' has been updated."
 		pip install -r requirements.txt
-		md5sum requirements.txt > requirements.txt.md5
+		md5sum requirements.txt > "requirements.txt.md5.$UNAME"
 	fi
 fi
 
 if [[ -f requirements-local.txt ]]; then
-	if [[ ! -s requirements-local.txt.md5 ]] || ! md5sum --status -c requirements-local.txt.md5 > /dev/null 2>&1; then
+	if [[ ! -s "requirements-local.txt.md5.$UNAME" ]] || ! md5sum --status -c "requirements-local.txt.md5.$UNAME" > /dev/null 2>&1; then
 		echo "Python requirements in '$DIR' directory are out of date, 'requirements-local.txt' has been updated."
 		pip install -r requirements-local.txt
-		md5sum requirements-local.txt > requirements-local.txt.md5
+		md5sum requirements-local.txt > "requirements-local.txt.md5.$UNAME"
 	fi
 fi

ixc_django_docker/bin/setup-tests.sh

@@ -27,10 +27,10 @@ if [[ "$DJANGO_VERSION_LESS_THAN_1_7" == 'True' ]]; then
 fi
 [[ -z "$QUICK" ]] && manage.py migrate --noinput
 
-# Run 'setup' script.
-if [[ -z "${QUICK+1}" ]]; then
-    echo "Executing: npm run ${SETUP_TESTS_NPM_RUN:-setup}..."
-    npm run "${SETUP_TESTS_NPM_RUN:-setup}" --if-present
+# Execute setup tests command.
+if [[ -z "${QUICK+1}" && -n "${SETUP_TESTS_COMMAND+1}" ]]; then
+	echo "Executing setup tests command: ${SETUP_TESTS_COMMAND}"
+	bash -c "${SETUP_TESTS_COMMAND}"
 fi
 
 # Execute command, only if not sourced.

ixc_django_docker/bin/setup-wait.sh

@@ -9,7 +9,7 @@ GIT_COMMIT="$(git-commit.sh)"
 # Wait for setup.
 COUNT=0
 while true; do
-	if [[ "$GIT_COMMIT" == $(cat "$PROJECT_DIR/var/setup-git-commit.txt" 2>&1) ]]; then
+	if [[ "$GIT_COMMIT" == $(cat "$PROJECT_DIR/var/setup-git-commit.txt.$(uname)" 2>&1) ]]; then
 		break
 	fi
 	if [[ "$COUNT" == 0 ]]; then

ixc_django_docker/bin/setup.sh

@@ -12,9 +12,12 @@ fi
 
 mkdir -p "$PROJECT_DIR/var"
 
+UNAME="$(uname)"
+
 if [[ -n "${SETUP_FORCE+1}" ]]; then
-	>&2 echo 'SETUP_FORCE is set. Delete "*.md5" files.'
-	find . -name "*.md5" -delete
+	>&2 echo "SETUP_FORCE is set. Delete '*.md5.$UNAME' files."
+	find . -name "*.md5.$UNAME" -delete
+	rm -f "$PROJECT_DIR/var/migrate.txt.$UNAME"
 fi
 
 # Install Node modules.
@@ -29,16 +32,20 @@ pip-install.sh "$PROJECT_DIR"
 # Create a database.
 setup-postgres.sh
 
-# Apply migrations.
-migrate.sh "$PROJECT_DIR/var"
+# Execute setup command.
+if [[ -n "${SETUP_COMMAND+1}" ]]; then
+	echo "Executing setup command: ${SETUP_COMMAND}"
+	bash -c "${SETUP_COMMAND}"
+fi
 
-# Run 'setup' script.
-echo "Executing: npm run ${SETUP_NPM_RUN:-setup}..."
-npm run "${SETUP_NPM_RUN:-setup}" --if-present
+# Apply migrations last. Because the database is shared state and there might be
+# backwards incompatible changes. This should reduce the duration of any potential
+# downtime before recreating old containers, caused by a slow `SETUP_COMMAND` (above).
+migrate.sh "$PROJECT_DIR/var"
 
 # Save git commit.
-echo "$(git-commit.sh)" > "$PROJECT_DIR/var/setup-git-commit.txt"
-echo "Updated '$PROJECT_DIR/var/setup-git-commit.txt' ($(cat $PROJECT_DIR/var/setup-git-commit.txt))"
+echo "$(git-commit.sh)" > "$PROJECT_DIR/var/setup-git-commit.txt.$UNAME"
+echo "Updated '$PROJECT_DIR/var/setup-git-commit.txt.$UNAME' ($(cat $PROJECT_DIR/var/setup-git-commit.txt.$UNAME))"
 
 # Execute command.
 exec "$@"

ixc_django_docker/bin/supervisor.sh

@@ -4,17 +4,19 @@
 
 set -e
 
-# Create context specific config directory.
+# Create context specific config and run directories.
 if [[ -z "${CONTEXT+1}" ]]; then
-	DIR="$PROJECT_DIR/var/etc"
+	ETC_DIR="$PROJECT_DIR/var/etc"
+	export RUN_DIR="$PROJECT_DIR/var/run"
 else
-	DIR="$PROJECT_DIR/var/etc/$CONTEXT"
+	ETC_DIR="$PROJECT_DIR/var/etc/$CONTEXT"
+	export RUN_DIR="$PROJECT_DIR/var/run/$CONTEXT"
 fi
-mkdir -p "$DIR"
+mkdir -p "$ETC_DIR" "$RUN_DIR"
 
 # Get a random-ish port in the 59000-59999 range.
 # See: https://github.com/rfk/django-supervisor/blob/be2013c4826ae49730664b359ee285fa03b16c09/djsupervisor/config.py#L107-L111
-export SUPERVISOR_PORT="${SUPERVISOR_PORT:-59$(python.sh -c "import hashlib; print('%03d' % (int(hashlib.md5(u'$DIR'.encode('utf-8')).hexdigest()[:3], 16) % 1000));")}"
+export SUPERVISOR_PORT="${SUPERVISOR_PORT:-59$(python.sh -c "import hashlib; print('%03d' % (int(hashlib.md5(u'$ETC_DIR'.encode('utf-8')).hexdigest()[:3], 16) % 1000));")}"
 
 # Default environment variables for config template.
 export NGINX_PROXY_PORT="${NGINX_PROXY_PORT:-8080}"
@@ -23,11 +25,11 @@ export SUPERVISOR_PROGRAM_ENVIRONMENT="${SUPERVISOR_PROGRAM_ENVIRONMENT:-}"
 export SUPERVISOR_PROGRAM_NUMPROCS="${SUPERVISOR_PROGRAM_NUMPROCS:-1}"
 
 # Render config templates.
-dockerize -template "${SUPERVISOR_TMPL_CONF:-$IXC_DJANGO_DOCKER_DIR/etc/supervisord.tmpl.conf}:$DIR/supervisord.conf"
-dockerize -template "${SUPERVISOR_INCLUDE_TMPL_CONF:-$IXC_DJANGO_DOCKER_DIR/etc/supervisord.include.tmpl.conf}:$DIR/supervisord.include.conf"
+dockerize -template "${SUPERVISOR_TMPL_CONF:-$IXC_DJANGO_DOCKER_DIR/etc/supervisord.tmpl.conf}:$ETC_DIR/supervisord.conf"
+dockerize -template "${SUPERVISOR_INCLUDE_TMPL_CONF:-$IXC_DJANGO_DOCKER_DIR/etc/supervisord.include.tmpl.conf}:$ETC_DIR/supervisord.include.conf"
 
 if [[ -z "$@" ]]; then
-	exec supervisord --configuration "$DIR/supervisord.conf"
+	exec supervisord --configuration "$ETC_DIR/supervisord.conf"
 else
-	exec supervisorctl --configuration "$DIR/supervisord.conf" --serverurl "http://localhost:$SUPERVISOR_PORT" "$@"
+	exec supervisorctl --configuration "$ETC_DIR/supervisord.conf" --serverurl "http://localhost:$SUPERVISOR_PORT" "$@"
 fi

ixc_django_docker/etc/nginx.tmpl.conf

@@ -90,5 +90,5 @@ http {
     {{- end }}
 }
 
-pid               {{ .Env.PROJECT_DIR }}/var/run/nginx.pid;
+pid               {{ .Env.RUN_DIR }}/nginx.pid;
 worker_processes  {{ .Env.NGINX_WORKER_PROCESSES }};

ixc_django_docker/etc/supervisord.include.tmpl.conf

@@ -1,7 +1,7 @@
 [program:nginx]
 autorestart = true
 autostart = true
-command = prefix-logs.sh entrypoint.sh nginx.sh
+command = prefix-logs.sh nginx.sh
 stderr_logfile = /dev/stderr
 stderr_logfile_maxbytes = 0
 stdout_logfile = /dev/stdout
@@ -11,7 +11,7 @@ stopasgroup = true
 [program:{{ .Env.SUPERVISOR_PROGRAM }}]
 autorestart = true
 autostart = true
-command = prefix-logs.sh entrypoint.sh {{ .Env.SUPERVISOR_PROGRAM }}.sh
+command = prefix-logs.sh {{ .Env.SUPERVISOR_PROGRAM }}.sh
 environment = SUPERVISOR_PROCESS_NUM="%(process_num)d",{{ .Env.SUPERVISOR_PROGRAM_ENVIRONMENT }}
 numprocs = {{ .Env.SUPERVISOR_PROGRAM_NUMPROCS }}
 process_name = {{ .Env.SUPERVISOR_PROGRAM }}_%(process_num)d

ixc_django_docker/etc/supervisord.tmpl.conf

@@ -1,7 +1,7 @@
 [supervisord]
 logfile = /dev/null
 nodaemon = true
-pidfile = {{ .Env.PROJECT_DIR }}/var/run/supervisord.pid
+pidfile = {{ .Env.RUN_DIR }}/supervisord.pid
 
 [inet_http_server]
 port = 127.0.0.1:{{ .Env.SUPERVISOR_PORT }}

ixc_django_docker/settings/__init__.py

@@ -65,6 +65,9 @@ def _(module, from_dir):
     PROJECT_SETTINGS.append(
         optional(os.path.join(PROJECT_SETTINGS_DIR, OVERRIDE_SETTINGS)))
 
+# Add Django compatibility to base settings.
+BASE_SETTINGS.append('django_compat.py')
+
 # Local settings.
 PROJECT_SETTINGS.append(
     optional(os.path.join(PROJECT_SETTINGS_DIR, 'local.py')))