Skip to content

Commit

Permalink
Slow worker restart
Browse files Browse the repository at this point in the history
  • Loading branch information
aequitas authored and mxsasha committed Nov 11, 2024
1 parent b699aa1 commit 0094ca3
Show file tree
Hide file tree
Showing 2 changed files with 25 additions and 0 deletions.
24 changes: 24 additions & 0 deletions docker/cron-docker/periodic/daily/restart_slow_worker
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#!/bin/sh

# restart slow worker every day to prevent slow memory leaks
# as the slow worker can run very long tasks (eg: report generation)
# we first start a new container before stopping the previous one

set -e

cd /opt/Internet.nl

SERVICE=worker-slow
REPLICAS=$WORKER_SLOW_REPLICAS
COMPOSE_CMD="docker compose --env-file=docker/defaults.env --env-file=docker/host.env --env-file=docker/local.env"

OLD_CONTAINERS=$($COMPOSE_CMD ps --format "{{ .Name }}"|grep "$SERVICE")

# bring up new containers, wait until healthy
$COMPOSE_CMD up --no-deps --no-recreate --wait --scale="$SERVICE=$(($REPLICAS*2))" "$SERVICE"

# graceful shutdown and remove old containers
docker rm --force "$OLD_CONTAINERS"

# restore replica number to original
$COMPOSE_CMD scale $SERVICE=$REPLICAS
1 change: 1 addition & 0 deletions docker/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -734,6 +734,7 @@ services:
command: crond -f -d7 -c /etc/crontabs-docker
environment:
- AUTO_UPDATE_TO
- WORKER_SLOW_REPLICAS

restart: unless-stopped
logging:
Expand Down

0 comments on commit 0094ca3

Please sign in to comment.