Add images for celery, rabbitmq etc. for use in production

[clbarnes]

No description provided.

[clbarnes]

This would probably need a separate repo as it's doing a different job to the docker config in the CATMAID repo. In particular, we'd probably want to be more Docker-y, i.e. running a single service per container rather than treating it like a VM and having supervisor handle subprocesses. Therefore, we'd need

Existing:

• postgis
• nginx
• uwsgi

New:

• daphne
• rabbitmq
• celery
• postgres backup? e.g. https://github.com/prodrigestivill/docker-postgres-backup-local

Daphne and celery could be run using the same docker image as uwsgi, just with a different entry point. They can have a relatively simple set of environment variables as docker-compose applications are by default started on their own network bridge, so all the exposed ports can be identical for a whole cloud of compose applications and each would only differ by which ports they published, which could be set with an environment variable or .env file. Because we have a separate rabbitmq instance for each instance, we don't need to bother with namespaces or different users.

With any luck, this will primarily be a case of refactoring the catmaid entry point script to split out the different elements and calling them with different CMDs (in the same way that the special command platform is handled now).

Lastly, we'd want to roll the R environment into the container. This is currently a problem because of the rate limits for downloading all the required R packages from github (could any come from CRAN?), unless we also used a personal access token.

Possible additional volumes:

• template brains for R (~/.local/share/rpkg-nat.templatebrains), per-instance
• cached skeletons/ point clouds for NBLAST
• log files (keep them in the compose repo)
• media?

[clbarnes]

I think it breaks down like this:

• rabbitmq
• nginx
• postgres/postGIS
• DB backup (just running cron or similar), unless this could trivially be rolled into the postgres container
• "startup" image, with CATMAID code, python environment, R environment. Its CMD would generate the static files in a volume, handle database migrations, install the data files for transformations in a volume, create a run file in a volume, and then quit.
• "uWSGI", which uses "startup" as a parent image and docker-compose dependency and doesn't start until the "startup" run file is created
• "daphne", as above
• "celery", as above

Volumes would be

• postgres data (postgres)
• logs (all)
• postgres dumps/ ingest (postgres)
• data files for transformations
• static files, hdf5, media etc.
• run files etc.

Either the postgres or "startup" container would have CMD which could load a pg_dump (preferred over a pg_dumpall). As settings would need to respond to environment variables, they'd need to be generated in every catmaid image.

If we set up the R environment first (as python deps change more frequently) and use a catmaid org PAT in CI, we wouldn't duplicate any downloads and the rate limits should be fine. The layered file system should mean that quadruplicating catmaid images doesn't take up any extra space. The nginx container routes external requests to the 3 web images, static file directories, and so the host nginx should just have to route to the nginx container.

Does this get complicated in that the internal django config needs to know the route used to reach the host nginx? Or does django just need to know the setup with the nginx container? I suppose it can be controlled with environment variables, so you'd just need a .env file which matches with the host's nginx.