This page provides instructions on how to install the pipeline, including how to install the required software such as Docker and Kubernetes.
Because the software runs in an isolated software environment, it is quite agnostic about the operating system running on your host machine. We have tested it using MacOS and Ubuntu Linux. We believe that it is probably impossible to run minikube on an M1 (ARM-based) MacBook Pro at the current time, though.
The simplest way to run the pipeline on a single laptop or desktop computer is by using minikube. This is a minimal Kubernetes implementation which can be run on a single machine, but which does not allow multiple machines to be connected into a cluster. It nonetheless provides a test environment which is almost identical to Kubernetes on large clusters, and one of the convenient features of Kubernetes is this ability to scale code from a single laptop to a large cluster with minimal modification.
It is also possible to run an individual worker container in a stand-alone fashion, working on test data which is built into the worker container. In this mode, the EAS interface code mimics communications with an external cluster, but in fact the communication is with a lightweight task database hosted within the container itself using sqlite3.
Instructions here provide a step-by-step guide to installing minikube, and the pipeline itself, on a laptop or desktop machine: minikube-setup.
Instructions here provide a step-by-step guide to running a worker node in a stand-alone fashion: testing.
Instuctions here provide Dominic's notes on running minikube on the Spartan cluster in Cambridge: installation
Instuctions here provide Dominic's notes on running a multi-node Kubernetes cluster on the Spartan cluster in Cambridge: installation
Instuctions here provide Dominic's notes on running minikube on the Iris / OpenStack cluster in Cambridge: installation
Instuctions here provide Dominic's notes on running a multi-node Kubernetes cluster on the Iris / OpenStack cluster in Cambridge: installation
This code is developed and maintained by Dominic Ford, at the Institute of Astronomy, Cambridge.