How to continuously deploy Bob itself?

[lispyclouds]

Currently Bob executes the build on one of the many nodes in a build cluster. During a deployment this node will be taken down along with its potentially running build(s).

We need to come up with a way to continuously deploy new versions of Bob while not affecting the running builds.

[lispyclouds]

One approach:
Given one or more nodes of Bob wherein one or more of them are executing build(s) this approach requires external orchestration to maintain zero disruptions. Here node is the common term used for a container or a full instance.

Method:

• Build an endpoint in Bob which tells that the node is executing a build currently. This is to be called by the external orchestration tool.
• Build another endpoint which tells a node not to accept any new builds. Also to be called via the tool.
• the update tool will iterate in parallel over the nodes and for each of the non busy node:
  • it will bring up a new node with the new code
  • when the liveness check passes for the new node, it will ask the old node to not take builds
  • add the new node to the load balancer
  • kill the old node
• this continues till no old nodes are left

Caveats:

• For a single node setup this will block till the build completes in the old node if its running something.
• Some discovery mechanism is needed for listing the node, if using something like Kubernetes, this is trivial