"Reflector" Distribution

[michaelquigley]

Implement a distribution stage, which can do an initial upload to a "distribution bastion" (created as part of the model), which is then used as a high-bandwidth staging point to distribute components to hosts in the model.

When on low-bandwidth links, this will drastically improve model update performance. The components will only transit the low-bandwidth link once, regardless of model host count.

[michaelquigley]

Instead of dedicating a host for this, we can probably re-use the hosts in the model. Maybe we end up nominating one host in each region to be the reflector host. The local developer outbound is used to "seed" the model into one host in one region, and then that host is used to seed the other hosts in-region, and also seed the other regions (which are then reflected to hosts in their same region).

That'd probably be a lot more efficient than just setting up a distribution bastion somewhere in the cloud.

[plorenz]

The rsync code now does this. Once the first rsync to a host is done, we run rsyncs in parallel from places it's been deployed to hosts that still need to be synced.