RFC: Director subscriptions

[nigoroll]

Layering directors works at the moment by adding references to a director as a backend of another director.

This is all good and well, but there is an additional use case idea which could be powerful and of which I was reminded by a question: Directors currently combine the functions of discovering backends and distributing requests. Most directors use "static" discovery (configuration via VCL), but "dynamic" directors don't and, for example, the dynamic director gets its backend from DNS or other name services.

Currently, the dynamic director needs to (re)implement all distribution functions it wants to offer. But would it not be nice to be able to split them, and use the dynamic director's discovery for arbitrary load balancing directors, such as, in particular, shard?

This ticket is to ask for opinions about this idea. I am aware that a proper proposal should come in the form of a VIP, but I would first like to gather feedback on the viability of the idea and also the potential design.

For now, I would think that we could have something like the following vcl strawman:

sub vcl_init {
  new discovery = dynamic.backend("foo.bar");
  new shard = directors.shard();
  shard.subscribe(discovery.backend());
  shard.parametrize("^[a-f]\.", rampup=5m);
}

This would configure a dynamic director discovery and a shard director, which subscribes to backend updates of discovery. A .parametrize() method could be used to add additional parameters to backends learned from the subscription, for example by a regex of their name.