Allow implementations to extend discovery

[justinsb]

Discovery can be extended arbitrary, but only in response to operator
configuration. Implementations are encouraged to adopt mechanisms similar to
the existing meta-discovery mechanism, and implementations must support the
public mechanisms by default.

[justinsb]

For discussion: allow implementations to define alternative discovery mechanisms that support use-cases such as private repositories and custom mirrors. As long as these are opt-in, we need not define them in the spec, but we should probably do so as it becomes more clear exactly what they look like.

If we like the idea here (if not the implementation!) then I will try to implement private-repositories on S3 for rkt, to verify that this is realistic (and see what problems are hit in practice)

[krobertson]

How does something like the S3 use case differ from having and endpoint that returns a 3xx URL to the S3 location? Or using an http proxy to handle mirroring?

[justinsb]

@krobertson A 3xx URL would still require maintaining a server with a valid SSL certificate. (Though this is a great idea for offloading traffic from that server). Also, for repositories that require auth, S3 has an authentication mechanism and it probably makes sense to just leverage that.

For public image repositories, I would imagine they would just use the existing discovery mechanism but specify the "awkward" S3 url as their backing-store i.e. https://s3.amazonaws.com/mybucket/... So I don't think they need explicit S3 support.

An http proxy is a nice idea, though I think that the adaptation to support S3 is relatively minor and would mean you wouldn't need the extra moving-piece.