Proposal: Support canary builds

[gdestuynder]

Support multiple builds that can be selected at runtime by the user (canary builds).
E.g.: NLX previous version , NLX current stable version , NLX edge/next version and NLX maintenance page .

To do this we need:

M1 (POC):

• All NLX code hosted on the CDN
• Lightweight wrapper hosted on the auth0 hosted login page, which can select which version of NLX to load (including the possible hardened maintenance page).
  • Default to "current stable"
  • Support manually changing the default through a LocalStorage entry
  • Setup in a config array, so that values can be changed easily when updating the auth0 hosted login page

M2:

• Allow users to switch to previous/current/edge by setting a preference (which reflects the value in LocalStorage)
• Allow for incremental rollouts over time automatically (current becomes previous, edge becomes current, and new edge is added)
• Collect errors with Sentry and alert us on errors

M3:

• Automatically tell the users there's a problem if Sentry sees too many errors
• Propose to the user to go back to previous in case of too many errors

[hidde]

I love this idea, but having given it some thought over the weekend, I would like to suggest we don't try and do this in the short term (sorry to be that person), for the various reasons listed below.

Makes adding features and QA'ing more complex

Adding features would need more abstract thinking as we'd need to decide in which of our versions to ship and why, whereas currently we just have one place to ship features in.

We currently QA everything before it goes to PROD and one important principle in our process is that we test the exact version that is going to go live. In a reality where we wrap NLX in something that can serve multiple versions, we not only will have to test those versions, but also the wrapper around it.

Canary builds are easier in Firefox than NLX

Because NLX is (a) built 100% on the client side and (b) dependent on a complex and hard to locally reproduce web of RPs, settings, Rules and APIs. Having canary builds in NLX has downsides that canary builds in native software like Firefox does not have.

Re: client side: everything happens in the browser and can come at a cost. To facilitate multiple versions, we either have to ship all users multiple versions, which would have a file size (download time) cost, or we have to load each visitor's version on demand, which would have an extra request and thus performance cost.

Re: complex: we currently have two places to test, one is close to reality (dev), the other is reality (prod). I think to really do this well and in a testable way we would need to create a reality whenever we create a version (e.g. new-feature-1.mozilla.auth0.com, new-feature-2.mozilla.auth0.com with a close to reality set up of RPs and connections). Sounds like a lot of work (and iirc not automatable in our current world).

We already have a place to test out new features

New features are tested in Auth0 Dev. Admittedly there are things to improve here, i.e. make the dev tentant 100% the same as the prod tenant.

We currently don't have new features planned that scream for being beta-tested first. All of the stuff we currently have in the pipeline seems very suitable for testing in dev and shipping in prod.

Other priorities in roadmap

In the near future, we have more important NLX priorities that affect users more, time will be a problem. I'm worried that although the initial set up time doesn't seem much (earlier convos around this yielded some apt solutions), it will prove to cost a lot of time to get the last 20% working.