load balanced spawner

[oliver-sanders]

closes #137

Turned out that having a spawner as a plugin to a spawner was a little tricky as there's all sorts of setup Jupyterhub does.

We could also use the existing SSH spawner plugin by subclassing and overriding the choose_remote_host method and bolting on some bits. This plugin is a little over the top for us as it does file transfers etc.

Simple solution, runs the command through an SSH connection, similar to how cylc play does it.

Note: JupyterHub spawners run as the user so this spawner, the SSH connection and the UIS is spawns are all in userland even if the hub is not.

Here's the configuration to use it:

c.JupyterHub.spawner_class = 'cylc.uiserver.distributed_spawner.DistributedSpawner'
c.DistributedSpawner.hosts = ['localhost']  

Unfortunately the UIS doesn't like being spawned onto other hosts, I think we need to patch the handlers with the location of the Hub. I've tried patching the MainHandler but that leaves me with the following error (Hub stdout):

tornado.application WARNING  oauth state '<key>' != None

So ATM this can't be used to spawn UIS on other hosts which makes it a bit of a chocolate teapot.

Been looking at the JupyterHub single-user thinggy trying to figure out how to fix it. They configure their handlers quite differently (redirect rather than render) and seem to provide hub-host and hub-prefix CLI opts?

https://github.com/jupyterhub/jupyterhub/blob/3fec19d191119e4203c243f44c43e74c8fe62f04/jupyterhub/singleuser/mixins.py#L141-L155

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).

• Appropriate tests are included (unit and/or functional).
• Already covered by existing tests.
• Does not need tests (why?).

• Appropriate change log entry included.
• No change log entry required (why? e.g. invisible to users).

• I have opened a documentation PR at cylc/cylc-doc/pull/XXXX.
• No documentation update required.

• Created an issue at cylc-uiserver conda-forge repository with version changes (if you changed dependencies in setup.py, see recipe/meta.yaml).
• No dependency changes.

[oliver-sanders]

Will rebase after #230 at which point this should work,

[datamel]

Having a brief glance at this with #230, and not marked ready for review but this line will need changing.
Nice looking pr though, happy to review when it is ready.

[oliver-sanders]

Yeah this was hacked a bit, should be much nicer with Jupyter Server 🤞.

[oliver-sanders]

Rebased this now we are using Jupyter Server.

The spawner works, the server is started on the remote host and the polling works correctly, but, from the hub's URL I get a 503 error and from the servers URL, a 404 so it's still not useable.

I think it's missing some of the config/initialisation required to work with the hub's remote API? More investigation required.