Who is interested in collaborating with NFDI around spawning user sessions across multiple clusters?

[choldgraf]

In a recent collaboration cafe we had a presentation from a team at NFDI that is building out a JupyterHub service across much of Germany. @arnim and several others were represented here (Arnim can you cc others as you wish?).

They are interested in enhancing JupyterHub's ability to spawn user sessions across many clusters and/or hubs, so that a single hub could be the launching point for multiple other hubs, potentially in many other clusters.

In the meeting, we agreed that some pieces of this functionality could be beneficial to build into JupyterHub or Z2JH, as many communities want the ability to spawn user sessions across different kinds of infrastructure. @minrk noted that we could try to align on a canonical implementation that JupyterHub recommends, developed in partnership with the NFDI team, and they could then build on top of it for their service. I think @consideRatio and @MridulS were also interested.

Question: who is interested in this?

• Several didn't attend the meeting, is anybody else interested in collaborating on this effort?

References

• Presentation about their vision
• Their current BackendSpawner implementation
  • Currently used backend “manager” implementations (using Django Rest Framework):
  • UNICORE Manager: https://gitlab.jsc.fz-juelich.de/jupyterjsc/k8s/images/drf-unicoremgr
  • Kubernetes Manager: https://gitlab.jsc.fz-juelich.de/jupyterjsc/k8s/images/drf-k8smgr
• Similar prototype effort from @yuvipanda

[yuvipanda]

https://github.com/yuvipanda/jupyterhub-multicluster-kubespawner is my take on this. I'd love for us to find funding to finish this.

Ah sorry i see you have already linked it :D

[arnim]

Hey everyone, thanks for your help & thoughts on this!
I think one of the central points here is that integrating a national project, like the Jupyter-NFDI, in an international context is something we should plan from the very beginning. We are currently on a by-weekly schedule for drafting the proposal and I'm sure that we will put something nice together :)

Let's stay in contact on this!

cc @berndflemisch, @bhagemeier

[kreuzert]

Hey,
Unfortunately, I couldn't join the last collaboration cafe. I am one of the administrators of the mentioned JupyterHub service. I have extracted our Spawner class to an extra repo ( https://github.com/kreuzert/jupyterhub-backendspawner ). With the next update of our JupyterHub service, we will use this one to start JupyterLabs on different systems.

@yuvipanda that's a cool Spawner. But we also want to start JupyterLabs on HPC Systems (and possibly other things, too). That's why we need a more generic approach.

[yuvipanda]

@kreuzert I just took a quick look at outpost and I think I love this model! I am going to try to find some time to dive into it some more, and if it works out I'll probably archive my repo and point people to outpost.

[kreuzert]

@yuvipanda
That sounds great. Just let me know, if you or anyone else wants to have a short introduction.
Feedback and contributions are always welcome.