Hi folks,
Is it possible to use two different Kubernetes clusters for spawning servers in a single JupyterHub? I’m looking into setting up a hub for my team and wanted to run it on AWS fargate. Fargate doesn’t support GPUs though so I was wondering if it would eventually be possible to create a named server for users that had GPU workers which would be managed by a second kubernetes cluster using EKS or EC2.
If any of that sounds like a terrible idea let me know!
Thanks!
Hi! It’s not currently possible with https://zero-to-jupyterhub.readthedocs.io/. I imagine it would be a lot of work to add it, and if you did you’d still need a way to control which k8s cluster your users ended up on. I think your best option is to run two separate JupyterHubs. You might be able to configure shared user storage between them, e.g. using NFS, but that would depend on your cloud and it’s not something I’m familiar with.
Okay thanks! It is possible to mix different types of resources (e.g. GPU vs non-GPU) within a single kubernetes cluster though, right? Probably using named servers?
You can configure multiple “spawners” with different hardware requests and limits, and even different base images for each type of spawner.