Is it possible to cluster / load balanced to spread resources transparently across a pool of Jupyter “nodes”. In other words, if 5 users each wanted to use 200G of RAM, then you would need a 1TB RAM machine. However, if Jupyter were clustered, then 5 machines with 200TGB would work.
Is there any mechanism for resource clustering in Jupyter hub or Jupyter Notebook?
Thanks in advance!
If you’re using JupyterHub it sounds like you want a remote spawner. Kubernetes is probably the best documented option since you can install it using Helm
But there are others, some of which are listed in Spawners — JupyterHub documentation
For example, you could take advantage of High Performance Cluster if you have one.
Alternatively you could look at running kernels remotely (I don’t have any experience of this, but others on this forum do):
https://jupyter-enterprise-gateway.readthedocs.io/