# GPU usage from all nodes while using JupyterHub on Kubernetes

**URL:** <https://discourse.jupyter.org/t/gpu-usage-from-all-nodes-while-using-jupyterhub-on-kubernetes/29052>\
**Category:** JupyterHub\
**Tags:** jupyterhub, how-to, help-wanted\
**Created:** [October 9, 2024, 5:18am UTC](https://discourse.jupyter.org/t/gpu-usage-from-all-nodes-while-using-jupyterhub-on-kubernetes/29052 "2024-10-09T05:18:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ross](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/ross/32/12645_2.png) [@Ross](https://discourse.jupyter.org/u/Ross)\
**Post date:** [October 9, 2024, 5:18am UTC](https://discourse.jupyter.org/t/gpu-usage-from-all-nodes-while-using-jupyterhub-on-kubernetes/29052/1 "2024-10-09T05:18:06Z")

</div>

Hello folks,  
I have 1 master - 3 worker cluster of Kubernetes. All 3 worker nodes have NVIDIA GPU. I am planning to use a dockerspawner to spin up new containers on this containers. Now I am interested about the GPU resource allocation here. If I run a deep learning model on one of the notebooks then it should be able to use all GPU’s from other nodes as well. I want to distribute my ML workload here.

My question is : Will combination of (kubernetes + jupyterhub + dockerspawner ) work for my usecase or should I look for any alternative architecture? Also please let me know what are other alternatives that can help me in my usecase

cc: @mahendrapaipuri @markperri @consideRatio

---

<div class="post-metadata">

**Author:** ![consideRatio](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/consideratio/32/1033_2.png) [@consideRatio](https://discourse.jupyter.org/u/consideRatio)\
**Post date:** [October 9, 2024, 5:25am UTC](https://discourse.jupyter.org/t/gpu-usage-from-all-nodes-while-using-jupyterhub-on-kubernetes/29052/2 "2024-10-09T05:25:17Z")

</div>

If you use kubernetes, you need kubespawner and not dockerspawner, and in practice you probably should use the jupyterhub helm chart.

---

<div class="post-metadata">

**Author:** ![consideRatio](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/consideratio/32/1033_2.png) [@consideRatio](https://discourse.jupyter.org/u/consideRatio)\
**Post date:** [October 9, 2024, 5:28am UTC](https://discourse.jupyter.org/t/gpu-usage-from-all-nodes-while-using-jupyterhub-on-kubernetes/29052/3 "2024-10-09T05:28:39Z")

</div>

> If I run a deep learning model on one of the notebooks then it should be able to use all GPU’s from other nodes as well. I want to distribute my ML workload here.

This is hard, and you need some software strategy involving k8s awareness to handle that then.

---

<div class="post-metadata">

**Author:** ![mahendrapaipuri](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/mahendrapaipuri/32/6800_2.png) [@mahendrapaipuri](https://discourse.jupyter.org/u/mahendrapaipuri)\
**Post date:** [October 9, 2024, 7:02am UTC](https://discourse.jupyter.org/t/gpu-usage-from-all-nodes-while-using-jupyterhub-on-kubernetes/29052/4 "2024-10-09T07:02:53Z")

</div>

Yes, you will need to use [k8s Job](https://kubernetes.io/docs/concepts/workloads/controllers/job/) (or something similar) that will launch pods on each node to distribute your ML workload. I dont think Kubespawner supports k8s jobs.

Maybe [Kubeflow](https://www.kubeflow.org/) might be an option for your use case and it supports Jupyter notebooks.