We are using JupyterHub in an OpenShift cluster where we create a separate Jupyter server for each user and they are able to spawn notebooks after selecting the required configuration. We provide an option to choose whether to attach a GPU with their notebook or not.
Currently, when a user chooses to use the GPU, a whole GPU is allocated to their notebook which as per our observation is more than required to them most of the times. Also, since the GPUs in the cluster are limited, sometimes users have to wait for the GPUs to be free which are being used by other users.
So, we were thinking of a way to mitigate this wastage and then we came across this idea of remote execution of code in Jupyter, i.e. running a piece of code (maybe a cell of the notebook) on remote GPU while the rest of the notebook is running on a CPU. This would keep the GPUs free for most of the time and available to all the users.
I have gone through few similar topics here but could not find a stable and working solution for the same or maybe I have missed some topic which has it.
So, can someone please help me to find the solution to this?