Transfering from a cpu cluster to a gpu cluster halfway through a class

Hi Folks, somewhat confused about this use case:

I want a simple cpu pool, then a cpu-with-gpu pool. For some length of time in the class i am running, i want to use a cpu cluster. When larger deep learning models happen, I want a cpu-with-gpu cluster.

Is the right way to do this to create two user node pools?

For the first part, label the cpu pool with “user”, then switch the “user” label to the gpu node-pool halfway through? Will the PV disks transfer to the new pool? (this is all on GKE)

I’d think you could either do the following two things, but I recommend the first option:

  • having a profile list that allows the user to choose which node-pool they end up on. You would then override details like user image and requests.
  • switching the default behavior to make all users arrive to the GPU node pool at some point.

I’ve written down quite a bit on how to work with GPUs ok GKE in the link below, I recommend reading through that for some inspiration on how you would go about this.

1 Like