Canvas + Jupyterhub + kubernetes issues

Hello,

I am teaching a class via jupyterhub + kubernetes+ canvas, with the kubernetes setup as described in the Zero-to-Jupyterhub tutorial.

I am having some issues with students being unable to start their servers. It works for me and the other instructor and one or two students, but most are seeing something like the following:

I am having a very hard time debugging, because this works from my canvas account. Any suggestions?

What size are your GKE nodes? There’s a limitation in Kubernetes that only 16 PV’s will be allocated per node; allegedly this is supposed to be fixed, but we still encounter this problem on GKE. This means that if you use “fat nodes” and put a lot of people on them, it will start to fail to mount the PV’s. It will be random based on the number of other people on that given node.

I bring this up because you have some messages about PV’s not available.