Hi folks. Managed to screw up my deployment, 3 days before semester start, and somehow I can’t duplicate what I did last year. K8s is running fine, my old helm configuration script seems to still work (updated the version numbers), but the storage isn’t initializing correctly so the hub itself is hanging with FailedScheduling with
0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
Everything I’m finding to try to debug says that JupyterHub should work fine so long as the default dynamic storage class is working ok. Which it is - doing tests, using that class, I can pop up a quick PVC that spawns a PV, does some stuff, and shuts down nicely. (I’m using the nfs-subdir-external-provisioner, like I did for the last year).
Anyone have a moment to help me figure out what I’m missing?
- bare metal kubernetes install on Ubuntu 18.04
- two nodes: one runs the primary node, tainted to also allow pods to deploy to it; second just runs user pods
- NFS available, and tested, using nfs-subdir-external-provisioner
- helm seems to be working fine, and old config.yaml was used
- storageclass is nfs-client, and the singleuser configuration points to that
Note: the old hub-db-dir directories are still on the NFS share, but new ones aren’t spawning. So it seems to be the hub-db-dir that isn’t initializing correctly. Not sure why it isn’t using either default (which is nfs-client) or the instructed (which is ALSO nfs-client), as it used to.