Spawned jupyterlab hangs for some users

We are using containerized jupyterhub with individual containers spawned for individual users set up in the conventional way (I think). A couple of times, say 3 times over a year, a users container gets corrupted or affected in some way such that they cannot login. When they attempt they get a “Clear Workspace or Keep Waiting” dialog, which doesn’t seem to do anything. Eventually jupyter-lab comes up but it is completely empty, no files, no launchers, just the empty frame.

I can see the users spawned container running, and I am able to docker exec into it. All the files appear to be there and intact. I don’t see anything suspicious in the jupyterhub logs.

Is there a log file or debug procedure that I can use to track down what is happening? I’m running out of ideas at this point.

A little more digging suggests the users home directory is not being found even though it is present in the running user-container. I’m still trying to find a more detailed log but no luck.

I was able to “fix” the problem by deleting the user’s volume that holds their home directory. We keep all user work under git so there was no data loss, but there was some inconvenience. Before applying that nuclear solution, I had tried removing the workspace data in ~/.jupyter/workspaces, but that didn’t fix it.

I wonder what data could have existed on the volume that was causing the jupyterlb spawner to hang? Any ideas?