I have a TLJH server set up and hosting multiple notebooks/scripts. One of them creates output that is written to shared directories in /srv/shared. The notebook accesses these directories as symlinks in each user’s home. The ownership of the source directories as well as the symlinks themselves is assigned to the jupyterhub-users group with chmod 777. I’m also using a umask 002 for each user for new files. The notebook is able to read and write files in these directories.
I am getting the File Load Error (in the title) when I select some of the files in the Jupyter Lab file browser. The files are relatively small log and tsv text files. Some open fine, including ones with the ‘log’ extension. But others raise this “unhandled exception”. The really weird thing is that for a fraction of a second, the content of the files is shown in a new tab before the error message appears. I thought initially this was a permissions issue, but I am able to use cat to view the contents from the terminal in the user’s account. The problem seems to be (linked) directory specific-- some work fine, other don’t-- but they are all configured the same.
I have used sudo journalctl -u to examine the jupyterhub, user and traefik logs, and didn’t find any useful output (traefik shows the files being accessed, but nothing else). I’m really at a loss as to what to try next…
Has anyone else encountered anything like this? Any suggestions?
Normally when you see errors of type unhandled exception, there should be something wrong on the backend jupyter server and they should be logged on single user logs. Could you post the single user logs here? Also, could you look into browser console logs to see if there are any issues with js?
Finally getting back to this issue. I looked more closely at the single-user logs-- thanks @mahendrapaipuri for this suggestion! The issue seems to be that when you open one of these files from jupyter lab, it creates an .ipynb_checkpoints subdirectory, and a file with the name of the file you are opening with -checkpoint inserted before the file extension.
Unfortunately, the user/group for this new directory and file are assigned to the current user. Since I am storing these files in a shared directory, this means that if another user tries to open another file, jupyter lab doesn’t have permission to write to the .ipynb_checkpoints subdirectory, and this generates the exception. Which is why I can briefly see that the file does, in fact, open-- the error is generated upon attempting to update the checkpoint file.
I suppose I could create the .ipynb_checkpoints subdirs preemptively and assign them to the jupyterhub-users group. But there would still be a problem for the individual log files; only the first user to open a given file would be able to open it again.
Any suggestions for a work around? Can I disable writing the checkpoint for non- *.ipynb files? Or automatically assign the group to jupyterhub-users for all files written by any user?
I dont think there is a native way to disable checkoutpoint. You can write your own class like the no-op checkpoint.
Or automatically assign the group to jupyterhub-users for all files written by any user?
You can use setgid to ensure all the new files created on a given directory has the same group as the parent.
If your shared directory is on NFS, you can use squash UID and GID feature to set UIDs and GIDs to the same user and group irrespective of the UID and GID on the client
It looks like the checkpoints class can be swapped by setting ContentsManager.checkpoints_class in single user server config. This would require subclassing the default one (FileCheckpoints) to customize the behaviour.