Suddenly unable to run code or save files

Hi!

Any help would be greatly appreciated! I’m completely at a loss on what caused this issue and how to fix it.

I’m using The Littlest Jupyter Hub on Ubuntu with nbgrader to manage a group of 22 students. Everything was working fine until suddenly I was unable to create new files, use any functionalities on the “Formgrader” tab (nbgrader), run any cells in any Jupyter Notebook, open a terminal, or save changes to any existing files. Whenever I attempt any of the above, the server gets stuck for a long time and then just outputs an “Error creating file” message with nothing else to go on, or the “Forbidden” flag at the Notebook toolbar when attempting to run a cell. All I can effectively do (and all my student accounts can do) is open files and view their contents.

I wish I could have more information about the issue, but that’s about it. It’s running on GCP on a Virtual Machine. I’ve made no updates, no new installations, and I’m not aware of what could have caused this. The last thing I did on the server was creating and releasing a new assignment, which was successfully completed by a few students before this error started to happen when returning from spring break. I’m unaware if any student managed to hack the system or did any unusual activity, which would be very surprising considering it’s an introductory Python programming course.
I’ve checked the logs and they don’t seem helpful in diagnosing or fixing this issue. It’s only within JupyterHub that I can’t make files- I can do it fine on the terminal when I SSH into the VM.
I’ve also checked that the VM has enough RAM memory and memory on its boot disk, which it does.

I’m open to any suggestions if anyone would be kind enough to give me some pointers to go on!

There should be an error in the logs when the file creation fails, either in the server log or in the browser console, or both. Sharing that will be the first step to finding out what’s wrong.