Issues with Storage / Locked out pod on Jupyter Hub

HI all - i hope that this is the right place to post. i’m running into a few issues with our jupyter hub on google cloud /kubernetes that i was hoping others with more experience could help me troubleshoot.

I am using gitpuller to bring in a repo of jupyter notebooks. My test pod is currently out of storage space (due to some testing i was doing). Now, when i try to login it hangs because of lack of storage space


ERROR message:

2020-05-07 22:50:13+00:00 [Warning] Exec lifecycle hook ([sh -c gitpuller master ea-2020-class-notebooks; ]) for Container "notebook" in Pod "jupyter-lwasser_ea-hub(0adcfc85-90b5-11ea-97b9-42010a800008)" failed - error: command 'sh -c gitpuller master ea-2020-class-notebooks; ' exited with 1: Traceback (most recent call last): File "/opt/conda/bin/gitpuller", line 10, in <module> sys.exit(main()) File "/opt/conda/lib/python3.7/site-packages/nbgitpuller/", line 251, in main args.repo_dir File "/opt/conda/lib/python3.7/site-packages/nbgitpuller/", line 81, in pull yield from self.update() File "/opt/conda/lib/python3.7/site-packages/nbgitpuller/", line 191, in update yield from self.update_remotes() File "/opt/conda/lib/python3.7/site-packages/nbgitpuller/", line 128, in update_remotes yield from execute_cmd(['git', 'fetch'], cwd=self.repo_dir) File "/opt/conda/lib/python3.7/site-packages/nbgitpuller/", line 43, in execute_cmd raise subprocess.CalledProcessError(ret, cmd) subprocess.CalledProcessError: Command '['git', 'fetch']' returned non-zero exit status 128. , message: "$ git fetch\n\nerror: unable to create temporary file: No space left on device\n\nfatal: failed to write object\n\nfatal: unpack-objects failed\n\nTraceback (most recent call last):\n File \"/opt/conda/bin/gitpuller\", line 10, in <module>\n sys.exit(main())\n File \"/opt/conda/lib/python3.7/site-packages/nbgitpuller/\", line 251, in main\n args.repo_dir\n File \"/opt/conda/lib/python3.7/site-packages/nbgitpuller/\", line 81, in pull\n yield from self.update()\n File \"/opt/conda/lib/python3.7/site-packages/nbgitpuller/\", line 191, in update\n yield from self.update_remotes()\n File \"/opt/conda/lib/python3.7/site-packages/nbgitpuller/\", line 128, in update_remotes\n yield from execute_cmd(['git', 'fetch'], cwd=self.repo_dir)\n File \"/opt/conda/lib/python3.7/site-packages/nbgitpuller/\", line 43, in execute_cmd\n raise subprocess.CalledProcessError(ret, cmd)\nsubprocess.CalledProcessError: Command '['git', 'fetch']' returned non-zero exit status 128.\n"

My questions

  1. is there a way to sneak into a pod’s storage and remove files? luckily this is my pod which i could easily delete but i wondered if there is a way to fix this issue without deleting the entire pod?
  2. if i wanted to adjust my helm chart to “try” to use gitpuller and then fail gracefully if storage is full or it can’t run, is there a way to use something like || or some sort of conditional in bash tha i could use to ensure that a pod can still load if gitpuller can’t run?
  3. When a pod won’t launch - i can’t “See” it when i call: kubectl get pods --all-namespaces because it tries to launch - i can’t see it when it is trying to launch but then it fails so the pod “goes away”. is there a way to check in on a pod that is not activate to see it’s status, storage size, etc?

Thank you for any suggestions! i’d like to figure out how to fix my pod so i can fix student pods in the future.