HI all - i hope that this is the right place to post. i’m running into a few issues with our jupyter hub on google cloud /kubernetes that i was hoping others with more experience could help me troubleshoot.
I am using gitpuller to bring in a repo of jupyter notebooks. My test pod is currently out of storage space (due to some testing i was doing). Now, when i try to login it hangs because of lack of storage space
LINK TO CALLING GITPULLER: https://github.com/earthlab/hub-ops/blob/60b44d7e0a4dbc1412491655c2bd233fc0f254eb/hub-charts/ea-hub/values.yaml#L40
ERROR message:
2020-05-07 22:50:13+00:00 [Warning] Exec lifecycle hook ([sh -c gitpuller https://github.com/earthlab-education/spring-2020-course-notebooks master ea-2020-class-notebooks; ]) for Container "notebook" in Pod "jupyter-lwasser_ea-hub(0adcfc85-90b5-11ea-97b9-42010a800008)" failed - error: command 'sh -c gitpuller https://github.com/earthlab-education/spring-2020-course-notebooks master ea-2020-class-notebooks; ' exited with 1: Traceback (most recent call last): File "/opt/conda/bin/gitpuller", line 10, in <module> sys.exit(main()) File "/opt/conda/lib/python3.7/site-packages/nbgitpuller/pull.py", line 251, in main args.repo_dir File "/opt/conda/lib/python3.7/site-packages/nbgitpuller/pull.py", line 81, in pull yield from self.update() File "/opt/conda/lib/python3.7/site-packages/nbgitpuller/pull.py", line 191, in update yield from self.update_remotes() File "/opt/conda/lib/python3.7/site-packages/nbgitpuller/pull.py", line 128, in update_remotes yield from execute_cmd(['git', 'fetch'], cwd=self.repo_dir) File "/opt/conda/lib/python3.7/site-packages/nbgitpuller/pull.py", line 43, in execute_cmd raise subprocess.CalledProcessError(ret, cmd) subprocess.CalledProcessError: Command '['git', 'fetch']' returned non-zero exit status 128. , message: "$ git fetch\n\nerror: unable to create temporary file: No space left on device\n\nfatal: failed to write object\n\nfatal: unpack-objects failed\n\nTraceback (most recent call last):\n File \"/opt/conda/bin/gitpuller\", line 10, in <module>\n sys.exit(main())\n File \"/opt/conda/lib/python3.7/site-packages/nbgitpuller/pull.py\", line 251, in main\n args.repo_dir\n File \"/opt/conda/lib/python3.7/site-packages/nbgitpuller/pull.py\", line 81, in pull\n yield from self.update()\n File \"/opt/conda/lib/python3.7/site-packages/nbgitpuller/pull.py\", line 191, in update\n yield from self.update_remotes()\n File \"/opt/conda/lib/python3.7/site-packages/nbgitpuller/pull.py\", line 128, in update_remotes\n yield from execute_cmd(['git', 'fetch'], cwd=self.repo_dir)\n File \"/opt/conda/lib/python3.7/site-packages/nbgitpuller/pull.py\", line 43, in execute_cmd\n raise subprocess.CalledProcessError(ret, cmd)\nsubprocess.CalledProcessError: Command '['git', 'fetch']' returned non-zero exit status 128.\n"
My questions
- is there a way to sneak into a pod’s storage and remove files? luckily this is my pod which i could easily delete but i wondered if there is a way to fix this issue without deleting the entire pod?
- if i wanted to adjust my helm chart to “try” to use gitpuller and then fail gracefully if storage is full or it can’t run, is there a way to use something like
||or some sort of conditional in bash tha i could use to ensure that a pod can still load if gitpuller can’t run? - When a pod won’t launch - i can’t “See” it when i call:
kubectl get pods --all-namespacesbecause it tries to launch - i can’t see it when it is trying to launch but then it fails so the pod “goes away”. is there a way to check in on a pod that is not activate to see it’s status, storage size, etc?
Thank you for any suggestions! i’d like to figure out how to fix my pod so i can fix student pods in the future.
Leah