Errors around KillContainer - fails to detach pvc from nod

JupyterHub
1

Hi!

We have a situation where we see:

 error killing pod: [failed to "KillContainer" for "notebook" with
  KillContainerError: "rpc error: code = DeadlineExceeded desc = context
  deadline exceeded", failed to "KillPodSandbox" for
  "5df9642a-8b7f-48bf-8056-af1119c4b555" with KillPodSandboxError: "rpc error:
  code = DeadlineExceeded desc = context deadline exceeded"]

for users. Then later they try to spin up a node with jupyterhub and it fails (picking a different node in the cluster). They get multi-attachment errors.

A manual way to fix is deleting pvs, pvcs, and the offending node. But it’s happening to multiple users now.

Not sure how to proceed.

We are currently using Jupyterhub / helm, 3.3.7

2

We discovered there was a node process running Netfliy that would totally freeze our singleuser environment, to the point where we couldn’t shut it down or even pkill the processes.

We figured out we could forcekill the terminating pods and using this command:

kubectl delete volumeattachment -n jupyterhub <id>

would detach the pvs.

Still wondering if there is a way to prevent this from happening in the future.