Jupyterhub stalls when choosing "Restart Kernel and Run All Cells" for R notebooks (IRKernel)

JupyterHub Zero to JupyterHub on Kubernetes
1

Describe the bug
After the manul kernel restart in the existing notebook session, the notebooks freeze and become inaccessible

To Reproduce

Go to Kernel
Click on ‘Restart’
Accept the prompt message
The notebook stalls/halts with no access

Expected behavior
The notebook should re -establish connection after kernel restart

R : 4.1.0
jupyterhub : 1.5.0
IPython : 8.4.0
ipykernel : 6.15.1
ipywidgets : not installed
jupyter_client : 7.3.4
jupyter_core : 4.11.1
jupyter_server : not installed
jupyterlab : not installed
nbclient : 0.6.6
nbconvert : 6.5.0
nbformat : 5.4.0
notebook : 6.4.11
qtconsole : not installed
traitlets : 5.3.0
packageVersion(‘IRkernel’)
[1] ‘1.3’

OS: [e.g. iOS]
chrome Version 103.0.5060.114 (Official Build) (64-bit)
Architecture - (zero to k8s jupyterhub) / docker-stacks / jupyterhub

I can see a same issue with jupyterlab but we are still in hub version, is there any compatible irkernel & R version to be used for this functionality to work?
notebook logs

notebook [I 2022-07-15 19:05:05.267 SingleUserNotebookApp handlers:202] Creating new notebook in                                                                                                                                          
 notebook [I 2022-07-15 19:05:05.686 SingleUserNotebookApp log:189] 201 POST /user/cb193632088/api/contents (cb193632088@10.169.120.18) 558.51ms                                                                                           
 notebook [I 2022-07-15 19:05:06.311 SingleUserNotebookApp log:189] 200 GET /user/cb193632088/notebooks/Untitled3.ipynb?kernel_name=ir (cb193632088@10.169.120.18) 557.92ms                                                                
 notebook [I 2022-07-15 19:05:09.598 SingleUserNotebookApp log:189] 200 GET /user/cb193632088/api/config/common?_=1657911909397 (cb193632088@10.169.120.18) 2.67ms                                                                         
 notebook [I 2022-07-15 19:05:09.611 SingleUserNotebookApp log:189] 200 GET /user/cb193632088/api/config/notebook?_=1657911909396 (cb193632088@10.169.120.18) 12.95ms                                                                      
 notebook [I 2022-07-15 19:05:09.793 SingleUserNotebookApp log:189] 200 GET /user/cb193632088/api/contents/Untitled3.ipynb?type=notebook&_=1657911909398 (cb193632088@10.169.120.18) 129.47ms                                              
 notebook [I 2022-07-15 19:05:10.001 SingleUserNotebookApp log:189] 200 GET /user/cb193632088/api/contents/Untitled3.ipynb/checkpoints?_=1657911909399 (cb193632088@10.169.120.18) 19.12ms                                                 
 notebook [I 2022-07-15 19:05:10.005 SingleUserNotebookApp log:189] 200 GET /user/cb193632088/api/metrics/v1 (cb193632088@10.169.120.18) 22.06ms                                                                                           
 notebook [I 2022-07-15 19:05:10.009 SingleUserNotebookApp kernelmanager:179] Kernel started: 35bdd0fe-f412-4cc5-acf9-8e30a92b12bb, name: ir                                                                                               
 notebook [I 2022-07-15 19:05:10.011 SingleUserNotebookApp log:189] 201 POST /user/cb193632088/api/sessions (cb193632088@10.169.120.18) 106.00ms                                                                                           
 notebook [I 2022-07-15 19:05:10.690 SingleUserNotebookApp log:189] 101 GET /user/cb193632088/api/kernels/35bdd0fe-f412-4cc5-acf9-8e30a92b12bb/channels?session_id=9d5cd2433d2c46939296aac0ea104a03 (cb193632088@10.169.120.80) 225.45ms   
 notebook [I 2022-07-15 19:05:14.969 SingleUserNotebookApp log:189] 200 GET /user/cb193632088/api/metrics/v1 (cb193632088@10.169.120.18) 1.45ms                                                                                            
 notebook [I 2022-07-15 19:05:34.956 SingleUserNotebookApp log:189] 200 GET /user/cb193632088/api/metrics/v1 (cb193632088@10.169.120.18) 1.29ms                                                                                            
 notebook [I 2022-07-15 19:06:24.946 SingleUserNotebookApp log:189] 200 GET /user/cb193632088/api/metrics/v1 (cb193632088@10.169.120.18) 1.34ms                                                                                            
 notebook [I 2022-07-15 19:07:09.973 SingleUserNotebookApp log:189] 200 GET /user/cb193632088/api/contents/Untitled3.ipynb?content=0&_=1657911909400 (cb193632088@10.169.120.18) 2.79ms                                                    
 notebook [I 2022-07-15 19:07:10.050 SingleUserNotebookApp handlers:213] Saving file at /Untitled3.ipynb                                                                                                                                   
 notebook [I 2022-07-15 19:07:10.249 SingleUserNotebookApp log:189] 200 PUT /user/cb193632088/api/contents/Untitled3.ipynb (cb193632088@10.169.120.18) 199.70ms                                                                            
 notebook [I 2022-07-15 19:07:17.352 SingleUserNotebookApp log:189] 200 GET /user/cb193632088/api/metrics/v1 (cb193632088@10.169.120.18) 1.44ms                                                                                            
 notebook [I 2022-07-15 19:08:00.095 SingleUserNotebookApp kernelmanager:222] Starting buffering for 35bdd0fe-f412-4cc5-acf9-8e30a92b12bb:9d5cd2433d2c46939296aac0ea104a03                                                                 
 notebook [I 2022-07-15 19:08:00.265 SingleUserNotebookApp multikernelmanager:378] Kernel restarted: 35bdd0fe-f412-4cc5-acf9-8e30a92b12bb                                                                                                  
 notebook [I 2022-07-15 19:08:00.268 SingleUserNotebookApp log:189] 200 POST /user/cb193632088/api/kernels/35bdd0fe-f412-4cc5-acf9-8e30a92b12bb/restart (cb193632088@10.169.120.18) 156.94ms                                               
 notebook [I 2022-07-15 19:08:00.631 SingleUserNotebookApp log:189] 101 GET /user/cb193632088/api/kernels/35bdd0fe-f412-4cc5-acf9-8e30a92b12bb/channels?session_id=9d5cd2433d2c46939296aac0ea104a03 (cb193632088@10.169.120.18) 3.72ms     
 notebook [I 2022-07-15 19:08:00.631 SingleUserNotebookApp handlers:403] Restoring connection for 35bdd0fe-f412-4cc5-acf9-8e30a92b12bb:9d5cd2433d2c46939296aac0ea104a03 ```
2

Hi! Please can you tell us how you setup your JupyterHub, e.g. what is the base platform, how was it installed, can you show us your config with secrets redacted?

Have you check the resource usage of your singleuser server, e.g. is it running out of memory?

3

Hi manics,

thanks for the reply

I created this new thread so it can be correctly addressed… but in the previous you mentioned its like a existing issue.

the latest update on the case is , I tried installing stand alone anaconda jupyter notebook, I experienced the same problem when I do “Restart & Run all”. I suppose if this is really a jupyterhub related issue or more with irkernel vs notebook communication…

w.r.t the base platform, I am using
zero-to-jupyterhub-k8s
docker-stacks
jupyterhub : 1.5.0
notebook : 6.4.11

No memory leaks, all my instance are 8g guaranteed and this happens with new notebooks too.

Please note: Also am not able to edit the initial post with notebook tag or notebook title

4

@manics just a quick check, do we have active issue opened for this known issue ?

5

This sounds like an issue with your singleuser server and not JupyterHub.

You could try turning on debug logging to get more information from your singleuser server. Otherwise the more information you can provide the more likely it is that someone will be able to reproduce your bug. For example, which the docker-stacks images are you using (including the tag), and are you able to share your notebook that’s causing problems?

6

@manics Thanks for reply. Just wanted to clarify even if we use a regular standalone jupyternotebook with R kernel enabled, one can experience this restart kernel stalling issue.

could you help to locate if this notebook or hub related problem? I guess that hub is not able to return the refreshed session to the notebook to work again.

7

it is very unlikely that anything involving kernels is a hub-related problem. These kinds of messaging issues can be very tricky to debug, but it is almost certainly an issue between the notebook server and the R kernel, possibly in how the R kernel sets up its sockets during startup that might differ from the reference implementation of ipykernel. The docker-stacks image:tag you are using and a sample notebook would help a lot.

1 Like
8

@minrk This is our docker-stacks image tag
jupyter/base-notebook
But we build our custom docker-stack image based on this, buy including the r installation in this file.

Architecture - (zero to k8s jupyterhub) / docker-stacks / jupyterhub
R : 4.1.0
jupyterhub : 1.5.0
IPython : 8.4.0
ipykernel : 6.15.1
ipywidgets : not installed
jupyter_client : 7.3.4
jupyter_core : 4.11.1
notebook : 6.4.11

Also am not able to attached sample notebook here
Thanks for your response