Not able to create new notebook or open exisiting notebooks

Hi,
I am using Anaconda with python3.8. I have jupyter notebook and jupyterhub installed. Until yesterday everything was working fine but since today I am not able to open or create new notebooks when using jupyter notebook or jupyterhub interface.

I spent a lot of time debugging what is wrong and I can’t figure it out.

When opening the notebook I see some of the network calls getting stuck (screenshot below)

While not being able to create/open a notebook, I am able to start a terminal and create .txt file. I have also made sure that there is at least one python3.8 kernel available for jupyter.

I would appreciate any help to debug this issue.

Hi! Can you try running with debug enabled and show us the logs?

jupyter notebook --debug

It’ll also be helpful to know your operating system, and other information about your setup.

OS : CentOS Linux release 7.9.2009 (Core)

If it helps, I am running jupyter on a remote server and accessing via personal machine.
The log produced by directly hitting the notebook link is below:

jupyter notebook --ip 0.0.0.0 --port 8888 --debug --allow-root
[D 11:07:46.231 NotebookApp] Searching ['/data/umanggup', '/nas/home/umanggup/.jupyter', '/nas/home/umanggup/anaconda3/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 11:07:46.232 NotebookApp] Looking for jupyter_config in /etc/jupyter
[D 11:07:46.232 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 11:07:46.232 NotebookApp] Looking for jupyter_config in /nas/home/umanggup/anaconda3/etc/jupyter
[D 11:07:46.232 NotebookApp] Looking for jupyter_config in /nas/home/umanggup/.jupyter
[D 11:07:46.232 NotebookApp] Looking for jupyter_config in /data/umanggup
[D 11:07:46.234 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter
[D 11:07:46.234 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 11:07:46.234 NotebookApp] Looking for jupyter_notebook_config in /nas/home/umanggup/anaconda3/etc/jupyter
[D 11:07:46.237 NotebookApp] Loaded config file: /nas/home/umanggup/anaconda3/etc/jupyter/jupyter_notebook_config.json
[D 11:07:46.237 NotebookApp] Looking for jupyter_notebook_config in /nas/home/umanggup/.jupyter
[D 11:07:46.239 NotebookApp] Loaded config file: /nas/home/umanggup/.jupyter/jupyter_notebook_config.json
[D 11:07:46.239 NotebookApp] Looking for jupyter_notebook_config in /data/umanggup
[D 11:07:46.249 NotebookApp] Paths used for configuration of jupyter_notebook_config: 
    	/etc/jupyter/jupyter_notebook_config.json
[D 11:07:46.250 NotebookApp] Paths used for configuration of jupyter_notebook_config: 
    	/usr/local/etc/jupyter/jupyter_notebook_config.json
[D 11:07:46.253 NotebookApp] Paths used for configuration of jupyter_notebook_config: 
    	/nas/home/umanggup/anaconda3/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
    	/nas/home/umanggup/anaconda3/etc/jupyter/jupyter_notebook_config.json
[D 11:07:46.256 NotebookApp] Paths used for configuration of jupyter_notebook_config: 
    	/nas/home/umanggup/.jupyter/jupyter_notebook_config.json
[I 11:07:48.658 NotebookApp] JupyterLab extension loaded from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/jupyterlab
[I 11:07:48.658 NotebookApp] JupyterLab application directory is /nas/home/umanggup/anaconda3/share/jupyter/lab
[I 11:07:48.665 NotebookApp] Serving notebooks from local directory: /data/umanggup
[I 11:07:48.665 NotebookApp] Jupyter Notebook 6.1.4 is running at:
[I 11:07:48.665 NotebookApp] http://mindsgpu05:8888/
[I 11:07:48.665 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 11:07:48.680 NotebookApp] No web browser found: could not locate runnable browser.
[D 11:07:50.006 NotebookApp] Paths used for configuration of notebook: 
    	/etc/jupyter/nbconfig/notebook.json
[D 11:07:50.008 NotebookApp] Paths used for configuration of notebook: 
    	/usr/local/etc/jupyter/nbconfig/notebook.json
[D 11:07:50.010 NotebookApp] Paths used for configuration of notebook: 
    	/nas/home/umanggup/anaconda3/etc/jupyter/nbconfig/notebook.d/widgetsnbextension.json
    	/nas/home/umanggup/anaconda3/etc/jupyter/nbconfig/notebook.json
[D 11:07:50.012 NotebookApp] Paths used for configuration of notebook: 
    	/nas/home/umanggup/.jupyter/nbconfig/notebook.json
[D 11:07:50.014 NotebookApp] 200 GET /api/config/notebook?_=1610737669843 (10.255.250.77) 11.19ms
[D 11:07:50.017 NotebookApp] Paths used for configuration of common: 
    	/etc/jupyter/nbconfig/common.json
[D 11:07:50.018 NotebookApp] Paths used for configuration of common: 
    	/usr/local/etc/jupyter/nbconfig/common.json
[D 11:07:50.019 NotebookApp] Paths used for configuration of common: 
    	/nas/home/umanggup/anaconda3/etc/jupyter/nbconfig/common.json
[D 11:07:50.020 NotebookApp] Paths used for configuration of common: 
    	/nas/home/umanggup/.jupyter/nbconfig/common.json
[D 11:07:50.021 NotebookApp] 200 GET /api/config/common?_=1610737669844 (10.255.250.77) 5.31ms
[D 11:07:50.027 NotebookApp] Path components/MathJax/extensions/TeX/newcommand.js served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/components/MathJax/extensions/TeX/newcommand.js
[D 11:07:50.028 NotebookApp] 304 GET /static/components/MathJax/extensions/TeX/newcommand.js?V=2.7.9 (10.255.250.77) 2.51ms
[D 11:07:50.033 NotebookApp] Path components/MathJax/extensions/TeX/begingroup.js served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/components/MathJax/extensions/TeX/begingroup.js
[D 11:07:50.034 NotebookApp] 304 GET /static/components/MathJax/extensions/TeX/begingroup.js?V=2.7.9 (10.255.250.77) 1.77ms
[D 11:07:50.040 NotebookApp] Path components/MathJax/extensions/MathML/content-mathml.js served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/components/MathJax/extensions/MathML/content-mathml.js
[D 11:07:50.041 NotebookApp] 304 GET /static/components/MathJax/extensions/MathML/content-mathml.js?V=2.7.9 (10.255.250.77) 1.78ms
[D 11:07:50.072 NotebookApp] Found kernel test-kernel in /nas/home/umanggup/.local/share/jupyter/kernels
[D 11:07:50.073 NotebookApp] Found kernel python3 in /nas/home/umanggup/anaconda3/share/jupyter/kernels
[D 11:07:50.085 NotebookApp] 304 GET /api/kernelspecs (10.255.250.77) 15.84ms
[D 11:07:50.151 NotebookApp] Path components/MathJax/jax/output/HTML-CSS/fonts/STIX-Web/fontdata.js served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/components/MathJax/jax/output/HTML-CSS/fonts/STIX-Web/fontdata.js
[D 11:07:50.208 NotebookApp] 200 GET /static/components/MathJax/jax/output/HTML-CSS/fonts/STIX-Web/fontdata.js?V=2.7.9 (10.255.250.77) 120.96ms

and the server gets stuck after this. I have to kill is via htop and ctrl+C does not work. I am also not able to navigate to another page from the browser after this.

Below is the log for creating a new notebook

[D 11:11:38.262 NotebookApp] Searching ['/data/umanggup', '/nas/home/umanggup/.jupyter', '/nas/home/umanggup/anaconda3/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 11:11:38.262 NotebookApp] Looking for jupyter_config in /etc/jupyter
[D 11:11:38.263 NotebookApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 11:11:38.263 NotebookApp] Looking for jupyter_config in /nas/home/umanggup/anaconda3/etc/jupyter
[D 11:11:38.264 NotebookApp] Looking for jupyter_config in /nas/home/umanggup/.jupyter
[D 11:11:38.265 NotebookApp] Looking for jupyter_config in /data/umanggup
[D 11:11:38.266 NotebookApp] Looking for jupyter_notebook_config in /etc/jupyter
[D 11:11:38.266 NotebookApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 11:11:38.266 NotebookApp] Looking for jupyter_notebook_config in /nas/home/umanggup/anaconda3/etc/jupyter
[D 11:11:38.268 NotebookApp] Loaded config file: /nas/home/umanggup/anaconda3/etc/jupyter/jupyter_notebook_config.json
[D 11:11:38.269 NotebookApp] Looking for jupyter_notebook_config in /nas/home/umanggup/.jupyter
[D 11:11:38.271 NotebookApp] Loaded config file: /nas/home/umanggup/.jupyter/jupyter_notebook_config.json
[D 11:11:38.271 NotebookApp] Looking for jupyter_notebook_config in /data/umanggup
[D 11:11:38.282 NotebookApp] Paths used for configuration of jupyter_notebook_config: 
    	/etc/jupyter/jupyter_notebook_config.json
[D 11:11:38.283 NotebookApp] Paths used for configuration of jupyter_notebook_config: 
    	/usr/local/etc/jupyter/jupyter_notebook_config.json
[D 11:11:38.286 NotebookApp] Paths used for configuration of jupyter_notebook_config: 
    	/nas/home/umanggup/anaconda3/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
    	/nas/home/umanggup/anaconda3/etc/jupyter/jupyter_notebook_config.json
[D 11:11:38.289 NotebookApp] Paths used for configuration of jupyter_notebook_config: 
    	/nas/home/umanggup/.jupyter/jupyter_notebook_config.json
[I 11:11:40.711 NotebookApp] JupyterLab extension loaded from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/jupyterlab
[I 11:11:40.712 NotebookApp] JupyterLab application directory is /nas/home/umanggup/anaconda3/share/jupyter/lab
[I 11:11:40.719 NotebookApp] Serving notebooks from local directory: /data/umanggup
[I 11:11:40.720 NotebookApp] Jupyter Notebook 6.1.4 is running at:
[I 11:11:40.720 NotebookApp] http://mindsgpu05:8888/
[I 11:11:40.720 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 11:11:40.736 NotebookApp] No web browser found: could not locate runnable browser.
[D 11:11:45.646 NotebookApp] Using contents: services/contents
[D 11:11:45.734 NotebookApp] Path base/images/favicon.ico served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/base/images/favicon.ico
[D 11:11:45.739 NotebookApp] Path components/jquery-ui/themes/smoothness/jquery-ui.min.css served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css
[D 11:11:45.742 NotebookApp] Path components/jquery-typeahead/dist/jquery.typeahead.min.css served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/components/jquery-typeahead/dist/jquery.typeahead.min.css
[D 11:11:45.744 NotebookApp] Path style/style.min.css served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/style/style.min.css
[D 11:11:45.748 NotebookApp] Path components/es6-promise/promise.min.js served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/components/es6-promise/promise.min.js
[D 11:11:45.751 NotebookApp] Path components/react/react.production.min.js served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/components/react/react.production.min.js
[D 11:11:45.753 NotebookApp] Path components/react/react-dom.production.min.js served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/components/react/react-dom.production.min.js
[D 11:11:45.755 NotebookApp] Path components/create-react-class/index.js served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/components/create-react-class/index.js
[D 11:11:45.758 NotebookApp] Path components/requirejs/require.js served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/components/requirejs/require.js
[D 11:11:45.760 NotebookApp] Path base/images/logo.png served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/base/images/logo.png
[D 11:11:45.764 NotebookApp] Path tree/js/main.min.js served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/tree/js/main.min.js
[D 11:11:45.773 NotebookApp] 200 GET /tree (10.255.250.77) 129.93ms
[D 11:11:45.844 NotebookApp] Path custom.css served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/custom/custom.css
[D 11:11:45.845 NotebookApp] 304 GET /custom/custom.css (10.255.250.77) 3.62ms
[D 11:11:45.983 NotebookApp] Path services/contents.js served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/services/contents.js
[D 11:11:45.985 NotebookApp] 200 GET /static/services/contents.js?v=20210115111138 (10.255.250.77) 4.15ms
[D 11:11:46.041 NotebookApp] Paths used for configuration of tree: 
    	/etc/jupyter/nbconfig/tree.json
[D 11:11:46.042 NotebookApp] Paths used for configuration of tree: 
    	/usr/local/etc/jupyter/nbconfig/tree.json
[D 11:11:46.044 NotebookApp] Paths used for configuration of tree: 
    	/nas/home/umanggup/anaconda3/etc/jupyter/nbconfig/tree.json
[D 11:11:46.045 NotebookApp] Paths used for configuration of tree: 
    	/nas/home/umanggup/.jupyter/nbconfig/tree.json
[D 11:11:46.045 NotebookApp] 200 GET /api/config/tree?_=1610737905942 (10.255.250.77) 7.26ms
[D 11:11:46.078 NotebookApp] Paths used for configuration of common: 
    	/etc/jupyter/nbconfig/common.json
[D 11:11:46.079 NotebookApp] Paths used for configuration of common: 
    	/usr/local/etc/jupyter/nbconfig/common.json
[D 11:11:46.080 NotebookApp] Paths used for configuration of common: 
    	/nas/home/umanggup/anaconda3/etc/jupyter/nbconfig/common.json
[D 11:11:46.081 NotebookApp] Paths used for configuration of common: 
    	/nas/home/umanggup/.jupyter/nbconfig/common.json
[D 11:11:46.082 NotebookApp] 200 GET /api/config/common?_=1610737905943 (10.255.250.77) 5.61ms
[D 11:11:46.091 NotebookApp] Found kernel test-kernel in /nas/home/umanggup/.local/share/jupyter/kernels
[D 11:11:46.095 NotebookApp] Found kernel python3 in /nas/home/umanggup/anaconda3/share/jupyter/kernels
[D 11:11:46.112 NotebookApp] 304 GET /api/kernelspecs (10.255.250.77) 27.71ms
[D 11:11:46.113 NotebookApp] 200 GET /api/terminals?_=1610737905944 (10.255.250.77) 27.72ms
[D 11:11:46.116 NotebookApp] 200 GET /api/sessions?_=1610737905945 (10.255.250.77) 29.59ms
[D 11:11:46.119 NotebookApp] 200 GET /api/terminals?_=1610737905946 (10.255.250.77) 1.31ms
[D 11:11:46.432 NotebookApp] 200 GET /api/contents?type=directory&_=1610737905947 (10.255.250.77) 9.39ms
[D 11:11:46.435 NotebookApp] Path custom.js served from /nas/home/umanggup/anaconda3/lib/python3.8/site-packages/notebook/static/custom/custom.js
[D 11:11:46.437 NotebookApp] 200 GET /custom/custom.js?v=20210115111138 (10.255.250.77) 3.50ms
[I 11:11:53.896 NotebookApp] Creating new notebook in 
[D 11:11:53.899 NotebookApp] Saving /data/umanggup/Untitled.ipynb

Once again … the server is stuck after this and needs to killed

Have you definitely got enough free disk space?

How is your nas mounted? Is it NFS?

Disk space is enough. I am able to create .txt files as well as open the terminal via jupyter.

I think NAS is mounted as xfs.

I also deleted my old conda setup and did a fresh Anaconda install; but no good.

Also if it is useful, this is the output of jupyter kernelspec list

Available kernels:
  test-kernel    /nas/home/umanggup/.local/share/jupyter/kernels/test-kernel
  python3        /nas/home/umanggup/anaconda3/share/jupyter/kernels/python3

This is resolved. I reached out to IT. There was some issue with DNS and hence connection the NAS/storage which was causing this problem.

Thanks for all the help. I appreciate it.