Jupyter notebook get stucks or starts giving 504 exceptions

Bug description

We are using Jupyterhub with a custom spawner that spawns notebook on AWS ECS cluster. We are seeing multiple instances of jupyter notebook becoming unusable(Not able to save files/performance degradation) or 504 exceptions while accessing the notebook

I had even increased the proxy timeout but the issue is still occurring

# Proxy config
c.ConfigurableHTTPProxy.command = ['configurable-http-proxy',
                                   '--log-level', 'debug',
                                   '--timeout', '300000',  # 5 minutes
                                   '--proxy-timeout', '300000',  # 5 minutes
                                   '--statsd-host', 'statsd',
                                   '--statsd-port', '9125',
                                   '--statsd-prefix', 'chp']

Actual behaviour

image
image

Notebook debug logs

[D 2021-12-10 11:20:51.423 SingleUserNotebookApp filemanager:469] Saving /home/jovyan/jfs/keras_test.ipynb
[I 2021-12-10 11:22:19.932 SingleUserNotebookApp log:174] 200 PUT /user/ayush.chauhan/api/contents/keras_test.ipynb?1639135251107 (ayush.chauhan@42.104.69.45) 88510.67ms
[W 2021-12-10 11:22:19.935 SingleUserNotebookApp zmqhandlers:253] zmq message arrived on closed channel
[W 2021-12-10 11:22:19.936 SingleUserNotebookApp zmqhandlers:253] zmq message arrived on closed channel
[D 2021-12-10 11:22:19.936 SingleUserNotebookApp handlers:555] Websocket closed cfb6ea5c-6ed3-4553-a0b2-55902e13865d:1863d368-dbaa-4063-aabd-1fc4df8d8cb5
[D 2021-12-10 11:22:19.936 SingleUserNotebookApp handlers:555] Websocket closed cfb6ea5c-6ed3-4553-a0b2-55902e13865d:eced1597-c8f3-409c-a93c-2200fd7c4c01
[I 2021-12-10 11:22:19.936 SingleUserNotebookApp kernelmanager:222] Starting buffering for cfb6ea5c-6ed3-4553-a0b2-55902e13865d:eced1597-c8f3-409c-a93c-2200fd7c4c01
[D 2021-12-10 11:22:19.936 SingleUserNotebookApp kernelmanager:272] Clearing buffer for cfb6ea5c-6ed3-4553-a0b2-55902e13865d
[D 2021-12-10 11:22:19.937 SingleUserNotebookApp auth:310] HubAuth cache miss: token:2b1fc732711749e896b5c7a99af9a8b0:f6afbe1da270476f8731f3d18400874c
[D 2021-12-10 11:22:19.975 SingleUserNotebookApp auth:316] Received request from Hub user {'kind': 'user', 'name': 'ayush.chauhan', 'admin': True, 'groups': ['ml-ds', 'development'], 'server': '/user/ayush.chauhan/', 'pending': None, 'created': '2020-07-01T10:35:26Z', 'last_activity': '2021-12-10T11:22:19.953055Z', 'servers': None}
[D 2021-12-10 11:22:19.975 SingleUserNotebookApp auth:857] Allowing Hub admin ayush.chauhan
[D 2021-12-10 11:22:19.975 SingleUserNotebookApp auth:744] Setting oauth cookie for 42.104.69.45: jupyterhub-user-ayush.chauhan, {'path': '/user/ayush.chauhan/', 'httponly': True}
[D 2021-12-10 11:22:19.976 SingleUserNotebookApp zmqhandlers:293] Initializing websocket connection /user/ayush.chauhan/api/kernels/cfb6ea5c-6ed3-4553-a0b2-55902e13865d/channels
[D 2021-12-10 11:22:19.977 SingleUserNotebookApp auth:857] Allowing Hub admin ayush.chauhan
[D 2021-12-10 11:22:19.977 SingleUserNotebookApp auth:744] Setting oauth cookie for 42.104.69.45: jupyterhub-user-ayush.chauhan, {'path': '/user/ayush.chauhan/', 'httponly': True}
[W 2021-12-10 11:22:19.978 SingleUserNotebookApp zmqstream:442] Got events for closed stream <zmq.eventloop.zmqstream.ZMQStream object at 0x7f20235dafd0>
[D 2021-12-10 11:22:19.978 SingleUserNotebookApp kernelmanager:235] Buffering msg on cfb6ea5c-6ed3-4553-a0b2-55902e13865d:iopub
[D 2021-12-10 11:22:19.978 SingleUserNotebookApp kernelmanager:235] Buffering msg on cfb6ea5c-6ed3-4553-a0b2-55902e13865d:iopub
[I 2021-12-10 11:22:19.978 SingleUserNotebookApp handlers:164] Saving file at /keras_test.ipynb
[D 2021-12-10 11:22:19.978 SingleUserNotebookApp filemanager:469] Saving /home/jovyan/jfs/keras_test.ipynb
[I 2021-12-10 11:22:20.071 SingleUserNotebookApp log:174] 200 PUT /user/ayush.chauhan/api/contents/keras_test.ipynb?1639135251107 (ayush.chauhan@42.104.69.45) 133.96ms
[D 2021-12-10 11:22:20.071 SingleUserNotebookApp auth:857] Allowing Hub admin ayush.chauhan
[I 2021-12-10 11:22:20.072 SingleUserNotebookApp log:174] 200 GET /user/ayush.chauhan/api/terminals?1639135315307 (ayush.chauhan@42.104.69.45) 95.35ms
[D 2021-12-10 11:22:20.072 SingleUserNotebookApp singleuser:503] Notifying Hub of activity 2021-12-10T11:22:19.934928Z
[D 2021-12-10 11:22:20.073 SingleUserNotebookApp kernelmanager:235] Buffering msg on cfb6ea5c-6ed3-4553-a0b2-55902e13865d:iopub
[D 2021-12-10 11:22:20.073 SingleUserNotebookApp kernelmanager:235] Buffering msg on cfb6ea5c-6ed3-4553-a0b2-55902e13865d:iopub
[D 2021-12-10 11:22:20.074 SingleUserNotebookApp kernelmanager:235] Buffering msg on cfb6ea5c-6ed3-4553-a0b2-55902e13865d:iopub
[D 2021-12-10 11:22:20.074 SingleUserNotebookApp kernelmanager:235] Buffering msg on cfb6ea5c-6ed3-4553-a0b2-55902e13865d:iopub
[D 2021-12-10 11:22:20.074 SingleUserNotebookApp kernelmanager:235] Buffering msg on cfb6ea5c-6ed3-4553-a0b2-55902e13865d:iopub
[D 2021-12-10 11:22:20.075 SingleUserNotebookApp kernelmanager:235] Buffering msg on cfb6ea5c-6ed3-4553-a0b2-55902e13865d:iopub
[D 2021-12-10 11:22:20.075 SingleUserNotebookApp kernelmanager:235] Buffering msg on cfb6ea5c-6ed3-4553-a0b2-55902e13865d:iopub
[D 2021-12-10 11:22:20.075 SingleUserNotebookApp kernelmanager:235] Buffering msg on cfb6ea5c-6ed3-4553-a0b2-55902e13865d:iopub
[I 2021-12-10 11:22:20.076 SingleUserNotebookApp log:174] 101 GET /user/ayush.chauhan/api/kernels/cfb6ea5c-6ed3-4553-a0b2-55902e13865d/channels?session_id=1863d368-dbaa-4063-aabd-1fc4df8d8cb5 (ayush.chauhan@125.19.104.6) 100.30ms
[D 2021-12-10 11:22:20.076 SingleUserNotebookApp zmqhandlers:154] Opening websocket /user/ayush.chauhan/api/kernels/cfb6ea5c-6ed3-4553-a0b2-55902e13865d/channels
[D 2021-12-10 11:22:20.076 SingleUserNotebookApp kernelmanager:252] Getting buffer for cfb6ea5c-6ed3-4553-a0b2-55902e13865d
[D 2021-12-10 11:22:20.076 SingleUserNotebookApp kernelmanager:272] Clearing buffer for cfb6ea5c-6ed3-4553-a0b2-55902e13865d
[I 2021-12-10 11:22:20.076 SingleUserNotebookApp kernelmanager:287] Discarding 10 buffered messages for cfb6ea5c-6ed3-4553-a0b2-55902e13865d:eced1597-c8f3-409c-a93c-2200fd7c4c01
[D 2021-12-10 11:22:20.076 SingleUserNotebookApp connect:547] Connecting to: tcp://127.0.0.1:34887
[D 2021-12-10 11:22:20.076 SingleUserNotebookApp connect:547] Connecting to: tcp://127.0.0.1:42949
[D 2021-12-10 11:22:20.076 SingleUserNotebookApp connect:547] Connecting to: tcp://127.0.0.1:53383
[D 2021-12-10 11:22:20.077 SingleUserNotebookApp connect:547] Connecting to: tcp://127.0.0.1:40155
[D 2021-12-10 11:22:20.077 SingleUserNotebookApp handlers:151] Nudge: not nudging busy kernel cfb6ea5c-6ed3-4553-a0b2-55902e13865d
[W 2021-12-10 11:22:20.077 SingleUserNotebookApp zmqstream:442] Got events for closed stream <zmq.eventloop.zmqstream.ZMQStream object at 0x7f20234c2710>
[W 2021-12-10 11:22:20.077 SingleUserNotebookApp zmqstream:442] Got events for closed stream <zmq.eventloop.zmqstream.ZMQStream object at 0x7f20234c2710>
[D 2021-12-10 11:22:20.416 SingleUserNotebookApp zmqhandlers:293] Initializing websocket connection /user/ayush.chauhan/api/kernels/cfb6ea5c-6ed3-4553-a0b2-55902e13865d/channels
[D 2021-12-10 11:22:20.417 SingleUserNotebookApp auth:857] Allowing Hub admin ayush.chauhan
[I 2021-12-10 11:22:20.418 SingleUserNotebookApp log:174] 101 GET /user/ayush.chauhan/api/kernels/cfb6ea5c-6ed3-4553-a0b2-55902e13865d/channels?session_id=eced1597-c8f3-409c-a93c-2200fd7c4c01 (ayush.chauhan@42.104.69.45) 1.80ms
[D 2021-12-10 11:22:20.418 SingleUserNotebookApp zmqhandlers:154] Opening websocket /user/ayush.chauhan/api/kernels/cfb6ea5c-6ed3-4553-a0b2-55902e13865d/channels
[D 2021-12-10 11:22:20.418 SingleUserNotebookApp kernelmanager:252] Getting buffer for cfb6ea5c-6ed3-4553-a0b2-55902e13865d
[D 2021-12-10 11:22:20.418 SingleUserNotebookApp connect:547] Connecting to: tcp://127.0.0.1:34887
[D 2021-12-10 11:22:20.418 SingleUserNotebookApp connect:547] Connecting to: tcp://127.0.0.1:42949
[D 2021-12-10 11:22:20.418 SingleUserNotebookApp connect:547] Connecting to: tcp://127.0.0.1:53383
[D 2021-12-10 11:22:20.418 SingleUserNotebookApp connect:547] Connecting to: tcp://127.0.0.1:40155
[D 2021-12-10 11:22:20.419 SingleUserNotebookApp handlers:151] Nudge: not nudging busy kernel cfb6ea5c-6ed3-4553-a0b2-55902e13865d
[W 2021-12-10 11:22:19.935 SingleUserNotebookApp zmqhandlers:253] zmq message arrived on closed channel
[W 2021-12-10 11:22:19.936 SingleUserNotebookApp zmqhandlers:253] zmq message arrived on closed channel
[D 2021-12-10 11:22:19.936 SingleUserNotebookApp handlers:555] Websocket closed cfb6ea5c-6ed3-4553-a0b2-55902e13865d:1863d368-dbaa-4063-aabd-1fc4df8d8cb5
[D 2021-12-10 11:22:19.936 SingleUserNotebookApp handlers:555] Websocket closed cfb6ea5c-6ed3-4553-a0b2-55902e13865d:eced1597-c8f3-409c-a93c-2200fd7c4c01
[I 2021-12-10 11:22:19.936 SingleUserNotebookApp kernelmanager:222] Starting buffering for cfb6ea5c-6ed3-4553-a0b2-55902e13865d:eced1597-c8f3-409c-a93c-2200fd7c4c01
[D 2021-12-10 11:22:19.936 SingleUserNotebookApp kernelmanager:272] Clearing buffer for cfb6ea5c-6ed3-4553-a0b2-55902e13865d
[D 2021-12-10 11:22:19.937 SingleUserNotebookApp auth:310] HubAuth cache miss: token:2b1fc732711749e896b5c7a99af9a8b0:f6afbe1da270476f8731f3d18400874c
[D 2021-12-10 11:22:19.975 SingleUserNotebookApp auth:316] Received request from Hub user {'kind': 'user', 'name': 'ayush.chauhan', 'admin': True, 'groups': ['ml-ds', 'development'], 'server': '/user/ayush.chauhan/', 'pending': None, 'created': '2020-07-01T10:35:26Z', 'last_activity': '2021-12-10T11:22:19.953055Z', 'servers': None}
[I 2021-12-10 11:22:20.076 SingleUserNotebookApp log:174] 101 GET /user/ayush.chauhan/api/kernels/cfb6ea5c-6ed3-4553-a0b2-55902e13865d/channels?session_id=1863d368-dbaa-4063-aabd-1fc4df8d8cb5 (ayush.chauhan@125.19.104.6) 100.30ms
[D 2021-12-10 11:22:20.076 SingleUserNotebookApp zmqhandlers:154] Opening websocket /user/ayush.chauhan/api/kernels/cfb6ea5c-6ed3-4553-a0b2-55902e13865d/channels
[D 2021-12-10 11:22:20.076 SingleUserNotebookApp kernelmanager:252] Getting buffer for cfb6ea5c-6ed3-4553-a0b2-55902e13865d
[D 2021-12-10 11:22:20.076 SingleUserNotebookApp kernelmanager:272] Clearing buffer for cfb6ea5c-6ed3-4553-a0b2-55902e13865d
[I 2021-12-10 11:22:20.076 SingleUserNotebookApp kernelmanager:287] Discarding 10 buffered messages for cfb6ea5c-6ed3-4553-a0b2-55902e13865d:eced1597-c8f3-409c-a93c-2200fd7c4c01
[D 2021-12-10 11:22:20.076 SingleUserNotebookApp connect:547] Connecting to: tcp://127.0.0.1:34887
[D 2021-12-10 11:22:20.076 SingleUserNotebookApp connect:547] Connecting to: tcp://127.0.0.1:42949
[D 2021-12-10 11:22:20.076 SingleUserNotebookApp connect:547] Connecting to: tcp://127.0.0.1:53383
[D 2021-12-10 11:22:20.077 SingleUserNotebookApp connect:547] Connecting to: tcp://127.0.0.1:40155
[D 2021-12-10 11:22:20.077 SingleUserNotebookApp handlers:151] Nudge: not nudging busy kernel cfb6ea5c-6ed3-4553-a0b2-55902e13865d
[D 2021-12-10 11:22:20.077 SingleUserNotebookApp kernelmanager:425] activity on cfb6ea5c-6ed3-4553-a0b2-55902e13865d: update_display_data
[W 2021-12-10 11:22:20.077 SingleUserNotebookApp zmqstream:442] Got events for closed stream <zmq.eventloop.zmqstream.ZMQStream object at 0x7f20234c2710>
[W 2021-12-10 11:22:20.077 SingleUserNotebookApp zmqstream:442] Got events for closed stream <zmq.eventloop.zmqstream.ZMQStream object at 0x7f20234c2710>
Exception in callback ZMQChannelsHandler.open.<locals>.subscribe(<Future finished result=None>) at /opt/conda/lib/python3.7/site-packages/notebook/services/kernels/handlers.py:409
handle: <Handle ZMQChannelsHandler.open.<locals>.subscribe(<Future finished result=None>) at /opt/conda/lib/python3.7/site-packages/notebook/services/kernels/handlers.py:409>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/opt/conda/lib/python3.7/site-packages/notebook/services/kernels/handlers.py", line 411, in subscribe
    stream.on_recv_stream(self._on_zmq_reply)
  File "/opt/conda/lib/python3.7/site-packages/zmq/eventloop/zmqstream.py", line 189, in on_recv_stream
    self.on_recv(lambda msg: callback(self, msg), copy=copy)
  File "/opt/conda/lib/python3.7/site-packages/zmq/eventloop/zmqstream.py", line 168, in on_recv
    self._check_closed()
  File "/opt/conda/lib/python3.7/site-packages/zmq/eventloop/zmqstream.py", line 503, in _check_closed
    raise IOError("Stream is closed")
OSError: Stream is closed

Hub Logs

11:27:32.928 [ConfigProxy] error: 503 GET /user/ayush.chauhan/api/contents/keras_test.ipynb socket hang up

How to reproduce

Sample script used to reproduce the issue. If you run this script 2-3 times, notebook become unusable

import joblib
import numpy as np
import pandas as pd

def get_parallel_func(df, i):
    df = df.reset_index(drop=True)
    
    temp_arr = np.zeros((df.shape[0], 10, 17), np.float32)
    for j in range(df.shape[0]):
        temp_arr[j, :, :] = np.full((10, 17), 4.556789)*np.full((10, 17), 14.556792)
    
    np.save(f'test/item_{i}.npy', temp_arr)

def get_item_level_embeddings(df):
    m = (df.shape[0]//1000000)+1
    print('df shape: ', df.shape[0], ' and m is: ', m)
    print('Applying a function in df in a parallel manner..')
    joblib.Parallel(n_jobs=16)(joblib.delayed(get_parallel_func)(df.loc[i*1000000:((i+1)*1000000)-1], i) for i in range(m))
    
    for i in range(m):
        if i==0:
            arr = np.load(f'test/item_{i}.npy')
        else:
            temp_arr = np.load(f'test/item_{i}.npy')
            arr = np.vstack([arr, temp_arr])
    
    print(f'array shape: {arr.shape}')
    np.save('arr.npy', arr)

df = pd.DataFrame({'col_a': range(20000000)})
df['col_b'] = list(range(20000000))
get_item_level_embeddings(df)

Your personal set up

  • OS: ubuntu 20.04
  • Jupyter Hub parent docker image: jupyterhub/jupyterhub:1.4.1
  • Jupyter Notebook parent docker image: jupyter/base-notebook:hub-1.4.1
  • Version(s): 1.4.1
Jupyter Configuration
import requests
import sys
import os

from jupyterhub.auth import DummyAuthenticator
from oauthenticator.google import GoogleOAuthenticator


def get_host_ip():
    url = "http://169.254.169.254/latest/meta-data/local-ipv4"
    payload = {}
    headers = {}
    try:
        response = requests.request("GET", url, headers=headers, data=payload, timeout=1)
        host_ip = response.text
    except requests.exceptions.ConnectTimeout:
        host_ip = "127.0.0.1"
    return host_ip


c = get_config()

# c.JupyterHub.authenticator_class = DummyAuthenticator
c.JupyterHub.authenticator_class = GoogleOAuthenticator
c.GoogleOAuthenticator.hosted_domain = ['domain.com']
c.Authenticator.admin_users = {"ayush.chauhan"}

# Custom spawner config
c.JupyterHub.spawner_class = "ecsspawner.EcsTaskSpawner"
c.Spawner.environment = {'JUPYTER_ENABLE_LAB': 'yes'}
c.Spawner.available_kernels = {}
c.Spawner.available_notebook = {}

c.Spawner.available_notebook_task_roles = {}

# Timeout (in seconds) before giving up on a spawned HTTP server
c.Spawner.http_timeout = 600
c.Spawner.start_timeout = 600

c.Spawner.default_url = '/lab'

# The ip address for the Hub process to *bind* to.
c.JupyterHub.hub_ip = "0.0.0.0"
# IP address that other services(spawn notebooks) should use to connect to the Hub
c.JupyterHub.hub_connect_ip = get_host_ip()

# Disabling prometheus authentication
c.JupyterHub.authenticate_prometheus = False

# Shuts down all user servers on logout
c.JupyterHub.shutdown_on_logout = True

# Proxy config
c.ConfigurableHTTPProxy.command = ['configurable-http-proxy',
                                   '--log-level', 'debug',
                                   '--timeout', '300000',  # 5 minutes
                                   '--proxy-timeout', '300000',  # 5 minutes
                                   '--statsd-host', 'statsd',
                                   '--statsd-port', '9125',
                                   '--statsd-prefix', 'chp']

# cull_idle service to handle inactive notebooks
c.JupyterHub.services = [
    {
        "name": "cull-idle",
        "admin": True,
        "command": [sys.executable,
                    "/etc/jupyterhub/cull_idle_servers.py",
                    f"--timeout={os.getenv('JUPYTERHUB_CULL_IDLE_TIMEOUT')}",
                    f"--max_age={os.getenv('JUPYTERHUB_CULL_IDLE_MAX_AGE')}",
                    "--url=http://127.0.0.1:8081/hub/api"]
    }
]

c.JupyterHub.admin_access = True

# Path to SSL certificate file for the public facing interface of the proxy
#  When setting this, you should also set ssl_key
c.JupyterHub.ssl_cert = ''

# Path to SSL key file for the public facing interface of the proxy
#  When setting this, you should also set ssl_cert
c.JupyterHub.ssl_key = ''

# Allow named single-user servers per user
c.JupyterHub.allow_named_servers = False

# Will retrieve from ECS env through SSM
mysql_pass = os.getenv('JUPYTERHUB_MYSQL_PASSWORD')
mysql_host = os.getenv('JUPYTERHUB_MYSQL_HOST')
mysql_user = os.getenv('JUPYTERHUB_MYSQL_USER')
mysql_db = os.getenv('JUPYTERHUB_MYSQL_DB')
c.JupyterHub.db_url = f"mysql+mysqlconnector://{mysql_user}:{mysql_pass}@{mysql_host}:3306/{mysql_db}"

Full environment

# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
aiobotocore               1.4.2                    pypi_0    pypi
aiohttp                   3.8.1                    pypi_0    pypi
aioitertools              0.8.0                    pypi_0    pypi
aiosignal                 1.2.0                    pypi_0    pypi
alembic                   1.6.5              pyhd8ed1ab_0    conda-forge
anyio                     3.4.0            py37h89c1867_0    conda-forge
argon2-cffi               21.1.0           py37h5e8e339_2    conda-forge
asn1crypto                1.4.0                    pypi_0    pypi
async-timeout             4.0.1                    pypi_0    pypi
async_generator           1.10                       py_0    conda-forge
asynctest                 0.13.0                   pypi_0    pypi
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
autovizwidget             0.19.1                   pypi_0    pypi
babel                     2.9.1              pyh44b312d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
bcrypt                    3.2.0                    pypi_0    pypi
bleach                    4.0.0              pyhd8ed1ab_0    conda-forge
blinker                   1.4                        py_1    conda-forge
boto                      2.49.0                   pypi_0    pypi
boto3                     1.9.253                  pypi_0    pypi
botocore                  1.20.106                 pypi_0    pypi
brotlipy                  0.7.0           py37h5e8e339_1003    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.17.1               h7f98852_1    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
cached-property           1.5.2                    pypi_0    pypi
cachetools                4.2.4                    pypi_0    pypi
certifi                   2021.10.8        py37h89c1867_1    conda-forge
certipy                   0.1.3                      py_0    conda-forge
cffi                      1.14.6           py37hc58025e_0    conda-forge
chardet                   3.0.4           py37he5f6b98_1008    conda-forge
charset-normalizer        2.0.0              pyhd8ed1ab_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
conda                     4.10.3           py37h89c1867_4    conda-forge
conda-package-handling    1.7.3            py37h5e8e339_1    conda-forge
configurable-http-proxy   4.5.0           node14_hfc12e6c_0    conda-forge
coverage                  6.1.2                    pypi_0    pypi
cryptography              2.4.2                    pypi_0    pypi
cycler                    0.11.0                   pypi_0    pypi
decorator                 5.0.9              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
dill                      0.3.4                    pypi_0    pypi
dnspython                 1.16.0                   pypi_0    pypi
docutils                  0.15.2                   pypi_0    pypi
entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
frozenlist                1.2.0                    pypi_0    pypi
fsspec                    2021.7.0                 pypi_0    pypi
future                    0.18.2                   pypi_0    pypi
fuzzywuzzy                0.17.0                   pypi_0    pypi
gcsfs                     2021.11.0                pypi_0    pypi
gensim                    3.7.3                    pypi_0    pypi
geographiclib             1.52                     pypi_0    pypi
geopy                     1.21.0                   pypi_0    pypi
google-api-core           2.2.2                    pypi_0    pypi
google-auth               1.35.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-cloud-core         2.2.1                    pypi_0    pypi
google-cloud-storage      1.43.0                   pypi_0    pypi
google-crc32c             1.3.0                    pypi_0    pypi
google-resumable-media    2.1.0                    pypi_0    pypi
googleads                 23.0.1                   pypi_0    pypi
googleapis-common-protos  1.53.0                   pypi_0    pypi
greenlet                  1.1.2                    pypi_0    pypi
gspread                   4.0.1                    pypi_0    pypi
gssapi                    1.7.2                    pypi_0    pypi
h11                       0.12.0                   pypi_0    pypi
hdijupyterutils           0.19.1                   pypi_0    pypi
hybridcontents            0.3.0                    pypi_0    pypi
icu                       68.1                 h58526e2_0    conda-forge
idna                      2.8                      pypi_0    pypi
importlib-metadata        4.8.2            py37h89c1867_0    conda-forge
iniconfig                 1.1.1                    pypi_0    pypi
ipykernel                 5.5.5            py37h085eea5_0    conda-forge
ipython                   7.29.0           py37h6531663_2    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.6.5                    pypi_0    pypi
isodate                   0.6.0                    pypi_0    pypi
jedi                      0.18.1           py37h89c1867_0    conda-forge
jinja2                    3.0.1              pyhd8ed1ab_0    conda-forge
jmespath                  0.10.0                   pypi_0    pypi
joblib                    0.13.2                   pypi_0    pypi
json5                     0.9.5              pyh9f0ad1d_0    conda-forge
jsonnet                   0.16.0                   pypi_0    pypi
jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
jupyter                   1.0.0                    pypi_0    pypi
jupyter-autotime          1.1.0                    pypi_0    pypi
jupyter-console           6.4.0                    pypi_0    pypi
jupyter_client            6.1.12             pyhd8ed1ab_0    conda-forge
jupyter_core              4.9.1            py37h89c1867_1    conda-forge
jupyter_server            1.10.2             pyhd8ed1ab_0    conda-forge
jupyter_telemetry         0.1.0              pyhd8ed1ab_1    conda-forge
jupyterhub                1.1.0                    py37_0    conda-forge
jupyterlab                3.1.4              pyhd8ed1ab_0    conda-forge
jupyterlab-widgets        1.0.2                    pypi_0    pypi
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
jupyterlab_server         2.6.2              pyhd8ed1ab_0    conda-forge
kafka-python              2.0.2                    pypi_0    pypi
kiwisolver                1.3.2                    pypi_0    pypi
krb5                      0.2.0                    pypi_0    pypi
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libarchive                3.5.1                hccf745f_2    conda-forge
libcurl                   7.78.0               h2574ce0_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 11.1.0               hc902ee8_8    conda-forge
libgomp                   11.1.0               hc902ee8_8    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
libnghttp2                1.43.0               h812cca2_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsolv                   0.7.19               h780b84a_5    conda-forge
libssh2                   1.9.0                ha56f1ee_6    conda-forge
libstdcxx-ng              11.1.0               h56837e0_8    conda-forge
libuv                     1.42.0               h7f98852_0    conda-forge
libxml2                   2.9.12               h72842e0_0    conda-forge
lightgbm                  2.2.3                    pypi_0    pypi
lxml                      4.6.4                    pypi_0    pypi
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mako                      1.1.4              pyh44b312d_0    conda-forge
mamba                     0.15.3           py37h7f483ca_0    conda-forge
markupsafe                2.0.1            py37h5e8e339_1    conda-forge
matplotlib                3.3.4                    pypi_0    pypi
matplotlib-inline         0.1.2              pyhd8ed1ab_2    conda-forge
mistune                   0.8.4           py37h5e8e339_1005    conda-forge
mock                      4.0.3                    pypi_0    pypi
mpmath                    1.2.1                    pypi_0    pypi
multidict                 5.2.0                    pypi_0    pypi
nbclassic                 0.3.1              pyhd8ed1ab_1    conda-forge
nbclient                  0.5.3              pyhd8ed1ab_0    conda-forge
nbconvert                 6.3.0            py37h89c1867_1    conda-forge
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
nbval                     0.9.6                    pypi_0    pypi
ncurses                   6.2                  h58526e2_4    conda-forge
nest-asyncio              1.5.1              pyhd8ed1ab_0    conda-forge
nltk                      3.4.5                    pypi_0    pypi
nodejs                    14.17.4              h92b4a50_0    conda-forge
nomkl                     1.0                  h5ca1d4c_0    conda-forge
nose                      1.3.7                    pypi_0    pypi
notebook                  6.4.2              pyha770c72_0    conda-forge
numpy                     1.18.5                   pypi_0    pypi
oauthlib                  3.1.1              pyhd8ed1ab_0    conda-forge
openssl                   1.1.1l               h7f98852_0    conda-forge
outcome                   1.1.0                    pypi_0    pypi
packaging                 21.0               pyhd8ed1ab_0    conda-forge
pamela                    1.0.0                      py_0    conda-forge
pandarallel               1.5.4                    pypi_0    pypi
pandas                    1.0.4                    pypi_0    pypi
pandasql                  0.7.3                    pypi_0    pypi
pandoc                    2.14.1               h7f98852_0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
paramiko                  2.4.3                    pypi_0    pypi
parso                     0.8.2              pyhd8ed1ab_0    conda-forge
patsy                     0.5.2                    pypi_0    pypi
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    8.4.0                    pypi_0    pypi
pip                       21.0.1                   pypi_0    pypi
platformdirs              2.4.0                    pypi_0    pypi
plotly                    3.7.1                    pypi_0    pypi
pluggy                    1.0.0                    pypi_0    pypi
prometheus-client         0.8.0                    pypi_0    pypi
prompt-toolkit            3.0.19             pyha770c72_0    conda-forge
protobuf                  3.19.1                   pypi_0    pypi
psutil                    5.6.7                    pypi_0    pypi
psycopg2                  2.8.6                    pypi_0    pypi
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
py                        1.11.0                   pypi_0    pypi
pyarabic                  0.6.14                   pypi_0    pypi
pyarrow                   0.17.1                   pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycosat                   0.6.3           py37h5e8e339_1009    conda-forge
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pycrypto                  2.6.1                    pypi_0    pypi
pycryptodomex             3.11.0                   pypi_0    pypi
pycurl                    7.44.1           py37h88a64d2_0    conda-forge
pygments                  2.9.0              pyhd8ed1ab_0    conda-forge
pyhive                    0.6.1                    pypi_0    pypi
pyjumbo                   1.6.0                    pypi_0    pypi
pyjwt                     2.1.0              pyhd8ed1ab_0    conda-forge
pymongo                   3.9.0                    pypi_0    pypi
pynacl                    1.4.0                    pypi_0    pypi
pyopenssl                 19.0.0                   pypi_0    pypi
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyrsistent                0.18.0           py37h5e8e339_0    conda-forge
pysftp                    0.2.9                    pypi_0    pypi
pysocks                   1.7.1            py37h89c1867_4    conda-forge
pyspnego                  0.3.1                    pypi_0    pypi
pytest                    6.2.5                    pypi_0    pypi
pytest-cov                3.0.0                    pypi_0    pypi
python                    3.7.9           hffdb5ce_100_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-editor             1.0.4                      py_0    conda-forge
python-http-client        3.3.3                    pypi_0    pypi
python-json-logger        2.0.1              pyh9f0ad1d_0    conda-forge
python-levenshtein        0.12.2                   pypi_0    pypi
python_abi                3.7                     2_cp37m    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
pyxlsb                    1.0.9                    pypi_0    pypi
pyyaml                    5.4.1                    pypi_0    pypi
pyzipper                  0.3.3                    pypi_0    pypi
pyzmq                     22.3.0           py37h336d617_1    conda-forge
qtconsole                 5.2.1                    pypi_0    pypi
qtpy                      1.11.2                   pypi_0    pypi
readline                  8.1                  h46c0cb4_0    conda-forge
reproc                    14.2.1               h36c2ea0_0    conda-forge
reproc-cpp                14.2.1               h58526e2_0    conda-forge
requests                  2.25.1                   pypi_0    pypi
requests-file             1.5.1                    pypi_0    pypi
requests-kerberos         0.13.0                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
requests-toolbelt         0.9.1                    pypi_0    pypi
requests-unixsocket       0.2.0                      py_0    conda-forge
retrying                  1.3.3                    pypi_0    pypi
rsa                       4.8                      pypi_0    pypi
ruamel                    1.0              py37h89c1867_5    conda-forge
ruamel.yaml               0.16.5           py37h516909a_0    conda-forge
ruamel_yaml               0.15.80         py37h5e8e339_1006    conda-forge
s2cell                    1.4.0                    pypi_0    pypi
s2sphere                  0.2.5                    pypi_0    pypi
s3contents                0.7.0                    pypi_0    pypi
s3fs                      2021.8.0                 pypi_0    pypi
s3transfer                0.2.1                    pypi_0    pypi
scikit-learn              0.19.2                   pypi_0    pypi
scipy                     1.7.3                    pypi_0    pypi
selenium                  4.1.0                    pypi_0    pypi
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
sendgrid                  6.6.0                    pypi_0    pypi
setuptools                56.0.0                   pypi_0    pypi
shapely                   1.6.4.post2              pypi_0    pypi
six                       1.16.0             pyh6c4a22f_0    conda-forge
smart-open                5.2.1                    pypi_0    pypi
sniffio                   1.2.0            py37h89c1867_2    conda-forge
sortedcontainers          2.4.0                    pypi_0    pypi
sparkmagic                0.19.1                   pypi_0    pypi
sqlalchemy                1.4.0            py37h5e8e339_0    conda-forge
sqlite                    3.36.0               h9cd32fc_0    conda-forge
starkbank-ecdsa           2.0.3                    pypi_0    pypi
statsmodels               0.9.0                    pypi_0    pypi
sympy                     1.5.1                    pypi_0    pypi
tenacity                  5.0.4                    pypi_0    pypi
terminado                 0.12.1           py37h89c1867_1    conda-forge
testpath                  0.5.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.10               h21135ba_1    conda-forge
toml                      0.10.2                   pypi_0    pypi
tomli                     1.2.2                    pypi_0    pypi
toolz                     0.9.0                    pypi_0    pypi
tornado                   6.1              py37h5e8e339_2    conda-forge
tqdm                      4.26.0                   pypi_0    pypi
traitlets                 5.0.5                      py_0    conda-forge
trio                      0.19.0                   pypi_0    pypi
trio-websocket            0.9.2                    pypi_0    pypi
typing_extensions         4.0.0              pyha770c72_0    conda-forge
tzdata                    2021a                he74cb21_1    conda-forge
unidecode                 1.1.2                    pypi_0    pypi
urllib3                   1.26.7                   pypi_0    pypi
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          1.2.1            py37h89c1867_0    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
widgetsnbextension        3.5.2                    pypi_0    pypi
wrapt                     1.13.3                   pypi_0    pypi
wsproto                   1.0.0                    pypi_0    pypi
xgboost                   0.90                     pypi_0    pypi
xlrd                      1.2.0                    pypi_0    pypi
xmltodict                 0.12.0                   pypi_0    pypi
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h516909a_0    conda-forge
yarl                      1.7.2                    pypi_0    pypi
zeep                      4.1.0                    pypi_0    pypi
zeromq                    4.3.4                h9c3ff4c_0    conda-forge
zipp                      3.5.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
zstd                      1.5.0                ha95c52a_0    conda-forge

It looks like the version of JupyterHub in your environment is 1.1.0, not 1.4.1 as you thought:

This is worth fixing.

Can you share your JupyterHub logs with debug logging enabled?

Have you checked the resource usage of JupyerHub and Jupyter notebook?

I have fixed the jupyterhub dependency version but the issue is still happening.

Reattaching the Hub Logs

11:27:32.928 [ConfigProxy] error: 503 GET /user/ayush.chauhan/api/contents/keras_test.ipynb socket hang up

I have checked the resource usage of the notebook, they are well within the limit.

To help rule out some other problems, can you reproduce your error when using a standard jupyter/docker-stacks image Selecting an Image — docker-stacks latest documentation and a notebook that doesn’t use joblib? The reason it’d be good to try without joblib is that it does some parallel processing which might affect Jupyter.

Could you also share your full debug logs from the singleuser image?

1 Like