[ConfigProxy] error: 503 socket hang up

JupyterHub
,
#1

Installed jupyterhub in a docker container and trying to run in cloud. but i am getting this error. Been through tons of such post and tried all of them and yet i am having this issue. The same docker container in my local system works fine. How do i fix this?

Your personal set up

  • OS: ubuntu 16.04
  • Version(s):
  • jupyterhub --version
    1.3.0

jupyter --version
jupyter core : 4.7.1
jupyter-notebook : 6.2.0
qtconsole : 5.0.3
ipython : 7.21.0
ipykernel : 5.5.0
jupyter client : 6.1.12
jupyter lab : 3.0.12
nbconvert : 6.0.7
ipywidgets : 7.6.3
nbformat : 5.1.2
traitlets : 5.0.5

  • Full environment 
# paste output of `pip freeze` or `conda list` here
- alembic @ file:///tmp/wheelhouse/alembic-1.4.3-py2.py3-none-any.whl
anyio==2.2.0
argon2-cffi==20.1.0
async-generator @ file:///tmp/wheelhouse/async_generator-1.10-py3-none-any.whl
attrs @ file:///tmp/wheelhouse/attrs-20.3.0-py2.py3-none-any.whl
Babel==2.9.0
backcall==0.2.0
bleach==3.3.0
certifi @ file:///tmp/wheelhouse/certifi-2020.12.5-py2.py3-none-any.whl
certipy @ file:///tmp/wheelhouse/certipy-0.1.3-py3-none-any.whl
cffi @ file:///tmp/wheelhouse/cffi-1.14.4-cp38-cp38-manylinux1_x86_64.whl
chardet @ file:///tmp/wheelhouse/chardet-3.0.4-py2.py3-none-any.whl
cryptography @ file:///tmp/wheelhouse/cryptography-3.3.1-cp36-abi3-manylinux2010_x86_64.whl
decorator==4.4.2
defusedxml==0.7.1
entrypoints @ file:///tmp/wheelhouse/entrypoints-0.3-py2.py3-none-any.whl
idna @ file:///tmp/wheelhouse/idna-2.10-py2.py3-none-any.whl
ipykernel==5.5.0
ipython==7.21.0
ipython-genutils @ file:///tmp/wheelhouse/ipython_genutils-0.2.0-py2.py3-none-any.whl
ipywidgets==7.6.3
jedi==0.18.0
Jinja2 @ file:///tmp/wheelhouse/Jinja2-2.11.2-py2.py3-none-any.whl
json5==0.9.5
jsonschema @ file:///tmp/wheelhouse/jsonschema-3.2.0-py2.py3-none-any.whl
jupyter==1.0.0
jupyter-client==6.1.12
jupyter-console==6.3.0
jupyter-core==4.7.1
jupyter-packaging==0.7.12
jupyter-server==1.4.1
jupyter-telemetry @ file:///tmp/wheelhouse/jupyter_telemetry-0.1.0-py3-none-any.whl
jupyterhub @ file:///tmp/wheelhouse/jupyterhub-1.3.0-py3-none-any.whl
jupyterlab==3.0.12
jupyterlab-pygments==0.1.2
jupyterlab-server==2.3.0
jupyterlab-widgets==1.0.0
Mako @ file:///tmp/wheelhouse/Mako-1.1.3-py2.py3-none-any.whl
MarkupSafe @ file:///tmp/wheelhouse/MarkupSafe-1.1.1-cp38-cp38-manylinux1_x86_64.whl
mistune==0.8.4
nbclassic==0.2.6
nbclient==0.5.3
nbconvert==6.0.7
nbformat==5.1.2
nest-asyncio==1.5.1
notebook==6.2.0
oauthlib @ file:///tmp/wheelhouse/oauthlib-3.1.0-py2.py3-none-any.whl
packaging==20.9
pamela @ file:///tmp/wheelhouse/pamela-1.0.0-py2.py3-none-any.whl
pandocfilters==1.4.3
parso==0.8.1
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client @ file:///tmp/wheelhouse/prometheus_client-0.9.0-py2.py3-none-any.whl
prompt-toolkit==3.0.17
ptyprocess==0.7.0
pycparser @ file:///tmp/wheelhouse/pycparser-2.20-py2.py3-none-any.whl
pycurl==7.43.0.2
Pygments==2.8.1
pyOpenSSL @ file:///tmp/wheelhouse/pyOpenSSL-20.0.0-py2.py3-none-any.whl
pyparsing==2.4.7
pyrsistent @ file:///tmp/wheelhouse/pyrsistent-0.17.3-cp38-cp38-linux_x86_64.whl
python-dateutil @ file:///tmp/wheelhouse/python_dateutil-2.8.1-py2.py3-none-any.whl
python-editor @ file:///tmp/wheelhouse/python_editor-1.0.4-py3-none-any.whl
python-json-logger @ file:///tmp/wheelhouse/python_json_logger-2.0.1-py34-none-any.whl
pytz==2021.1
pyzmq==22.0.3
qtconsole==5.0.3
QtPy==1.9.0
requests @ file:///tmp/wheelhouse/requests-2.25.0-py2.py3-none-any.whl
ruamel.yaml @ file:///tmp/wheelhouse/ruamel.yaml-0.16.12-py2.py3-none-any.whl
ruamel.yaml.clib @ file:///tmp/wheelhouse/ruamel.yaml.clib-0.2.2-cp38-cp38-manylinux1_x86_64.whl
Send2Trash==1.5.0
six @ file:///tmp/wheelhouse/six-1.15.0-py2.py3-none-any.whl
sniffio==1.2.0
SQLAlchemy @ file:///tmp/wheelhouse/SQLAlchemy-1.3.20-cp38-cp38-manylinux2010_x86_64.whl
terminado==0.9.3
testpath==0.4.4
tornado==6.1
traitlets @ file:///tmp/wheelhouse/traitlets-5.0.5-py3-none-any.whl
urllib3 @ file:///tmp/wheelhouse/urllib3-1.26.2-py2.py3-none-any.whl
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.5.1
  • Configuration 
# jupyterhub_config.py

c.Authenticator.admin_users = {'dev'}
c.JupyterHub.admin_access = True
c.Authenticator.delete_invalid_users = True
c.JupyterHub.ip = '0.0.0.0'
c.JupyterHub.port = 8900

origin = '*'
c.JupyterHub.tornado_settings = {
    'headers': {
      'Access-Control-Allow-Origin': origin,
   },
}
c.Spawner.args = ['--NotebookApp.allow_origin=*']
  • Logs 
# [I 2021-03-22 10:56:11.735 SingleUserNotebookApp handlers:164] Saving file at /Untitled.ipynb
[I 2021-03-22 10:56:11.748 SingleUserNotebookApp log:181] 200 PUT /user/dev/api/contents/Untitled.ipynb (dev@172.17.0.1) 13.25ms
10:56:54.704 [ConfigProxy] error: 503 GET /user/dev/api/kernels/b2f3695b-bc19-4fcb-959c-26c7d09fd367/channels?session_id=304d2084af0f4c3681ee4cfbbb8e74a9 socket hang up
[I 2021-03-22 10:56:54.706 JupyterHub log:181] 200 GET /hub/error/503?url=%2Fuser%2Fdev%2Fapi%2Fkernels%2Fb2f3695b-bc19-4fcb-959c-26c7d09fd367%2Fchannels%3Fsession_id%3D304d2084af0f4c3681ee4cfbbb8e74a9 (@127.0.0.1) 1.19ms
[I 2021-03-22 10:56:59.725 JupyterHub log:181] 200 GET /hub/api/authorizations/token/[secret] (dev@127.0.0.1) 14.67ms
[W 2021-03-22 10:56:59.726 SingleUserNotebookApp handlers:366] Replacing stale connection: b2f3695b-bc19-4fcb-959c-26c7d09fd367:304d2084af0f4c3681ee4cfbbb8e74a9
10:57:59.709 [ConfigProxy] error: 503 GET /user/dev/api/kernels/b2f3695b-bc19-4fcb-959c-26c7d09fd367/channels?session_id=304d2084af0f4c3681ee4cfbbb8e74a9 socket hang up
[I 2021-03-22 10:57:59.711 JupyterHub log:181] 200 GET /hub/error/503?url=%2Fuser%2Fdev%2Fapi%2Fkernels%2Fb2f3695b-bc19-4fcb-959c-26c7d09fd367%2Fchannels%3Fsession_id%3D304d2084af0f4c3681ee4cfbbb8e74a9 (@127.0.0.1) 1.21ms
[W 2021-03-22 10:58:08.688 SingleUserNotebookApp handlers:366] Replacing stale connection: b2f3695b-bc19-4fcb-959c-26c7d09fd367:304d2084af0f4c3681ee4cfbbb8e74a9
10:59:08.737 [ConfigProxy] error: 503 GET /user/dev/api/kernels/b2f3695b-bc19-4fcb-959c-26c7d09fd367/channels?session_id=304d2084af0f4c3681ee4cfbbb8e74a9 socket hang up
#2

You say you are trying to run ‘in the cloud’. Can you provide more details on your deployment? On what service? Are there any cloud load-balancers or proxies or something like that?

The errors are on websocket endpoints, which suggests that there is a proxy layer between jupyterhub’s configurable-http-proxy and your browser somewhere that isn’t handling websocket requests correctly.

#3

Are you using an nginx container to proxy to the jupyterhub container? Check the proxy setup info: Using a reverse proxy — JupyterHub 1.3.0 documentation

#4

So i am running on nginx in a amazon cloud server. Can you help me with configuring the http-proxy. I am not sure how it works.

#5

Yes i am and i am not able to understand the proxy setup, I have it addressing to 0.0.0.0:8900, as you can see my configuration file. And its been set to the domain name. I am sorry i am unaware of how it works. can you help with configuring it, because i can see the nginx/sites.enabled folder but i don’t know what exactly to put in there.

#6

configurable-http-proxy is unlikely to be the issue. You should be able to leave that alone. What you need to do is make sure your nginx reverse proxy supports websockets. This is covered in the link @markperri shared, but the key part is this:

# top-level http config for websocket headers
# If Upgrade is defined, Connection = upgrade
# If Upgrade is empty, Connection = close
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

...
        # websocket headers
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header X-Scheme $scheme;
#7

So i just create the file as you mentioned and put it in the site.enabled folder? Would that work? or do i have to do some modifications to it? I am asking because i have no clue about nginx or proxy, or how it handles etc.

#8

Yes, you could put an nginx configuration file in sites.enabled, or edit the root nginx.conf, depending on how nginx is set up.

How have you set up nginx already to point to jupyterhub? Wherever that configuration already is is what you need to modify. I’d recommend using exactly the nginx configuration in the jupyterhub docs, only changing the target ip address and the paths to your SSL certificates.

#9

I’m not sure if it matters, but I put the nginx config into /etc/nginx/conf.d/jupyterhub.conf
Make sure you delete the /etc/nginx/conf.d/default.conf as it sets up a default server that you probably don’t want.

#10

I’ve been running into the same error message on a multi-user system, but only for one user. Everyone else but that user can login at the moment and most of the time this user can use JupyterHub ok. I noticed two of the lines above were different from our nginx configuration

				proxy_http_version 1.1;
                proxy_set_header X-Scheme $scheme;

I restarted nginx but still no luck.
Some details of my setup, Ubuntu 18.04.5 LTS,

$ python --version
Python 3.6.9
$ jupyterhub --version
1.4.1
$ jupyter --version
jupyter core     : 4.7.1
jupyter-notebook : 6.4.0
qtconsole        : 5.1.0
ipython          : 7.16.1
ipykernel        : 5.5.5
jupyter client   : 6.1.12
jupyter lab      : 3.0.16
nbconvert        : 6.0.7
ipywidgets       : 7.6.3
nbformat         : 5.1.3
traitlets        : 4.3.3
$ pip freeze
affine==2.3.0
alabaster==0.7.12
alembic==1.6.5
altair==4.1.0
anyio==3.1.0
appdirs==1.4.4
argon2-cffi==20.1.0
astropy==4.1
async-generator==1.10
attrs==21.2.0
Babel==2.9.1
backcall==0.2.0
baseband==3.1.1
black==21.6b0
bleach==3.3.0
bokeh==2.3.2
branca==0.4.2
centerline==0.6.3
certifi==2021.5.30
certipy==0.1.3
cffi==1.14.5
chardet==4.0.0
chart-studio==1.1.0
click==8.0.1
click-plugins==1.1.1
cligj==0.7.2
cloudpickle==1.6.0
colour==0.1.5
contextily==1.1.0
contextvars==2.4
convertdate==2.3.2
coverage==5.5
cryptography==3.4.7
cycler==0.10.0
Cython==0.29.23
dask==2021.3.0
dataclasses==0.8
decorator==4.4.2
defusedxml==0.7.1
descartes==1.1.0
distlib==0.3.2
docutils==0.17.1
entrypoints==0.3
filelock==3.0.12
Fiona==1.8.20
folium==0.12.1
frogtips==3.0.5
future==0.18.2
geographiclib==1.50
geopandas==0.9.0
geopy==2.1.0
great-expectations==0.13.19
greenlet==1.1.0
hide-code==0.2.0
hijri-converter==2.1.2
holidays==0.11.1
hyperopt==0.2.5
idna==2.10
imageio==2.9.0
imagesize==1.2.0
immutables==0.15
importlib-metadata==4.4.0
importlib-resources==5.1.4
iniconfig==1.1.1
ipykernel==5.5.5
ipython==7.16.1
ipython-genutils==0.2.0
ipywidgets==7.6.3
jedi==0.18.0
jenkspy==0.2.0
Jinja2==3.0.1
joblib==1.0.1
json5==0.9.5
jsonpatch==1.32
jsonpointer==2.1
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.12
jupyter-console==6.4.0
jupyter-core==4.7.1
jupyter-server==1.8.0
jupyter-telemetry==0.1.0
jupyterhub==1.4.1
jupyterlab==3.0.16
jupyterlab-pygments==0.1.2
jupyterlab-server==2.6.0
jupyterlab-widgets==1.0.0
jupyterthemes==0.20.0
kaleido==0.2.1
kiwisolver==1.3.1
korean-lunar-calendar==0.2.1
lesscpy==0.14.0
lorem==0.1.1
Mako==1.1.4
mapclassify==2.4.2
MarkupSafe==2.0.1
matplotlib==3.3.4
mercantile==1.2.1
mistune==0.8.4
munch==2.5.0
mypy-extensions==0.4.3
nbclassic==0.3.1
nbclient==0.5.3
nbconvert==6.0.7
nbformat==5.1.3
nest-asyncio==1.5.1
networkx==2.5.1
notebook==6.4.0
numpy==1.19.5
oauthlib==3.1.1
packaging==20.9
pamela==1.0.0
pandas==1.1.5
pandocfilters==1.4.3
parso==0.8.2
pathspec==0.8.1
patsy==0.5.1
pdfkit==0.6.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.2.0
pipenv==2021.5.29
plotly==4.14.3
pluggy==0.13.1
ply==3.11
prometheus-client==0.11.0
prompt-toolkit==3.0.18
psycopg2==2.8.6
psycopg2-binary==2.8.6
ptyprocess==0.7.0
py==1.10.0
pycparser==2.20
Pygments==2.9.0
PyMeeus==0.5.11
pyOpenSSL==20.0.1
pyparsing==2.4.7
pyproj==3.0.1
pyrsistent==0.17.3
pytest==6.2.4
pytest-cov==2.12.1
python-dateutil==2.8.1
python-editor==1.0.4
python-json-logger==2.0.1
pytz==2021.1
PyYAML==5.4.1
pyzmq==22.1.0
qtconsole==5.1.0
QtPy==1.9.0
rasterio==1.2.4
regex==2021.4.4
requests==2.25.1
retrying==1.3.3
-e git+https://github.com/CityofToronto/bdit_python_utilities.git@e33d8354e1fe354bf7f239f8598d5ddcf97797d3#egg=rick&subdirectory=charts
ruamel.yaml==0.17.7
ruamel.yaml.clib==0.2.2
scikit-learn==0.24.2
scipy==1.5.4
seaborn==0.11.1
Send2Trash==1.5.0
Shapely==1.7.1
simanneal==0.5.0
six==1.16.0
sniffio==1.2.0
snowballstemmer==2.1.0
snuggs==1.4.7
Sphinx==4.0.2
sphinx-gallery==0.9.0
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
SQLAlchemy==1.4.17
statsmodels==0.12.2
sudospawner==0.5.2
termcolor==1.1.0
terminado==0.10.0
testpath==0.5.0
threadpoolctl==2.1.0
toml==0.10.2
toolz==0.11.1
tornado==6.1
tqdm==4.61.0
traitlets==4.3.3
typed-ast==1.4.3
typing-extensions==3.10.0.0
tzlocal==2.1
urllib3==1.26.5
vininfo==1.5.0
virtualenv==20.4.7
virtualenv-clone==0.5.4
voila==0.2.10
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.0.1
widgetsnbextension==3.5.1
XlsxWriter==1.4.3
zipp==3.4.1

Apologies if I should start a separate thread!