Error 500 unsupported operand type

JupyterHub
, ,
1

Bug description

After logging in jupyterhub, 500 error with spawner occurs.

Expected behaviour

Jupyter should spawn container

Actual behaviour

I have an error 500 : Internal Server Error
Unhandled error starting server

How to reproduce

  1. add Dockerfile:
FROM jupyterhub/jupyterhub:2.0
RUN pip install dockerspawner
COPY jupyterhub_config.py /srv/jupyterhub
  1. add docker-compose:
    version: '2.3'

    services:
        jupyterhub:
          build: ./jupyterhub
          image: jupyterhub/jupyterhub:2.0
          ports:
            - '8000:8000'
          container_name: jupyterhub-user
          volumes:
            - /var/run/docker.sock:/var/run/docker.sock
            - ./jupyterhub/jupyterhub_config.py:/srv/jupyterhub/jupyterhub_config.py
          environment:
             DOCKER_JUPYTER_CONTAINER: jupyter-notebook
             DOCKER_JUPYTER_IMAGE: neuroml/seminar2:no_fs
             DOCKER_NETWORK_NAME: ${COMPOSE_PROJECT_NAME}_default
             HUB_IP: '127.0.0.1'
  1. docker compose build && docker compose up

Your personal set up

  • OS:
    ubuntu 18.04
  • Version(s):
    docker jupyterhub 2.0
alembic @ file:///tmp/wheelhouse/alembic-1.7.5-py3-none-any.whl
async-generator @ file:///tmp/wheelhouse/async_generator-1.10-py3-none-any.whl
attrs @ file:///tmp/wheelhouse/attrs-21.4.0-py2.py3-none-any.whl
certifi @ file:///tmp/wheelhouse/certifi-2021.10.8-py2.py3-none-any.whl
certipy @ file:///tmp/wheelhouse/certipy-0.1.3-py3-none-any.whl
cffi @ file:///tmp/wheelhouse/cffi-1.15.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
charset-normalizer @ file:///tmp/wheelhouse/charset_normalizer-2.0.10-py3-none-any.whl
cryptography @ file:///tmp/wheelhouse/cryptography-36.0.1-cp36-abi3-manylinux_2_24_x86_64.whl
docker==6.0.0
dockerspawner==12.1.0
entrypoints @ file:///tmp/wheelhouse/entrypoints-0.3-py2.py3-none-any.whl
escapism==1.0.1
greenlet @ file:///tmp/wheelhouse/greenlet-1.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
idna @ file:///tmp/wheelhouse/idna-3.3-py3-none-any.whl
importlib-metadata @ file:///tmp/wheelhouse/importlib_metadata-4.10.0-py3-none-any.whl
importlib-resources @ file:///tmp/wheelhouse/importlib_resources-5.4.0-py3-none-any.whl
Jinja2 @ file:///tmp/wheelhouse/Jinja2-3.0.3-py3-none-any.whl
jsonschema @ file:///tmp/wheelhouse/jsonschema-4.3.3-py3-none-any.whl
jupyter-telemetry @ file:///tmp/wheelhouse/jupyter_telemetry-0.1.0-py3-none-any.whl
jupyterhub @ file:///tmp/wheelhouse/jupyterhub-2.0.2-py3-none-any.whl
Mako @ file:///tmp/wheelhouse/Mako-1.1.6-py2.py3-none-any.whl
MarkupSafe @ file:///tmp/wheelhouse/MarkupSafe-2.0.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
oauthlib @ file:///tmp/wheelhouse/oauthlib-3.1.1-py2.py3-none-any.whl
packaging @ file:///tmp/wheelhouse/packaging-21.3-py3-none-any.whl
pamela @ file:///tmp/wheelhouse/pamela-1.0.0-py2.py3-none-any.whl
prometheus-client @ file:///tmp/wheelhouse/prometheus_client-0.12.0-py2.py3-none-any.whl
pycparser @ file:///tmp/wheelhouse/pycparser-2.21-py2.py3-none-any.whl
pycurl==7.43.0.2
pyOpenSSL @ file:///tmp/wheelhouse/pyOpenSSL-21.0.0-py2.py3-none-any.whl
pyparsing @ file:///tmp/wheelhouse/pyparsing-3.0.6-py3-none-any.whl
pyrsistent @ file:///tmp/wheelhouse/pyrsistent-0.18.0-cp38-cp38-manylinux1_x86_64.whl
python-dateutil @ file:///tmp/wheelhouse/python_dateutil-2.8.2-py2.py3-none-any.whl
python-json-logger @ file:///tmp/wheelhouse/python_json_logger-2.0.2-py3-none-any.whl
requests @ file:///tmp/wheelhouse/requests-2.27.1-py2.py3-none-any.whl
ruamel.yaml @ file:///tmp/wheelhouse/ruamel.yaml-0.17.20-py3-none-any.whl
ruamel.yaml.clib @ file:///tmp/wheelhouse/ruamel.yaml.clib-0.2.6-cp38-cp38-manylinux1_x86_64.whl
six @ file:///tmp/wheelhouse/six-1.16.0-py2.py3-none-any.whl
SQLAlchemy @ file:///tmp/wheelhouse/SQLAlchemy-1.4.29-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
tornado @ file:///tmp/wheelhouse/tornado-6.1-cp38-cp38-manylinux2010_x86_64.whl
traitlets @ file:///tmp/wheelhouse/traitlets-5.1.1-py3-none-any.whl
urllib3 @ file:///tmp/wheelhouse/urllib3-1.26.8-py2.py3-none-any.whl
websocket-client==1.4.1
zipp @ file:///tmp/wheelhouse/zipp-3.7.0-py3-none-any.whl

Configuration File:

import os


c.JupyterHub.spawner_class = "dockerspawner.DockerSpawner"

c.DockerSpawner.image = os.environ["DOCKER_JUPYTER_IMAGE"]
c.DockerSpawner.network_name = os.environ["DOCKER_NETWORK_NAME"]

#sc.JupyterHub.ip = 'server1'
c.JupyterHub.hub_ip = '127.0.0.1'
#c.JupyterHub.hub_ip = os.environ["HUB_IP"]

c.JupyterHub.authenticator_class = 'dummy'
c.DummyAuthenticator.password = "pass"
c.Authenticator.admin_users = {'user'}
notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR') or '/workspace'
c.DockerSpawner.notebook_dir = notebook_dir


# Mount the real user's Docker volume on the host to the notebook user's
# notebook directory in the container
# c.DockerSpawner.volumes = {
#           'jupyterhub-user-{username}': notebook_dir,
#           'jupyterhub-shared': '/workspaces/neuroml_course_data',
#           'jupyterhub-data': '/home/user/neuroml_course_data/data'
# }

c.DockerSpawner.remove = True
c.Spawner.default_url = '/notebook'

jupyterhub-user | [E 2022-09-20 09:40:58.471 JupyterHub log:189] 500 GET /hub/spawn (user@::ffff:10.16.112.1) 119.53ms
jupyterhub-user | [I 2022-09-20 09:41:01.630 JupyterHub log:189] 200 GET /hub/home (user@::ffff:10.16.112.1) 6.62ms
jupyterhub-user | [I 2022-09-20 09:41:02.556 JupyterHub roles:477] Adding role server to token: <APIToken('3ad5...', user='user', client_id='jupyterhub')>
jupyterhub-user | [I 2022-09-20 09:41:02.575 JupyterHub provider:607] Creating oauth client jupyterhub-user-user
jupyterhub-user | [I 2022-09-20 09:41:02.623 JupyterHub dockerspawner:988] Container 'jupyter-user' is gone
jupyterhub-user | [E 2022-09-20 09:41:02.628 JupyterHub user:762] Unhandled error starting user's server: unsupported operand type(s) for +: 'NoneType' and 'list'
jupyterhub-user | [I 2022-09-20 09:41:02.630 JupyterHub dockerspawner:988] Container 'jupyter-user' is gone
jupyterhub-user | [W 2022-09-20 09:41:02.630 JupyterHub dockerspawner:963] Container not found: jupyter-user
jupyterhub-user | [E 2022-09-20 09:41:02.670 JupyterHub pages:311] Error starting server user: unsupported operand type(s) for +: 'NoneType' and 'list'
jupyterhub-user |     Traceback (most recent call last):
jupyterhub-user |     None: None
jupyterhub-user |
jupyterhub-user | [W 2022-09-20 09:41:02.670 JupyterHub web:1787] 500 GET /hub/spawn/user (::ffff:10.16.112.1): Unhandled error starting server user
jupyterhub-user | [E 2022-09-20 09:41:02.671 JupyterHub log:181] {
jupyterhub-user |       "X-Forwarded-Host": "server1:8000",
jupyterhub-user |       "X-Forwarded-Proto": "http",
jupyterhub-user |       "X-Forwarded-Port": "8000",
jupyterhub-user |       "X-Forwarded-For": "::ffff:10.16.112.1",
jupyterhub-user |       "Cookie": "jupyterhub-hub-login=[secret]; grafana_session=[secret]; JSESSIONID.337955a1=[secret]; oauthenticator-state=[secret]; jupyterhub-session-id=[secret]",
jupyterhub-user |       "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
jupyterhub-user |       "Accept-Encoding": "gzip, deflate",
jupyterhub-user |       "Referer": "http://server1:8000/hub/home",
jupyterhub-user |       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
jupyterhub-user |       "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36",
jupyterhub-user |       "Upgrade-Insecure-Requests": "1",
jupyterhub-user |       "Connection": "close",
jupyterhub-user |       "Host": "server1:8000"
jupyterhub-user |     }
jupyterhub-user | [E 2022-09-20 09:41:02.671 JupyterHub log:189] 500 GET /hub/spawn/user (user@::ffff:10.16.112.1) 126.65ms
2

Hi! Can you turn on debug logging (e.g. jupyterhub --debug) and show us the full logs?

3

Hi @manics , thanks for your reply. I am not sure if i acquired full logs correctly, as I just run jupyterhub --debug in running container after docker-compose up:

root@89d150b02321:/srv/jupyterhub# jupyterhub --debug
[D 2022-09-21 18:10:14.110 JupyterHub application:731] Looking for jupyterhub_config in /srv/jupyterhub
[D 2022-09-21 18:10:14.111 JupyterHub application:753] Loaded config file: /srv/jupyterhub/jupyterhub_config.py
[I 2022-09-21 18:10:14.121 JupyterHub app:2766] Running JupyterHub version 2.0.2
[I 2022-09-21 18:10:14.121 JupyterHub app:2796] Using Authenticator: jupyterhub.auth.DummyAuthenticator-2.0.2
[I 2022-09-21 18:10:14.121 JupyterHub app:2796] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-12.1.0
[I 2022-09-21 18:10:14.121 JupyterHub app:2796] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-2.0.2
[I 2022-09-21 18:10:14.128 JupyterHub app:1606] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[D 2022-09-21 18:10:14.128 JupyterHub app:1773] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2022-09-21 18:10:14.141 JupyterHub orm:955] database schema version found: 833da8570507
[I 2022-09-21 18:10:14.178 JupyterHub proxy:496] Generating new CONFIGPROXY_AUTH_TOKEN
[D 2022-09-21 18:10:14.178 JupyterHub app:2019] Loading roles into database
[I 2022-09-21 18:10:14.187 JupyterHub app:1924] Not using allowed_users. Any authenticated user will be allowed.
[D 2022-09-21 18:10:14.191 JupyterHub app:2278] Purging expired APITokens
[D 2022-09-21 18:10:14.193 JupyterHub app:2278] Purging expired OAuthCodes
[D 2022-09-21 18:10:14.195 JupyterHub app:2111] Loading role assignments from config
[D 2022-09-21 18:10:14.204 JupyterHub app:2424] Initializing spawners
[D 2022-09-21 18:10:14.206 JupyterHub app:2555] Loaded users:

[I 2022-09-21 18:10:14.206 JupyterHub app:2835] Initialized 0 spawners in 0.002 seconds
[W 2022-09-21 18:10:14.207 JupyterHub proxy:565] Found proxy pid file: /srv/jupyterhub/jupyterhub-proxy.pid
[W 2022-09-21 18:10:14.207 JupyterHub proxy:582] Proxy still running at pid=10
[C 2022-09-21 18:10:17.211 JupyterHub app:3045] Failed to start proxy
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 3043, in start
        await self.proxy.start()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 656, in start
        self._check_previous_process()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 606, in _check_previous_process
        raise RuntimeError("Failed to stop proxy at pid=%s", pid)
    RuntimeError: ('Failed to stop proxy at pid=%s', 10)

[D 2022-09-21 18:10:17.213 JupyterHub application:835] Exiting application: jupyterhub
4

You need to pass the --debug flag to the jupyterhub process that’s started by your container, e.g. something like

    command: jupyterhub -f /srv/jupyterhub/jupyterhub_config.py --debug

in your docker-compose file. The logs should then be visible using docker-compose logs