tornado.curl_httpclient.CurlError: HTTP 599: gnutls_handshake() failed: The TLS connection was non-properly terminated

JupyterHub
,
1

Hi everybody!
Trying to launch jupyterhub, I got the following log file

[D 2020-05-17 13:25:53.654 JupyterHub application:556] Looking for /srv/jupyterhub/jupyterhub_config in /srv/jupyterhub
[D 2020-05-17 13:25:53.657 JupyterHub application:578] Loaded config file: /srv/jupyterhub/jupyterhub_config.py
[I 2020-05-17 13:25:53.673 JupyterHub app:2295] Running JupyterHub version 1.2.0dev
[I 2020-05-17 13:25:53.673 JupyterHub app:2326] Using Authenticator: oauthenticator.gitlab.LocalGitLabOAuthenticator-0.11.0
[I 2020-05-17 13:25:53.673 JupyterHub app:2326] Using Spawner: dockerspawner.dockerspawner.DockerSpawner-0.11.1
[I 2020-05-17 13:25:53.674 JupyterHub app:2326] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.2.0dev
[I 2020-05-17 13:25:53.697 JupyterHub app:1388] Loading cookie_secret from /data/jupyterhub_cookie_secret
[D 2020-05-17 13:25:53.698 JupyterHub app:1555] Connecting to db: sqlite:////data/jupyterhub.sqlite
[D 2020-05-17 13:25:53.721 JupyterHub orm:763] database schema version found: 4dc2d5a8c53c
[D 2020-05-17 13:25:53.768 JupyterHub auth:779] No UID for user donzel
[I 2020-05-17 13:25:53.768 JupyterHub auth:781] Creating user: adduser -q --gecos '""' --home /home/public/donzel --disabled-password donzel
[D 2020-05-17 13:25:54.059 JupyterHub auth:779] No UID for user berkachyr
[I 2020-05-17 13:25:54.060 JupyterHub auth:781] Creating user: adduser -q --gecos '""' --home /home/public/berkachyr --disabled-password berkachyr
[D 2020-05-17 13:25:54.336 JupyterHub auth:779] No UID for user sharednotebooks
[I 2020-05-17 13:25:54.338 JupyterHub auth:781] Creating user: adduser -q --gecos '""' --home /home/public/sharednotebooks --disabled-password sharednotebooks
[D 2020-05-17 13:25:54.617 JupyterHub auth:779] No UID for user testuser
[I 2020-05-17 13:25:54.618 JupyterHub auth:781] Creating user: adduser -q --gecos '""' --home /home/public/testuser --disabled-password testuser
[D 2020-05-17 13:25:54.923 JupyterHub app:1959] Initializing spawners
[D 2020-05-17 13:25:54.926 JupyterHub app:2092] Loaded users:
    
[I 2020-05-17 13:25:54.926 JupyterHub app:2366] Initialized 0 spawners in 0.004 seconds
[I 2020-05-17 13:25:54.930 JupyterHub proxy:646] Starting proxy @ https://:443/
[D 2020-05-17 13:25:54.930 JupyterHub proxy:647] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '443', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://jupyterhub:8080/hub/error', '--ssl-key', '/srv/jupyterhub/secrets/jupyterhub.key', '--ssl-cert', '/srv/jupyterhub/secrets/jupyterhub.crt']
[D 2020-05-17 13:25:54.973 JupyterHub proxy:562] Writing proxy pid file: jupyterhub-proxy.pid
13:25:55.640 [ConfigProxy] info: Proxying https://*:443 to (no default)
13:25:55.645 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2020-05-17 13:25:55.950 JupyterHub proxy:682] Proxy started and appears to be up
[D 2020-05-17 13:25:55.952 JupyterHub proxy:766] Proxy: Fetching GET https://127.0.0.1:8001/api/routes
[E 2020-05-17 13:25:56.039 JupyterHub app:2773]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 2771, in launch_instance_async
        await self.start()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/app.py", line 2579, in start
        await self.proxy.get_all_routes()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 806, in get_all_routes
        resp = await self.api_request('', client=client)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/proxy.py", line 774, in api_request
        result = await client.fetch(req)
    tornado.curl_httpclient.CurlError: HTTP 599: gnutls_handshake() failed: The TLS connection was non-properly terminated.
    
[D 2020-05-17 13:25:56.042 JupyterHub application:653] Exiting application: jupyterhub
10

I will appreciate any suggestion!

2

Hi. Please could you show us you jupyterhub_config.py file, redacting any secrets?
Thanks!

3

Hello !
Here is my config file:

# Configuration file for JupyterHub

c = get_config()

import os

c.JupyterHub.admin_access = True
c.JupyterHub.base_url = '/'
c.JupyterHub.cookie_max_age_days = 14
c.JupyterHub.debug_db = False
c.JupyterHub.generate_config = False
c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.hub_connect_ip = 'jupyterhub'
c.JupyterHub.logo_file = 'logo.jpg'

c.ConfigurableHTTPProxy.api_url = 'https://127.0.0.1:8001'
c.ConfigurableHTTPProxy.auth_token = 'xxxxx'

c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
c.DockerSpawner.image = os.environ['DOCKER_NOTEBOOK_IMAGE'] 

spawn_cmd = os.environ.get('DOCKER_SPAWN_CMD', "start-singleuser.sh") 
c.DockerSpawner.extra_create_kwargs.update({ 'command': spawn_cmd }) 

network_name = os.environ['DOCKER_NETWORK_NAME']
c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.network_name = network_name
c.DockerSpawner.extra_host_config = { 'network_mode': network_name }
notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR') or '/home/jovyan/work'
c.DockerSpawner.notebook_dir = notebook_dir
c.DockerSpawner.volumes = { 'jupyterhub-user-{username}': notebook_dir, 
                                                 'jupyterhub-user-sharednotebooks': {'bind': '/opt/shared_nbs/',
                                                          'mode': 'rw' }}                                                          
c.DockerSpawner.remove_containers = True
c.DockerSpawner.debug = True
c.JupyterHub.hub_port = 8080

# TLS config
c.JupyterHub.port = 443
c.JupyterHub.ssl_key = os.environ['SSL_KEY']
c.JupyterHub.ssl_cert = os.environ['SSL_CERT']

c.JupyterHub.authenticator_class = 'oauthenticator.gitlab.LocalGitLabOAuthenticator'

c.LocalGitLabOAuthenticator.oauth_callback_url = 'https://jupyterhub.groupe-asam.ch/hub/oauth_callback'
c.LocalGitLabOAuthenticator.client_id = 'xxxx'
c.LocalGitLabOAuthenticator.client_secret = 'xxxx'

data_dir = os.environ.get('DATA_VOLUME_CONTAINER', '/data')
c.JupyterHub.db_url = os.path.join('sqlite:///', data_dir, 'jupyterhub.sqlite')
c.JupyterHub.cookie_secret_file = os.path.join(data_dir,
    'jupyterhub_cookie_secret')

c.JupyterHub.services = [
    {
        'name': 'cull-idle',
        'admin': True,
        'command': 'python /scripts/cull_idle_servers.py --timeout=3600'.split(),
    }
]
c.Spawner.args = ['labhub \
            	   --allow-root \
            	   --port=8888 \
            	   --ip=0.0.0.0 \
            	   --user="$JPY_USER" \
            	   --cookie-name=$JPY_COOKIE_NAME \
            	   --base-url=$JPY_BASE_URL \
            	   --hub-prefix=$JPY_HUB_PREFIX \
            	   --hub-api-url=$JPY_HUB_API_URL \
            	   --HubAuth.api_token=$JPY_API_TOKEN \
            	   --notebook-dir="/home/jovyan/work" \
				   --SingleUserNotebookApp.default_url=/lab']

c.Spawner.cmd = ['jupyterhub-singleuser']
c.Spawner.default_url = '/lab'

c.Authenticator.admin_users = admin = {"donzel", "berkachyr"}
c.Authenticator.username_map = {'ldonze':'donzel'}
c.Authenticator.whitelist = whitelist = {"donzel", "berkachyr", "testuser", "sharednotebooks"}

pwd = os.path.dirname(__file__)
with open(os.path.join(pwd, 'userlist')) as f:
    for line in f:
        if not line:
            continue
        parts = line.split()
        name = parts[0]
        whitelist.add(name)
        if len(parts) > 1 and parts[1] == 'admin':
            admin.add(name)
            
c.LocalAuthenticator.add_user_cmd = ['adduser', '-q', '--gecos', '""', '--home', '/home/public/USERNAME', '--disabled-password']
c.LocalAuthenticator.create_system_users = True

Thank you for your help :smile:

4

This is the issue:

The proxy has not enabled internal SSL, but the Hub will try to connect with HTTPS, which will fail. Try removing this line and it should work.

5

Hi @minrk !
Thank you very much for the hint. My hub runs again. :smiley: