SSHSpawner timeout when trying to spawn user

JupyterHub
, ,
1

Hi All,

I am pretty new to SSHSpawner and have encountered multiple issues when trying to spawn a user from a remote machine. Here’s my scenario: I want to launch my JupyterHub on a machine (IP: 10.0.2.8) and store all user accounts on another machine (IP: 10.0.2.9). However when JupyterHub attempts to load up the spawner, it is hitting a timeout error indicating that the server didn’t respond in time.

Below is my jupyterhub_config.py

import os, pwd
import inspect, concurrent,asyncio
from subprocess import check_call, Popen, STDOUT
from sshspawner.sshspawner import SSHSpawner

c = get_config()
c.OAuthenticator.authorize_url = "https://auth.globus.org/v2/oauth2/authorize"
c.OAuthenticator.token_url = "https://auth.globus.org/v2/oauth2/token"
c.OAuthenticator.userdata_url = "https://auth.globus.org/v2/oauth2/userinfo"
c.OAuthenticator.allow_all = True
c.JupyterHub.authenticator_class = "globus"
c.OAuthenticator.oauth_callback_url = "http://127.0.0.1:8888/hub/oauth_callback"
c.OAuthenticator.client_id = <Client_id>
c.OAuthenticator.client_secret = <Secret>
c.OAuthenticator.enable_auth_state = True
c.OAuthenticator.scope = ['openid', 'profile', 'email']
c.OAuthenticator.exclude_tokens = ['auth.globus.org']
c.OAuthenticator.logout_redirect_url = 'https://globus.org/logout'
c.OAuthenticator.revoke_tokens_on_logout = False
c.JupyterHub.admin_access = True
c.JupyterHub.base_url = '/'
c.JupyterHub.bind_url = 'http://127.0.0.1:8888'
c.JupyterHub.data_files_path = '/opt/jupyterhub/share/jupyterhub/'
c.JupyterHub.debug_proxy = True
c.JupyterHub.hub_bind_url = 'http://127.0.0.1:8087'
c.JupyterHub.hub_port = 8087
c.JupyterHub.hub_ip = '127.0.0.1'
c.JupyterHub.internal_ssl = True
c.JupyterHub.recreate_internal_certs = True
c.JupyterHub.spawner_class = 'sshspawner.sshspawner.SSHSpawner'
c.SSHSpawner.remote_hosts = ['10.0.2.9']
c.SSHSpawner.remote_port = '22'
c.SSHSpawner.ssh_command = 'ssh'
c.SSHSpawner.debug = True
c.SSHSpawner.ssh_keyfile = '/home/<username>/.ssh/id_rsa'
c.SSHSpawner.remote_port_command = '/usr/bin/python3 /home/<username of remote computer>/get_port.py -i'
c.Spawner.args = ['--debug']
c.Spawner.cmd = ['jupyterhub-singleuser']
c.Spawner.debug = True
c.Spawner.default_url = '/lab'
c.Spawner.disable_user_config = True
c.Spawner.notebook_dir = '/opt/{username}'
c.Authenticator.auto_login = True

if 'JUPYTERHUB_CRYPT_KEY' not in os.environ:
    c.CryptKeeper.keys = [ os.urandom(32) ]

On the remote machine, I have created a user account in /opt/ that matches the username used to log in to JupyterHub, and I have ensured that I am able to log in using the certificate.

Below is the captured log

[E 2024-06-16 14:59:48.569 JupyterHub log:184] {
      "X-Forwarded-Host": "127.0.0.1:8888",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "8888",
      "X-Forwarded-For": "127.0.0.1",
      "Priority": "u=1",
      "If-None-Match": "\"<some ID>\"",
      "Sec-Fetch-Site": "same-origin",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Dest": "document",
      "Upgrade-Insecure-Requests": "1",
      "Cookie": "jupyterhub-hub-login=[secret]; _xsrf=[secret]; jupyterhub-session-id=[secret]",
      "Connection": "keep-alive",
      "Sec-Gpc": "1",
      "Dnt": "1",
      "Referer": "http://127.0.0.1:8888/hub/spawn-pending/jamesleong123098",
      "Accept-Encoding": "gzip, deflate, br, zstd",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0",
      "Host": "127.0.0.1:8888"
    }
[E 2024-06-16 14:59:48.569 JupyterHub log:192] 500 GET /hub/spawn-pending/jamesleong123098 (jamesleong123098@127.0.0.1) 40.31ms
[D 2024-06-16 14:59:49.782 JupyterHub scopes:1010] Checking access to /hub/spawn/jamesleong123098 via scope servers!server=jamesleong123098/
[D 2024-06-16 14:59:49.782 JupyterHub user:262] Discarding failed spawner jamesleong123098
[D 2024-06-16 14:59:49.782 JupyterHub user:496] Creating <class 'sshspawner.sshspawner.SSHSpawner'> for jamesleong123098:
[D 2024-06-16 14:59:49.785 JupyterHub pages:216] Triggering spawn with default options for jamesleong123098
[D 2024-06-16 14:59:49.785 JupyterHub base:411] Refreshing auth for jamesleong123098
[D 2024-06-16 14:59:49.786 JupyterHub base:1095] Initiating spawn for jamesleong123098
[D 2024-06-16 14:59:49.786 JupyterHub base:1099] 0/100 concurrent spawns
[D 2024-06-16 14:59:49.786 JupyterHub base:1104] 0 active servers
[I 2024-06-16 14:59:49.827 JupyterHub provider:661] Creating oauth client jupyterhub-user-jamesleong123098
[D 2024-06-16 14:59:49.874 JupyterHub user:909] Creating internal SSL certs for jamesleong123098
[I 2024-06-16 14:59:49.875 JupyterHub spawner:1226] Creating certs for jamesleong123098: DNS:localhost;IP:127.0.0.1
[D 2024-06-16 14:59:50.100 JupyterHub user:912] Calling Spawner.start for jamesleong123098
[D 2024-06-16 14:59:50.104 JupyterHub sshspawner:229] Remote host was set to 10.0.2.9.
[I 2024-06-16 14:59:50.787 JupyterHub log:192] 302 GET /hub/spawn/jamesleong123098 -> /hub/spawn-pending/jamesleong123098 (jamesleong123098@127.0.0.1) 1014.25ms
[D 2024-06-16 14:59:50.799 JupyterHub scopes:1010] Checking access to /hub/spawn-pending/jamesleong123098 via scope servers!server=jamesleong123098/
[I 2024-06-16 14:59:50.800 JupyterHub pages:397] jamesleong123098 is pending spawn
[I 2024-06-16 14:59:50.802 JupyterHub log:192] 200 GET /hub/spawn-pending/jamesleong123098 (jamesleong123098@127.0.0.1) 10.72ms
[D 2024-06-16 14:59:50.813 JupyterHub sshspawner:257] ip=10.0.2.9 port=39613
[D 2024-06-16 14:59:50.813 JupyterHub sshspawner:233] Remote IP was set to 10.0.2.9.
[D 2024-06-16 14:59:50.874 JupyterHub scopes:1010] Checking access to /hub/api/users/jamesleong123098/server/progress via scope read:servers!server=jamesleong123098/
[D 2024-06-16 14:59:51.823 JupyterHub sshspawner:298] /tmp/jamesleong123098_run.sh was written as:
   #!/bin/bash
    export PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
    export LANG=en_US.UTF-8
    export JUPYTERHUB_API_TOKEN=<Token>
    export JPY_API_TOKEN=<API Token>
    export JUPYTERHUB_ADMIN_ACCESS=1
    export JUPYTERHUB_CLIENT_ID=jupyterhub-user-jamesleong123098
    export JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED=0
    export JUPYTERHUB_HOST=
    export JUPYTERHUB_OAUTH_CALLBACK_URL=/user/jamesleong123098/oauth_callback
    export JUPYTERHUB_OAUTH_SCOPES=["access:servers!server=jamesleong123098/", "access:servers!user=jamesleong123098"]
    export JUPYTERHUB_OAUTH_ACCESS_SCOPES=["access:servers!server=jamesleong123098/", "access:servers!user=jamesleong123098"]
    export JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES=[]
    export JUPYTERHUB_USER=jamesleong123098
    export JUPYTERHUB_SERVER_NAME=
    export JUPYTERHUB_API_URL=
    export JUPYTERHUB_ACTIVITY_URL=https://127.0.0.1:8087/hub/api/users/jamesleong123098/activity
    export JUPYTERHUB_BASE_URL=/
    export JUPYTERHUB_SERVICE_PREFIX=/user/jamesleong123098/
    export JUPYTERHUB_SERVICE_URL=https://127.0.0.1:0/user/jamesleong123098/
    export JUPYTERHUB_PUBLIC_URL=
    export JUPYTERHUB_PUBLIC_HUB_URL=
    export JUPYTERHUB_SSL_KEYFILE=.jupyterhub-resources/user-jamesleong123098.key
    export JUPYTERHUB_SSL_CERTFILE=.jupyterhub-resources/user-jamesleong123098.crt
    export JUPYTERHUB_SSL_CLIENT_CA=.jupyterhub-resources/notebooks-ca_trust.crt
    export JUPYTERHUB_ROOT_DIR=/opt/jamesleong123098
    export JUPYTERHUB_DEFAULT_URL=/lab
    export JUPYTERHUB_DEBUG=1
    export JUPYTERHUB_DISABLE_USER_CONFIG=1
    export GLOBUS_LOCAL_ENDPOINT=
    export GLOBUS_DATA=<Some super long data>
    unset XDG_RUNTIME_DIR
    touch .jupyter.log
    chmod 600 .jupyter.log
    jupyterhub-singleuser --debug < /dev/null >> .jupyter.log 2>&1 & pid=$!
    echo $pid
[D 2024-06-16 14:59:52.276 JupyterHub sshspawner:306] exec_notebook status=0
[D 2024-06-16 14:59:52.277 JupyterHub sshspawner:183] Starting User: jamesleong123098, PID: 1829
[D 2024-06-16 14:59:52.293 JupyterHub spawner:1432] Polling subprocess every 30s
[D 2024-06-16 14:59:52.303 JupyterHub utils:292] Waiting 30s for server at https://10.0.2.9:39613/user/jamesleong123098/api
[D 2024-06-16 15:00:00.203 JupyterHub sshspawner:330] command: kill -s 0 1829 < /dev/null returned  --- bash: line 0: kill: (1829) - No such process
     --- 1
[D 2024-06-16 15:00:00.203 JupyterHub sshspawner:203] Polling returned False
[D 2024-06-16 15:00:00.203 JupyterHub sshspawner:233] Remote IP was set to remote_ip.
[W 2024-06-16 15:00:23.085 JupyterHub user:1050] jamesleong123098's server never showed up at https://10.0.2.9:39613/user/jamesleong123098/ after 30 seconds. Giving up.
[D 2024-06-16 15:00:23.086 JupyterHub user:1095] Stopping jamesleong123098
[D 2024-06-16 15:00:23.095 JupyterHub user:1117] Deleting oauth client jupyterhub-user-jamesleong123098
[D 2024-06-16 15:00:23.118 JupyterHub user:1120] Finished stopping jamesleong123098
[E 2024-06-16 15:00:23.140 JupyterHub gen:630] Exception in Future <Task finished name='Task-27692' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /opt/jupyterhub/lib/python3.8/site-packages/jupyterhub/handlers/base.py:1115> exception=TimeoutError("Server at https://10.0.2.9:39613/user/jamesleong123098/api didn't respond in 30 seconds")> after timeout
    Traceback (most recent call last):
      File "/opt/jupyterhub/lib/python3.8/site-packages/tornado/gen.py", line 625, in error_callback
        future.result()
      File "/opt/jupyterhub/lib/python3.8/site-packages/jupyterhub/handlers/base.py", line 1122, in finish_user_spawn
        await spawn_future
      File "/opt/jupyterhub/lib/python3.8/site-packages/jupyterhub/user.py", line 1028, in spawn
        await self._wait_up(spawner)
      File "/opt/jupyterhub/lib/python3.8/site-packages/jupyterhub/user.py", line 1071, in _wait_up
        raise e
      File "/opt/jupyterhub/lib/python3.8/site-packages/jupyterhub/user.py", line 1042, in _wait_up
        resp = await server.wait_up(
      File "/opt/jupyterhub/lib/python3.8/site-packages/jupyterhub/utils.py", line 322, in wait_for_http_server
        re = await exponential_backoff(
      File "/opt/jupyterhub/lib/python3.8/site-packages/jupyterhub/utils.py", line 265, in exponential_backoff
        raise asyncio.TimeoutError(fail_message)
    asyncio.exceptions.TimeoutError: Server at https://10.0.2.9:39613/user/jamesleong123098/api didn't respond in 30 seconds

Below is a sample of where it got stuck

image
image1430×780 48.4 KB

From this point onward, I am not entirely sure what is happening with the spawner. Any help or direction is greatly appreciated.

Best,
James

2

This line suggests that jupyterhub-singleuser command failed on the remote server. As you are redirecting the single user server logs to .jupyter.log file, please check the logs and post them here.

1 Like
3

As far as I know, that’s the only log file I have. Is there any other log file I should be aware of? Is there any parameter that I have to set to true in order to capture the log for a single-user server log? On my remote side, I don’t think there’s a log file specifically for capturing log.

4

The script is written to /tmp folder in the remote machine and script has the following line

jupyterhub-singleuser --debug < /dev/null >> .jupyter.log 2>&1

So, I imagine you will have single user server logs in /tmp/.jupyter.log. That is file you need to look at.

1 Like
5

I have checked my remote side, and I couldn’t find any file created there. However, I am seeing a jamesleong123098_run.sh file getting generated on my 10.0.2.8 machine instead of the remote machine. My guess is that the script isn’t executed yet before it killed the process. Apart from that, I saw a singleuser.log file generated on both my 10.0.2.8 machine and remote 10.0.2.9 machine.

My singleuser.log file


[I 2024-06-17 13:19:20.665 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-06-17 13:19:20.665 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 5.0.0
[E 2024-06-17 13:19:20.665 JupyterHubSingleUser] Failed to load JupyterHubSingleUser server extension
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/singleuser/extension.py", line 284, in wrapped
        r = f(self, *args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/singleuser/extension.py", line 491, in load_config_file
        raise KeyError("Missing required environment $JUPYTERHUB_SERVICE_URL")
    KeyError: 'Missing required environment $JUPYTERHUB_SERVICE_URL'

In my jupyterhub_config.py file, I have specify already the service url, but not sure why it is still showing me the error. My concern is that, do I also need to setup another JupyterHub on the remote side as well?

6

This line indicates that the command jupyterhub-singleuser --debug < /dev/null >> .jupyter.log 2>&1 & pid=$! has been executed on remote machine and it returned the PID. Now looking again into the generated script, Hub API URL is set to empty: export JUPYTERHUB_API_URL=. Probably this is the reason that single user server is not able to reach Hub.

Seems like SSHSpawner is not updated for a while and that is why you are facing these issues.

My concern is that, do I also need to setup another JupyterHub on the remote side as well?

No, you dont need to although JupyterHub must be installed on the remote server as jupyterhub-singleuser command is provided by JupyterHub package.

7

After giving the jupyterhub_api_url, it is still unable to reach the hub. Carefully going through the log file again, I noticed that I got this error message before the timeout error: “Failed to retrieve encrypted auth_state.” I am not sure if this causes an issue with spawning the user thread. Besides, the jamesleong123098_run.sh bash script was generated on my 10.0.2.8 machine instead of the remote 10.0.2.9 machine. I have tried to change multiple configurations but no luck.

8

This is another issue. This should be IP of remote machine and port of single user server. Unforturnately SSHSpawner must be updated to make it work with latest JupyterHub. Could you try the following config?

import os, pwd
import inspect, concurrent,asyncio
from subprocess import check_call, Popen, STDOUT
from sshspawner.sshspawner import SSHSpawner

c = get_config()
c.OAuthenticator.authorize_url = "https://auth.globus.org/v2/oauth2/authorize"
c.OAuthenticator.token_url = "https://auth.globus.org/v2/oauth2/token"
c.OAuthenticator.userdata_url = "https://auth.globus.org/v2/oauth2/userinfo"
c.OAuthenticator.allow_all = True
c.JupyterHub.authenticator_class = "globus"
c.OAuthenticator.oauth_callback_url = "http://127.0.0.1:8888/hub/oauth_callback"
c.OAuthenticator.client_id = <Client_id>
c.OAuthenticator.client_secret = <Secret>
c.OAuthenticator.enable_auth_state = True
c.OAuthenticator.scope = ['openid', 'profile', 'email']
c.OAuthenticator.exclude_tokens = ['auth.globus.org']
c.OAuthenticator.logout_redirect_url = 'https://globus.org/logout'
c.OAuthenticator.revoke_tokens_on_logout = False
c.JupyterHub.admin_access = True
c.JupyterHub.base_url = '/'
c.JupyterHub.data_files_path = '/opt/jupyterhub/share/jupyterhub/'
c.JupyterHub.debug_proxy = True
c.JupyterHub.hub_port = 8087
c.JupyterHub.hub_ip = '10.0.2.8'  # Note that this should be IP that is reachable by your remote machine
c.JupyterHub.internal_ssl = True
c.JupyterHub.recreate_internal_certs = True

class CustomSSHSpawner(SSHSpawner):
    async def start(self):
        username = self.user.name
        kf = self.ssh_keyfile.format(username=username)
        cf = kf + "-cert.pub"
        k = asyncssh.read_private_key(kf)
        c = asyncssh.read_certificate(cf)

        self.remote_host = self.choose_remote_host()
        
        self.remote_ip, port = await self.remote_random_port()
        if self.remote_ip is None or port is None or port == 0:
            return False
        self.remote_port = str(port)
        self.ip = self.remote_ip
        self.port = self.remote_port
        cmd = []

        cmd.extend(self.cmd)
        cmd.extend(self.get_args())

        if self.user.settings["internal_ssl"]:
            with TemporaryDirectory() as td:
                local_resource_path = td

                self.cert_paths = self.stage_certs(
                        self.cert_paths,
                        local_resource_path
                    )

                # create resource path dir in user's home on remote
                async with asyncssh.connect(self.remote_ip, username=username,client_keys=[(k,c)],known_hosts=None) as conn:
                    mkdir_cmd = "mkdir -p {path} 2>/dev/null".format(path=self.resource_path)
                    result = await conn.run(mkdir_cmd)

                # copy files
                files = [os.path.join(local_resource_path, f) for f in os.listdir(local_resource_path)]
                async with asyncssh.connect(self.remote_ip, username=username,client_keys=[(k,c)],known_hosts=None) as conn:
                    await asyncssh.scp(files, (conn, self.resource_path))

        remote_cmd = ' '.join(cmd)

        self.pid = await self.exec_notebook(remote_cmd)

        self.log.debug("Starting User: {}, PID: {}".format(self.user.name, self.pid))

        if self.pid < 0:
            return None
        self.log.info("Single user server for user %s will be launched on host %s and port %d", % (self.user.name, self.ip, self.port))
        return (self.ip, self.port)

c.JupyterHub.spawner_class = CustomSSHSpawner
c.CustomSSHSpawner.remote_hosts = ['10.0.2.9']
c.CustomSSHSpawner.remote_port = '22'
c.CustomSSHSpawner.ssh_command = 'ssh'
c.CustomSSHSpawner.hub_api_url = 'http://10.0.2.8:8087/hub/api'
c.CustomSSHSpawner.debug = True
c.CustomSSHSpawner.ssh_keyfile = '/home/<username>/.ssh/id_rsa'
c.CustomSSHSpawner.remote_port_command = '/usr/bin/python3 /home/<username of remote computer>/get_port.py -i'
c.Spawner.args = ['--debug']
c.Spawner.cmd = ['jupyterhub-singleuser']
c.Spawner.debug = True
c.Spawner.default_url = '/lab'
c.Spawner.disable_user_config = True
c.Spawner.notebook_dir = '/opt/{username}'
c.Authenticator.auto_login = True

if 'JUPYTERHUB_CRYPT_KEY' not in os.environ:
    c.CryptKeeper.keys = [ os.urandom(32) ]

Note that I have changed the hub_ip to an IP accessible from remote machine and removed bind_url configs. CustomSSHSpawner attempts to fix the issues of the upstream SSHSpawner.

9

I have implemented the above configuration, and it is still hitting the timeout error. However, I am able to see a .jupyter.log file existed on the remote side within the user account instead of the path /tmp/. When I open up the file, I am seeing this error “bash: line 34: jupyterhub-singleuser: command not found.”

This is what my updated /tmp/jamesleong123098_run.sh script looks like

    #!/bin/bash
    export PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
    export LANG=en_US.UTF-8
    export JUPYTERHUB_API_TOKEN=<Token>
    export JPY_API_TOKEN=<Token API>
    export JUPYTERHUB_ADMIN_ACCESS=1
    export JUPYTERHUB_CLIENT_ID=jupyterhub-user-jamesleong123098
    export JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED=0
    export JUPYTERHUB_HOST=
    export JUPYTERHUB_OAUTH_CALLBACK_URL=/user/jamesleong123098/oauth_callback
    export JUPYTERHUB_OAUTH_SCOPES=["access:servers!server=jamesleong123098/", "access:servers!user=jamesleong123098"]
    export JUPYTERHUB_OAUTH_ACCESS_SCOPES=["access:servers!server=jamesleong123098/", "access:servers!user=jamesleong123098"]
    export JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES=[]
    export JUPYTERHUB_USER=jamesleong123098
    export JUPYTERHUB_SERVER_NAME=
    export JUPYTERHUB_API_URL=http://10.0.2.8:8087/hub/api
    export JUPYTERHUB_ACTIVITY_URL=https://127.0.0.1:8087/hub/api/users/vincet_iastate_edu/activity
    export JUPYTERHUB_BASE_URL=/
    export JUPYTERHUB_SERVICE_PREFIX=/user/jamesleong123098/
    export JUPYTERHUB_SERVICE_URL=https://10.0.2.9:41859/user/jamesleong123098/
    export JUPYTERHUB_PUBLIC_URL=
    export JUPYTERHUB_PUBLIC_HUB_URL=
    export JUPYTERHUB_SSL_KEYFILE=.jupyterhub-resources/user-jamesleong123098.key
    export JUPYTERHUB_SSL_CERTFILE=.jupyterhub-resources/user-jamesleong123098.crt
    export JUPYTERHUB_SSL_CLIENT_CA=.jupyterhub-resources/notebooks-ca_trust.crt
    export JUPYTERHUB_ROOT_DIR=/opt/jamesleong123098
    export JUPYTERHUB_DEFAULT_URL=/lab
    export JUPYTERHUB_DEBUG=1
    export JUPYTERHUB_DISABLE_USER_CONFIG=1
    export GLOBUS_LOCAL_ENDPOINT=
    unset XDG_RUNTIME_DIR
    touch .jupyter.log
    chmod 600 .jupyter.log
    jupyterhub-singleuser --debug < /dev/null >> .jupyter.log 2>&1 & pid=$!
    echo $pid

On my remote machine, I am having the following files in /usr/local/bin folder
image

10

That is the issue. jupyterhub-singleuser command is provided by JupyterHub and so you need to install jupyterhub on the remote machine as well. I see in your /usr/local/bin, there is no jupyter-lab neither, so you will need to install it as well.

11

Now that you have mentioned that, I have installed both JupyterHub and JupyterLab. Below is what my remote machine look like in /usr/local/bin folder.
image
After I installed those libraries, I am getting some error messages in .jupyter.log from my remote user account. Do I have to setup my jupyterhub_config.py file on the remote side as well?

Below is the log I got from .jupyter.log from my remote user account

[D 2024-06-18 13:36:30.558 ServerApp] Checking for /opt/jamesleong123098/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-06-18 13:36:30.558 ServerApp] Checking for /usr/local/share/jupyter/lab/staging/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-06-18 13:36:30.558 ServerApp] Checking for /usr/lib/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-06-18 13:36:30.558 ServerApp] Checking for /usr/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-06-18 13:36:30.558 ServerApp] Checking for /usr/local/lib/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-06-18 13:36:30.558 ServerApp] Checking for /usr/local/node_modules/yaml-language-server/bin/yaml-language-server
[D 2024-06-18 13:36:30.558 ServerApp] yaml-language-server/bin/yaml-language-server not found in node_modules of [PosixPath('/opt/jamesleong123098'), PosixPath('/usr/local/share/jupyter/lab/staging'), PosixPath('/usr/lib'), PosixPath('/usr'), PosixPath('/usr/local/lib'), PosixPath('/usr/local')]
[D 2024-06-18 13:36:30.590 ServerApp] [lsp] None of the installed servers require virtual documents disabling shadow filesystem.
[D 2024-06-18 13:36:30.590 ServerApp] [lsp] The following Language Servers will be available: {}
[E 2024-06-18 13:36:31.626 JupyterHubSingleUser] Failed to connect to my Hub at http://10.0.2.8:8087/hub/api (attempt 2/5). Is it running?
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/singleuser/extension.py", line 346, in check_hub_version
        resp = await client.fetch(self.hub_auth.api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2024-06-18 13:36:35.633 JupyterHubSingleUser] Failed to connect to my Hub at http://10.0.2.8:8087/hub/api (attempt 3/5). Is it running?
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/singleuser/extension.py", line 346, in check_hub_version
        resp = await client.fetch(self.hub_auth.api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2024-06-18 13:36:43.641 JupyterHubSingleUser] Failed to connect to my Hub at http://10.0.2.8:8087/hub/api (attempt 4/5). Is it running?
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/singleuser/extension.py", line 346, in check_hub_version
        resp = await client.fetch(self.hub_auth.api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[E 2024-06-18 13:36:59.656 JupyterHubSingleUser] Failed to connect to my Hub at http://10.0.2.8:8087/hub/api (attempt 5/5). Is it running?
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/singleuser/extension.py", line 346, in check_hub_version
        resp = await client.fetch(self.hub_auth.api_url)
    ConnectionRefusedError: [Errno 111] Connection refused
[C 2024-06-18 13:37:00.193 ServerApp] received signal 15, stopping
[I 2024-06-18 13:37:00.194 ServerApp] Shutting down 5 extensions
[D 2024-06-18 13:37:00.195 ServerApp] jupyter_server_terminals | extension app 'jupyter_server_terminals' stopping
[D 2024-06-18 13:37:00.195 ServerApp] jupyter_server_terminals | extension app 'jupyter_server_terminals' stopped
[D 2024-06-18 13:37:00.196 ServerApp] jupyterhub | extension app 'jupyterhub-singleuser' stopping
[D 2024-06-18 13:37:00.196 ServerApp] jupyterhub | extension app 'jupyterhub-singleuser' stopped
[D 2024-06-18 13:37:00.196 ServerApp] jupyterlab | extension app 'lab' stopping
[D 2024-06-18 13:37:00.196 ServerApp] jupyterlab | extension app 'lab' stopped
[E 2024-06-18 13:37:00.197 JupyterHubSingleUser] Failed to load JupyterHubSingleUser server extension
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/singleuser/extension.py", line 284, in wrapped
        r = f(self, *args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/singleuser/extension.py", line 655, in initialize
        app.io_loop.run_sync(self.check_hub_version)
      File "/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py", line 538, in run_sync
        raise TimeoutError("Operation timed out after %s seconds" % timeout)
    asyncio.exceptions.TimeoutError: Operation timed out after None seconds
[D 2024-06-18 13:37:00.207 JupyterHubSingleUser] Exiting application: jupyterhub-singleuser
12

As you turned on internal SSL, I guess Hub API URL is accessible at https. Could you try with c.CustomSSHSpawner.hub_api_url = 'https://10.0.2.8:8087/hub/api'?

13

Once I did that, it seems to login without error. However, it is getting stuck in a blank page with nothing is showing up.

Below is the log I got from .jupyter.log from my remote user account

[E 2024-06-19 18:45:37.811 ServerApp] Uncaught exception GET /user/jamesleong123098/lab?redirects=4 (::ffff:127.0.0.1)
    HTTPServerRequest(protocol='http', host='127.0.0.1:8888', method='GET', uri='/user/jamesleong123098/lab?redirects=4', version='HTTP/1.1', remote_ip='::ffff:127.0.0.1')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1769, in _execute
        result = await result  # type: ignore
      File "/usr/local/lib/python3.8/dist-packages/jupyter_server/base/handlers.py", line 623, in prepare
        _user = await _user
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/singleuser/extension.py", line 176, in get_user
        f"Checking user {user['name']} with scopes {user['scopes']} against {self.hub_auth.access_scopes}"
      File "/usr/local/lib/python3.8/dist-packages/traitlets/traitlets.py", line 687, in __get__
        return t.cast(G, self.get(obj, cls))  # the G should encode the Optional
      File "/usr/local/lib/python3.8/dist-packages/traitlets/traitlets.py", line 635, in get
        default = obj.trait_defaults(self.name)
      File "/usr/local/lib/python3.8/dist-packages/traitlets/traitlets.py", line 1897, in trait_defaults
        return t.cast(Sentinel, self._get_trait_default_generator(names[0])(self))
      File "/usr/local/lib/python3.8/dist-packages/traitlets/traitlets.py", line 1241, in __call__
        return self.func(*args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/services/auth.py", line 518, in _default_scopes
        return set(json.loads(env_scopes))
      File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
        return _default_decoder.decode(s)
      File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)
[E 2024-06-19 18:45:37.812 ServerApp] Uncaught exception in write_error
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1769, in _execute
        result = await result  # type: ignore
      File "/usr/local/lib/python3.8/dist-packages/jupyter_server/base/handlers.py", line 623, in prepare
        _user = await _user
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/singleuser/extension.py", line 176, in get_user
        f"Checking user {user['name']} with scopes {user['scopes']} against {self.hub_auth.access_scopes}"
      File "/usr/local/lib/python3.8/dist-packages/traitlets/traitlets.py", line 687, in __get__
        return t.cast(G, self.get(obj, cls))  # the G should encode the Optional
      File "/usr/local/lib/python3.8/dist-packages/traitlets/traitlets.py", line 635, in get
        default = obj.trait_defaults(self.name)
      File "/usr/local/lib/python3.8/dist-packages/traitlets/traitlets.py", line 1897, in trait_defaults
        return t.cast(Sentinel, self._get_trait_default_generator(names[0])(self))
      File "/usr/local/lib/python3.8/dist-packages/traitlets/traitlets.py", line 1241, in __call__
        return self.func(*args, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/services/auth.py", line 518, in _default_scopes
        return set(json.loads(env_scopes))
      File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
        return _default_decoder.decode(s)
      File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1298, in send_error
        self.write_error(status_code, **kwargs)
      File "/usr/local/lib/python3.8/dist-packages/jupyter_server/base/handlers.py", line 739, in write_error
        html = self.render_template("%s.html" % status_code, **ns)
      File "/usr/local/lib/python3.8/dist-packages/jupyter_server/base/handlers.py", line 665, in render_template
        ns.update(self.template_namespace)
      File "/usr/local/lib/python3.8/dist-packages/jupyter_server/base/handlers.py", line 675, in template_namespace
        logged_in=self.logged_in,
      File "/usr/local/lib/python3.8/dist-packages/jupyter_server/base/handlers.py", line 198, in logged_in
        user = self.current_user
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1424, in current_user
        self._current_user = self.get_current_user()
      File "/usr/local/lib/python3.8/dist-packages/jupyter_server/base/handlers.py", line 177, in get_current_user
        raise RuntimeError(msg)
    RuntimeError: Calling `LabHandler.get_current_user()` directly is deprecated in jupyter-server 2.0. Use `self.current_user` instead (works in all versions).
[E 2024-06-19 18:45:37.814 ServerApp] {
      "X-Forwarded-Host": "127.0.0.1:8888",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "8888",
      "X-Forwarded-For": "::ffff:127.0.0.1",
      "Priority": "u=1",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Site": "same-origin",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Dest": "document",
      "Upgrade-Insecure-Requests": "1",
      "Cookie": "_xsrf=[secret]; jupyterhub-user-jamesleong123098=[secret]; jupyterhub-session-id=[secret]",
      "Connection": "keep-alive",
      "Sec-Gpc": "1",
      "Dnt": "1",
      "Referer": "http://127.0.0.1:8888/hub/user/jamesleong123098/lab?redirects=3",
      "Accept-Encoding": "gzip, deflate, br, zstd",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0",
      "Host": "127.0.0.1:8888"
    }
[E 2024-06-19 18:45:37.814 ServerApp] 500 GET /user/jamesleong123098/lab?redirects=4 (@::ffff:127.0.0.1) 3.89ms

I have also printed out the value of “env_scopes” from “/usr/local/lib/python3.8/dist-packages/jupyterhub/services/auth.py.” The value “[access:servers!server=jamesleong123098/” is a string and this value is being parse into json.loads like this “set(json.loads(env_scopes))” as shown from the error message above. Is there a place where I can specify the env_scopes to be a json object instead of a string. Apart from that, the jamesleong123098_run.sh bash file is not generated on the remote machine (10.0.2.9) instead, it is generated on the (10.0.2.8) machine.

14

I guess this is due to the following exported env vars in the bash script:

export JUPYTERHUB_OAUTH_SCOPES=["access:servers!server=jamesleong123098/", "access:servers!user=jamesleong123098"]
export JUPYTERHUB_OAUTH_ACCESS_SCOPES=["access:servers!server=jamesleong123098/", "access:servers!user=jamesleong123098"]
export JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES=[]

They must be quoted when exporting. You will have to overload exec_notebook method to quote them before exporting in the bash script.

1 Like
15

Thanks for the direction, after changing the line of “bash_script_str += ‘export %s=%s\n’ % item” from the exec_notebook function to “bash_script_str += “export %s=‘%s’\n” % item,” it is working now. My last question would be, how can I create an account that doesn’t exist on my remote side. Is there a parameter for me to add some code to create user on the remote side. I really appreciate all your help and support.