#I want to start a jupyterhub-singleuser on a remote server to connect to a jupyterhub running on another machine. I used sshspawner from github, but when jupyterhub-singleuser cannot communicate with the hub, can someone help me?Even if I pay a certain amount of remuneration
The startup configuration file of jupyterhub is as follows:**
##I have two machines, where the hub running is 172.16.160.106, and the ip running jupyterhub-singleuser is 172.16.160.105
jupyterhub_config.py
c.JupyterHub.ip = '172.16.160.106'
c.JupyterHub.hub_connect_ip = '172.16.160.106'
c.JupyterHub.hub_ip = '172.16.160.106'
c.JupyterHub.port = 8898
c.JupyterHub.init_spawners_timeout = -1
c.JupyterHub.authenticator_class = 'ldapauthenticator.LDAPAuthenticator'
c.LDAPAuthenticator.server_address = 'ldap://172.16.150.27'
c.LDAPAuthenticator.lookup_dn = True
c.LDAPAuthenticator.lookup_dn_search_filter = '({login_attr}={login})'
c.LDAPAuthenticator.lookup_dn_search_user = 'cn=Manager,dc=exp,dc=yoyosys,dc=com'
c.LDAPAuthenticator.lookup_dn_search_password = 'bdap1234'
c.LDAPAuthenticator.user_search_base = 'dc=exp,dc=yoyosys,dc=com'
c.LDAPAuthenticator.user_attribute = 'uid'
c.LDAPAuthenticator.lookup_dn_user_dn_attribute = 'cn'
c.SSHSpawner.debug =True
c.SSHSpawner.ssh_command = 'ssh'
c.JupyterHub.spawner_class = 'sshspawner.sshspawner.SSHSpawner'
c.SSHSpawner.slow_spawn_timeout='50'
c.SSHSpawner.remote_hosts = ['bogon']
c.SSHSpawner.remote_ip ='bogon'
c.SSHSpawner.remote_host = 'bogon'
c.SSHSpawner.remote_port = '22'
c.SSHSpawner.path = '/usr/bin:/usr/local/anaconda3/bin/:/bin/:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/aws/bin'
c.SSHSpawner.remote_port_command = '/usr/local/anaconda3/bin/python /data/sshspawner-master/scripts/get_port.py'
c.JupyterHub.BaseHandler.debug =True
c.Spawner.mem_limit = '2 G'
c.Spawner.cpu_limit = 1
c.Spawner.mem_guarantee = '200 M'
##This is the startup jupyterhub-singleuser script of sshspawner that I printed out. This script should be executed on the remote 105 node
#!/bin/bash
export PATH=/usr/bin:/usr/local/anaconda3/bin/:/bin/:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/aws/bin
export CONDA_DEFAULT_ENV=base
export LANG=en_US.UTF-8
export JUPYTERHUB_API_TOKEN=8b67171e21794e418ed819b417615593
export JPY_API_TOKEN=8b67171e21794e418ed819b417615593
export JUPYTERHUB_CLIENT_ID=jupyterhub-user-app1
export JUPYTERHUB_HOST=172.16.160.106
export JUPYTERHUB_OAUTH_CALLBACK_URL=/user/app1/oauth_callback
export JUPYTERHUB_USER=app1
export JUPYTERHUB_SERVER_NAME=
export JUPYTERHUB_API_URL=http://172.16.160.106:8081/hub/api
export JUPYTERHUB_ACTIVITY_URL=http://172.16.160.106:8081/hub/api/users/app1/activity
export JUPYTERHUB_BASE_URL=/
export JUPYTERHUB_SERVICE_PREFIX=/user/app1/
export MEM_LIMIT=2147483648
export MEM_GUARANTEE=209715200
export CPU_LIMIT=1.0
unset XDG_RUNTIME_DIR
touch jupyter.log
chmod 600 jupyter.log
jupyterhub-singleuser --port=34055 --debug >> jupyter.log 2>&1 & pid=$!
echo $PWD > /tmp/pwd.txt
echo $pid
##This is the log printed by hub
(base) [root@bogon2 local]# jupyterhub -f /data/jupyterhub_config.py
[I 2021-12-31 16:12:24.370 JupyterHub app:2240] Running JupyterHub version 1.1.0
[I 2021-12-31 16:12:24.374 JupyterHub app:2271] Using Authenticator: ldapauthenticator.ldapauthenticator.LDAPAuthenticator-1.3.0
[I 2021-12-31 16:12:24.374 JupyterHub app:2271] Using Spawner: sshspawner.sshspawner.SSHSpawner
[I 2021-12-31 16:12:24.374 JupyterHub app:2271] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.1.0
[I 2021-12-31 16:12:24.380 JupyterHub app:1394] Writing cookie_secret to /usr/local/jupyterhub_cookie_secret
[I 2021-12-31 16:12:24.424 alembic.runtime.migration migration:155] Context impl SQLiteImpl.
[I 2021-12-31 16:12:24.424 alembic.runtime.migration migration:162] Will assume non-transactional DDL.
[I 2021-12-31 16:12:24.444 alembic.runtime.migration migration:515] Running stamp_revision -> 4dc2d5a8c53c
[I 2021-12-31 16:12:24.524 JupyterHub proxy:461] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2021-12-31 16:12:24.527 JupyterHub app:1624] No admin users, admin interface will be unavailable.
[W 2021-12-31 16:12:24.527 JupyterHub app:1626] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-12-31 16:12:24.527 JupyterHub app:1655] Not using whitelist. Any authenticated user will be allowed.
86400
[I 2021-12-31 16:12:24.594 JupyterHub app:2311] Initialized 0 spawners in 0.002 seconds
[W 2021-12-31 16:12:24.597 JupyterHub proxy:643] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else...
[I 2021-12-31 16:12:24.598 JupyterHub proxy:646] Starting proxy @ http://172.16.160.106:8898/
16:12:25.306 [ConfigProxy] info: Proxying http://172.16.160.106:8898 to (no default)
16:12:25.310 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
16:12:25.508 [ConfigProxy] info: 200 GET /api/routes
[I 2021-12-31 16:12:25.509 JupyterHub app:2556] Hub API listening on http://172.16.160.106:8081/hub/
16:12:25.511 [ConfigProxy] info: 200 GET /api/routes
[I 2021-12-31 16:12:25.511 JupyterHub proxy:320] Checking routes
[I 2021-12-31 16:12:25.511 JupyterHub proxy:400] Adding default route for Hub: / => http://172.16.160.106:8081
16:12:25.513 [ConfigProxy] info: Adding route / -> http://172.16.160.106:8081
16:12:25.514 [ConfigProxy] info: Route added / -> http://172.16.160.106:8081
16:12:25.514 [ConfigProxy] info: 201 POST /api/routes/
[I 2021-12-31 16:12:25.515 JupyterHub app:2631] JupyterHub is now running at http://172.16.160.106:8898/
[W 2021-12-31 16:12:33.089 JupyterHub base:392] Invalid or expired cookie token
[I 2021-12-31 16:12:33.091 JupyterHub log:174] 302 GET /hub/home -> /hub/login?next=%2Fhub%2Fhome (@172.16.7.221) 2.90ms
[I 2021-12-31 16:12:33.138 JupyterHub log:174] 200 GET /hub/login?next=%2Fhub%2Fhome (@172.16.7.221) 37.39ms
[I 2021-12-31 16:12:38.577 JupyterHub base:707] User logged in: app1
[I 2021-12-31 16:12:38.578 JupyterHub log:174] 302 POST /hub/login?next=%2Fhub%2Fhome -> /hub/home (app1@172.16.7.221) 140.99ms
[W 2021-12-31 16:12:38.599 JupyterHub configurable:168] Config option `slow_spawn_timeout` not recognized by `SSHSpawner`. Did you mean one of: `http_time out, start_timeout`?
[W 2021-12-31 16:12:38.600 JupyterHub configurable:168] Config option `remote_ip` not recognized by `SSHSpawner`. Did you mean one of: `remote_hosts, remo te_port`?
[W 2021-12-31 16:12:38.601 JupyterHub configurable:168] Config option `remote_host` not recognized by `SSHSpawner`. Did you mean one of: `remote_hosts, re mote_port, remote_port_command`?
[I 2021-12-31 16:12:38.625 JupyterHub log:174] 200 GET /hub/home (app1@172.16.7.221) 37.41ms
[E 2021-12-31 16:12:40.917 JupyterHub sshspawner:109] start_start
[E 2021-12-31 16:12:40.964 JupyterHub sshspawner:117] self.remote_random_port_start
[E 2021-12-31 16:12:41.204 JupyterHub sshspawner:119] self.remote_random_port_end
[W 2021-12-31 16:12:41.205 JupyterHub spawner:946] Setting port from user.server is deprecated as of JupyterHub 0.7.
[E 2021-12-31 16:12:41.205 JupyterHub sshspawner:254]
[E 2021-12-31 16:12:41.221 JupyterHub sshspawner:285] /tmp/app1_run.sh was written as:
#!/bin/bash
export PATH=/usr/bin:/usr/local/anaconda3/bin/:/bin/:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/aws/bin
export CONDA_DEFAULT_ENV=base
export LANG=en_US.UTF-8
export JUPYTERHUB_API_TOKEN=8b67171e21794e418ed819b417615593
export JPY_API_TOKEN=8b67171e21794e418ed819b417615593
export JUPYTERHUB_CLIENT_ID=jupyterhub-user-app1
export JUPYTERHUB_HOST=172.16.160.106
export JUPYTERHUB_OAUTH_CALLBACK_URL=/user/app1/oauth_callback
export JUPYTERHUB_USER=app1
export JUPYTERHUB_SERVER_NAME=
export JUPYTERHUB_API_URL=http://172.16.160.106:8081/hub/api
export JUPYTERHUB_ACTIVITY_URL=http://172.16.160.106:8081/hub/api/users/app1/activity
export JUPYTERHUB_BASE_URL=/
export JUPYTERHUB_SERVICE_PREFIX=/user/app1/
export MEM_LIMIT=2147483648
export MEM_GUARANTEE=209715200
export CPU_LIMIT=1.0
unset XDG_RUNTIME_DIR
touch jupyter.log
chmod 600 jupyter.log
jupyterhub-singleuser --port=34055 --debug >> jupyter.log 2>&1 & pid=$!
echo $PWD > /tmp/pwd.txt
echo $pid
[E 2021-12-31 16:12:41.351 JupyterHub sshspawner:295] exec_notebook status=0
[E 2021-12-31 16:12:41.351 JupyterHub sshspawner:163] Starting User: app1, PID: 22568
[E 2021-12-31 16:12:41.351 JupyterHub sshspawner:167] start_end
[I 2021-12-31 16:12:41.884 JupyterHub log:174] 302 GET /hub/spawn/app1 -> /hub/spawn-pending/app1 (app1@172.16.7.221) 1004.45ms
[I 2021-12-31 16:12:41.896 JupyterHub pages:347] app1 is pending spawn
[I 2021-12-31 16:12:41.900 JupyterHub log:174] 200 GET /hub/spawn-pending/app1 (app1@172.16.7.221) 7.31ms
[I 2021-12-31 16:12:44.827 JupyterHub log:174] 200 GET /hub/api (@172.16.160.105) 1.07ms
[I 2021-12-31 16:12:44.850 JupyterHub log:174] 200 POST /hub/api/users/app1/activity (app1@172.16.160.105) 14.42ms
[E 2021-12-31 16:12:51.007 JupyterHub sshspawner:321] command: kill -s 0 22568 < /dev/null returned --- --- 0
[W 2021-12-31 16:12:51.007 JupyterHub base:982] User app1 is slow to become responsive (timeout=10)
[W 2021-12-31 16:13:09.477 JupyterHub user:692] app1's server never showed up at http://172.16.160.105:34055/user/app1/ after 30 seconds. Giving up
[E 2021-12-31 16:13:09.605 JupyterHub sshspawner:321] command: kill -s 0 22568 < /dev/null returned --- --- 0
[E 2021-12-31 16:13:09.715 JupyterHub sshspawner:321] command: kill -s 0 22568 < /dev/null returned --- --- 0
[E 2021-12-31 16:13:09.735 JupyterHub gen:599] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, de fined at /usr/local/anaconda3/lib/python3.7/site-packages/jupyterhub/handlers/base.py:845> exception=TimeoutError("Server at http://172.16.160.105:34055/us er/app1/ didn't respond in 30 seconds")> after timeout
Traceback (most recent call last):
File "/usr/local/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 593, in error_callback
future.result()
File "/usr/local/anaconda3/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 852, in finish_user_spawn
await spawn_future
File "/usr/local/anaconda3/lib/python3.7/site-packages/jupyterhub/user.py", line 668, in spawn
await self._wait_up(spawner)
File "/usr/local/anaconda3/lib/python3.7/site-packages/jupyterhub/user.py", line 715, in _wait_up
raise e
File "/usr/local/anaconda3/lib/python3.7/site-packages/jupyterhub/user.py", line 683, in _wait_up
http=True, timeout=spawner.http_timeout, ssl_context=ssl_context
File "/usr/local/anaconda3/lib/python3.7/site-packages/jupyterhub/utils.py", line 234, in wait_for_http_server
timeout=timeout,
File "/usr/local/anaconda3/lib/python3.7/site-packages/jupyterhub/utils.py", line 177, in exponential_backoff
raise TimeoutError(fail_message)
TimeoutError: Server at http://172.16.160.105:34055/user/app1/ didn't respond in 30 seconds
[I 2021-12-31 16:13:09.738 JupyterHub log:174] 200 GET /hub/api/users/app1/server/progress (app1@172.16.7.221) 27789.44ms
[W 2021-12-31 16:17:17.491 JupyterHub log:174] 403 POST /hub/api/users/app1/activity (@172.16.160.105) 6.64ms
[W 2021-12-31 16:17:18.249 JupyterHub log:174] 403 POST /hub/api/users/app1/activity (@172.16.160.105) 3.55ms
[W 2021-12-31 16:17:18.440 JupyterHub log:174] 403 POST /hub/api/users/app1/activity (@172.16.160.105) 3.20ms
[W 2021-12-31 16:17:21.342 JupyterHub log:174] 403 POST /hub/api/users/app1/activity (@172.16.160.105) 4.71ms
16:17:25.520 [ConfigProxy] info: 200 GET /api/routes
[I 2021-12-31 16:17:25.520 JupyterHub proxy:320] Checking routes
[W 2021-12-31 16:17:29.039 JupyterHub log:174] 403 POST /hub/api/users/app1/activity (@172.16.160.105) 4.98ms
This is the log printed by jupyterhub-singleuser
[D 2021-12-31 16:12:41.913 SingleUserNotebookApp application:556] Looking for jupyter_config in /etc/jupyter
[D 2021-12-31 16:12:41.914 SingleUserNotebookApp application:556] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2021-12-31 16:12:41.914 SingleUserNotebookApp application:556] Looking for jupyter_config in /usr/local/anaconda3/etc/jupyter
[D 2021-12-31 16:12:41.914 SingleUserNotebookApp application:556] Looking for jupyter_config in /home/app1/.jupyter
[D 2021-12-31 16:12:41.914 SingleUserNotebookApp application:556] Looking for jupyter_config in /home/app1
[D 2021-12-31 16:12:41.915 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /etc/jupyter
[D 2021-12-31 16:12:41.915 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2021-12-31 16:12:41.915 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /usr/local/anaconda3/etc/jupyter
[D 2021-12-31 16:12:41.916 SingleUserNotebookApp application:578] Loaded config file: /usr/local/anaconda3/etc/jupyter/jupyter_notebook_config.json
[D 2021-12-31 16:12:41.916 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /home/app1/.jupyter
[D 2021-12-31 16:12:41.916 SingleUserNotebookApp application:556] Looking for jupyter_notebook_config in /home/app1
[D 2021-12-31 16:12:41.918 SingleUserNotebookApp notebookapp:1426] Raising open file limit: soft 1024->4096; hard 4096->4096
[D 2021-12-31 16:12:41.923 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
/etc/jupyter/jupyter_notebook_config.json
[D 2021-12-31 16:12:41.924 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
/usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2021-12-31 16:12:41.924 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
/usr/local/anaconda3/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
/usr/local/anaconda3/etc/jupyter/jupyter_notebook_config.json
[D 2021-12-31 16:12:41.925 SingleUserNotebookApp config_manager:97] Paths used for configuration of jupyter_notebook_config:
/home/app1/.jupyter/jupyter_notebook_config.json
[I 2021-12-31 16:12:43.350 SingleUserNotebookApp extension:158] JupyterLab extension loaded from /usr/local/anaconda3/lib/python3.7/site-packages/jupyterlab
[I 2021-12-31 16:12:43.351 SingleUserNotebookApp extension:159] JupyterLab application directory is /usr/local/anaconda3/share/jupyter/lab
[D 2021-12-31 16:12:44.356 SingleUserNotebookApp commands:1792] Yarn configuration loaded.
[I 2021-12-31 16:12:44.821 SingleUserNotebookApp handlers:154] sparkmagic extension enabled!
[I 2021-12-31 16:12:44.822 SingleUserNotebookApp singleuser:561] Starting jupyterhub-singleuser server version 1.1.0
[D 2021-12-31 16:12:44.828 SingleUserNotebookApp _version:60] jupyterhub and jupyterhub-singleuser both on version 1.1.0
[I 2021-12-31 16:12:44.828 SingleUserNotebookApp notebookapp:1924] Serving notebooks from local directory: /home/app1
[I 2021-12-31 16:12:44.828 SingleUserNotebookApp notebookapp:1924] The Jupyter Notebook is running at:
[I 2021-12-31 16:12:44.829 SingleUserNotebookApp notebookapp:1924] http://127.0.0.1:34055/user/app1/
[I 2021-12-31 16:12:44.829 SingleUserNotebookApp notebookapp:1925] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2021-12-31 16:12:44.833 SingleUserNotebookApp singleuser:542] Updating Hub with activity every 300 seconds
[D 2021-12-31 16:12:44.834 SingleUserNotebookApp singleuser:503] Notifying Hub of activity 2021-12-31T08:12:41.929508Z
[D 2021-12-31 16:17:17.480 SingleUserNotebookApp singleuser:503] Notifying Hub of activity 2021-12-31T08:12:41.929508Z
[E 2021-12-31 16:17:17.493 SingleUserNotebookApp singleuser:523] Error notifying Hub of activity
Traceback (most recent call last):
File "/usr/local/anaconda3/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 521, in notify
await client.fetch(req)
tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
[D 2021-12-31 16:17:18.242 SingleUserNotebookApp singleuser:503] Notifying Hub of activity 2021-12-31T08:12:41.929508Z
[E 2021-12-31 16:17:18.249 SingleUserNotebookApp singleuser:523] Error notifying Hub of activity
Traceback (most recent call last):
File "/usr/local/anaconda3/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 521, in notify
await client.fetch(req)
tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
[D 2021-12-31 16:17:18.434 SingleUserNotebookApp singleuser:503] Notifying Hub of activity 2021-12-31T08:12:41.929508Z
[E 2021-12-31 16:17:18.441 SingleUserNotebookApp singleuser:523] Error notifying Hub of activity
Traceback (most recent call last):
File "/usr/local/anaconda3/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 521, in notify
await client.fetch(req)
tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
[D 2021-12-31 16:17:21.334 SingleUserNotebookApp singleuser:503] Notifying Hub of activity 2021-12-31T08:12:41.929508Z
[E 2021-12-31 16:17:21.342 SingleUserNotebookApp singleuser:523] Error notifying Hub of activity
Traceback (most recent call last):
File "/usr/local/anaconda3/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 521, in notify
await client.fetch(req)
tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
[D 2021-12-31 16:17:29.031 SingleUserNotebookApp singleuser:503] Notifying Hub of activity 2021-12-31T08:12:41.929508Z
[E 2021-12-31 16:17:29.040 SingleUserNotebookApp singleuser:523] Error notifying Hub of activity
Traceback (most recent call last):
File "/usr/local/anaconda3/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 521, in notify
await client.fetch(req)
tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
[D 2021-12-31 16:17:42.724 SingleUserNotebookApp singleuser:503] Notifying Hub of activity 2021-12-31T08:12:41.929508Z
[E 2021-12-31 16:17:42.732 SingleUserNotebookApp singleuser:523] Error notifying Hub of activity
Traceback (most recent call last):
File "/usr/local/anaconda3/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 521, in notify
await client.fetch(req)
tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
[D 2021-12-31 16:17:57.747 SingleUserNotebookApp singleuser:503] Notifying Hub of activity 2021-12-31T08:12:41.929508Z
[E 2021-12-31 16:17:57.754 SingleUserNotebookApp singleuser:523] Error notifying Hub of activity
Traceback (most recent call last):
File "/usr/local/anaconda3/lib/python3.7/site-packages/jupyterhub/singleuser.py", line 521, in notify
await client.fetch(req)
tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
[D 2021-12-31 16:18:11.665 SingleUserNotebookApp singleuser:503] Notifying Hub of activity
This is juputerhub-singleuser for viewing 105 successful startup
bash-4.2$ jupyter notebook list
Currently running servers:
http://127.0.0.1:34240/user/app1/ :: /home/app1
http://127.0.0.1:34055/user/app1/ :: /home/app1
bash-4.2$