I launched jupyterhub on my own server and accessed to it on my local computer via nginx with this config file:
import pwd, subprocess, requests
# Settings
c.JupyterHub.authenticator_class = 'jupyterhub.auth.PAMAuthenticator'
# Admin User
c.Authenticator.admin_users = {'jupyter'}
# Network settings
c.JupyterHub.ip = '127.0.0.1' #Ip used by users to access server
c.JupyterHub.hub_ip = '0.0.0.0' #Listen all interfaces
# Directory for the lab and notebooks
c.Spawner.default_url = '/lab'
c.Spawner.notebook_dir = '~/notebooks'
def pre_spawn_hook(spawner):
username = spawner.user.name
try:
pwd.getpwnam(username)
except KeyError:
subprocess.check_call(['useradd', '-ms', '/bin/bash', username])
c.Spawner.pre_spawn_hook = pre_spawn_hook
But I got the error API request failed (403): Forbidden and here is the output on my terminal :
[I 2021-05-06 19:09:21.299 JupyterHub app:2853] JupyterHub is now running at http://127.0.0.1:8000
[W 2021-05-06 19:09:27.331 JupyterHub base:51] Blocking API request with no referer
[W 2021-05-06 19:09:27.332 JupyterHub log:189] 403 POST /hub/api/users/jupyter/server (@5.133.68.138) 12.65ms
I checked several times my config file but I can’t find where’s the problem
I also think it may be caused by nginx and I reviewed my config file that seems correct
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 80;
listen [::]:80;
server_name jupyter.kdata.fr;
return 301 https://$host$request_uri;
}
# HTTPS server to handle JupyterHub
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name jupyter.kdata.fr;
include /etc/nginx/snippets/ssl_common_options.conf;
include /etc/nginx/snippets/ssl_kdata_options.conf;
# HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
add_header Strict-Transport-Security max-age=15768000;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header X-Forwarded-User $http_authorization;
proxy_pass http://127.0.0.1:8081;
# websocket headers
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade²;
proxy_set_header X-Scheme $scheme;
#proxy_redirect default;
proxy_buffering off;
}
location ~ /hub/api/ {
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#proxy_set_header X-Forwarded-User $http_authorization;
proxy_pass http://127.0.0.1:8081;
# proxy_redirect http://127.0.0.1:8081/hub/api /hub/api;
}
location ~ /\.ht {
deny all;
}
location ^~ /.git {
deny all;
return 403;
}
}
Anyone have already encountered this problem and fixed it?
Hi! Could you point to the instructions you were following, as it looks like they’re incorrect.
JupyterHub defaults to listening on port 8000. 8081 is for internal API use only.
You don’t need a separate route for /hub/api as this should be handled by port 8000. As mentioned 8081 is only for internal use.
Hi, thanks you in advance for your help !
Basically, I added another route to try to catch the API request to /hub/api as it wasn’t working. So, I set up my nginx config file and passed the proxy to the right port :
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 80;
listen [::]:80;
server_name jupyter.kdata.fr;
return 301 https://$host$request_uri;
}
# HTTPS server to handle JupyterHub
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name jupyter.kdata.fr;
include /etc/nginx/snippets/ssl_common_options.conf;
include /etc/nginx/snippets/ssl_kdata_options.conf;
access_log /home/jupyter/logs/jupyter-access.log;
error_log /home/jupyter/logs/jupyter-error.log;
# HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
add_header Strict-Transport-Security max-age=15768000;
location / {
proxy_set_header Host $host;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:8000;
# proxy_redirect https:127.0.0.1:8000 https://jupyterhub.kdata;
# websocket headers
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Scheme $scheme;
#proxy_redirect default;
proxy_buffering off;
}
location ~ /\.ht {
deny all;
}
location ^~ /.git {
deny all;
return 403;
}
}
But I still get the API request failed (403): Forbidden error with Blocking API request with no referer…
Is my jupyterhub_config.py right too according to you?
Hi,
Since now, I didn’t find the solution… I think maybe there’s a problem on my Jupyterhub config file but I can’t find the solution. Basically, it should work because I used a basic config file that’s working for people but not for me.
Here’s the new config file :
#c.LocalAuthenticator.create_system_users = True
# Settings
c.JupyterHub.authenticator_class = 'jupyterhub.auth.PAMAuthenticator'
# Admin User
c.Authenticator.admin_users = {'jupyter','mndiaye'}
# Network settings
c.JupyterHub.bind_url = 'http://127.0.0.1:8000'
# Directory for the lab and notebooks
c.Spawner.default_url = '/lab'
c.Spawner.notebook_dir = '~/notebooks'
As you can see, with this config file not complicated, I should not have this kind of error … I’m completely lost.
Does JupyterHub work if you connect directly to port 8000 instead of going via Nginx?
I’m launching JupyterHub from a server, so unfortunately I can’t because there’s no web interface.
Can you enable debug logging in JupyterHub and show us your logs?
Here’s my log :
root@ds0-rstudio1:/etc/jupyterhub# jupyterhub
[I 2021-05-26 10:59:45.913 JupyterHub app:2463] Running JupyterHub version 1.4.0
[I 2021-05-26 10:59:45.913 JupyterHub app:2494] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.4.0
[I 2021-05-26 10:59:45.914 JupyterHub app:2494] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-1.4.0
[I 2021-05-26 10:59:45.914 JupyterHub app:2494] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.4.0
[I 2021-05-26 10:59:45.919 JupyterHub app:1534] Loading cookie_secret from /etc/jupyterhub/jupyterhub_cookie_secret
[D 2021-05-26 10:59:45.919 JupyterHub app:1701] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2021-05-26 10:59:45.929 JupyterHub orm:815] database schema version found: 4dc2d5a8c53c
[I 2021-05-26 10:59:45.932 JupyterHub proxy:490] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2021-05-26 10:59:45.954 JupyterHub app:1839] Not using allowed_users. Any authenticated user will be allowed.
[D 2021-05-26 10:59:45.959 JupyterHub app:1990] Purging expired APITokens
[D 2021-05-26 10:59:45.962 JupyterHub app:1990] Purging expired OAuthAccessTokens
[D 2021-05-26 10:59:45.964 JupyterHub app:1990] Purging expired OAuthCodes
[D 2021-05-26 10:59:45.971 JupyterHub app:2117] Initializing spawners
[D 2021-05-26 10:59:45.972 JupyterHub app:2250] Loaded users:
[I 2021-05-26 10:59:45.972 JupyterHub app:2533] Initialized 0 spawners in 0.002 seconds
[W 2021-05-26 10:59:45.975 JupyterHub proxy:693] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else...
[I 2021-05-26 10:59:45.975 JupyterHub proxy:696] Starting proxy @ http://127.0.0.1:8000
[D 2021-05-26 10:59:45.975 JupyterHub proxy:697] Proxy cmd: ['configurable-http-proxy', '--ip', '127.0.0.1', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://127.0.0.1:8081/hub/error']
[D 2021-05-26 10:59:45.978 JupyterHub proxy:604] Writing proxy pid file: jupyterhub-proxy.pid
10:59:46.480 [ConfigProxy] info: Proxying http://127.0.0.1:8000 to (no default)
10:59:46.483 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2021-05-26 10:59:46.789 JupyterHub proxy:732] Proxy started and appears to be up
[D 2021-05-26 10:59:46.789 JupyterHub proxy:825] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
10:59:46.803 [ConfigProxy] info: 200 GET /api/routes
[I 2021-05-26 10:59:46.803 JupyterHub app:2778] Hub API listening on http://127.0.0.1:8081/hub/
[D 2021-05-26 10:59:46.803 JupyterHub proxy:335] Fetching routes to check
[D 2021-05-26 10:59:46.803 JupyterHub proxy:825] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
10:59:46.804 [ConfigProxy] info: 200 GET /api/routes
[I 2021-05-26 10:59:46.804 JupyterHub proxy:340] Checking routes
[I 2021-05-26 10:59:46.804 JupyterHub proxy:425] Adding route for Hub: / => http://127.0.0.1:8081
[D 2021-05-26 10:59:46.805 JupyterHub proxy:825] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
10:59:46.806 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
10:59:46.806 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
10:59:46.807 [ConfigProxy] info: 201 POST /api/routes/
[I 2021-05-26 10:59:46.807 JupyterHub app:2853] JupyterHub is now running at http://127.0.0.1:8000
[D 2021-05-26 10:59:46.807 JupyterHub app:2456] It took 0.924 seconds for the Hub to start
[D 2021-05-26 10:59:54.300 JupyterHub base:328] Refreshing auth for mndiaye
[D 2021-05-26 10:59:54.300 JupyterHub user:288] Creating <class 'jupyterhub.spawner.LocalProcessSpawner'> for mndiaye:
[I 2021-05-26 10:59:54.325 JupyterHub log:189] 200 GET /hub/home (mndiaye@5.133.68.138) 37.83ms
10:59:54.459 [ConfigProxy] error: 503 GET /hub/static/favicon.ico socket hang up
[D 2021-05-26 10:59:54.463 JupyterHub pages:652] No template for 503
[I 2021-05-26 10:59:54.468 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fhub%2Fstatic%2Ffavicon.ico%3Fv%3Dfde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@127.0.0.1) 6.48ms
[D 2021-05-26 10:59:54.594 JupyterHub log:189] 200 GET /hub/static/js/home.js?v=20210526105945 (@5.133.68.138) 1.06ms
[D 2021-05-26 10:59:54.655 JupyterHub log:189] 200 GET /hub/static/components/moment/moment.js?v=20210526105945 (@5.133.68.138) 0.85ms
[D 2021-05-26 10:59:54.727 JupyterHub log:189] 200 GET /hub/static/js/jhapi.js?v=20210526105945 (@5.133.68.138) 0.76ms
[D 2021-05-26 10:59:54.777 JupyterHub log:189] 200 GET /hub/static/js/utils.js?v=20210526105945 (@5.133.68.138) 0.69ms
[D 2021-05-26 10:59:55.334 JupyterHub pages:218] Triggering spawn with default options for mndiaye
[D 2021-05-26 10:59:55.334 JupyterHub base:880] Initiating spawn for mndiaye
[D 2021-05-26 10:59:55.334 JupyterHub base:887] 0/100 concurrent spawns
[D 2021-05-26 10:59:55.334 JupyterHub base:892] 0 active servers
[I 2021-05-26 10:59:55.346 JupyterHub provider:574] Creating oauth client jupyterhub-user-mndiaye
Last login: Wed May 26 10:47:01 CEST 2021
Linux ds0-rstudio1 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64
SERVER MANAGED BY EVOLIX VIA ANSIBLE
------------------------------------
https://forge.evolix.org/projects/capitaldata/repository
[D 2021-05-26 10:59:55.366 JupyterHub user:602] Calling Spawner.start for mndiaye
[I 2021-05-26 10:59:55.367 JupyterHub spawner:1471] Spawning jupyterhub-singleuser --port=35879 '--notebook-dir=~/notebooks' --SingleUserNotebookApp.default_url=/lab --debug
[D 2021-05-26 10:59:55.408 JupyterHub spawner:1167] Polling subprocess every 30s
[C 2021-05-26 10:59:55.913 SingleUserNotebookApp application:89] Bad config encountered during initialization: No such notebook dir: '/home/mndiaye/notebooks'
[I 2021-05-26 10:59:56.336 JupyterHub log:189] 302 GET /hub/spawn/mndiaye -> /hub/spawn-pending/mndiaye (mndiaye@5.133.68.138) 1004.60ms
[I 2021-05-26 10:59:56.512 JupyterHub pages:402] mndiaye is pending spawn
[I 2021-05-26 10:59:56.516 JupyterHub log:189] 200 GET /hub/spawn-pending/mndiaye (mndiaye@5.133.68.138) 6.68ms
[W 2021-05-26 10:59:56.679 JupyterHub base:51] Blocking API request with no referer
[W 2021-05-26 10:59:56.680 JupyterHub log:189] 403 GET /hub/api/users/mndiaye/server/progress (@5.133.68.138) 1.82ms
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<BaseHandler.spawn_single_user() done, defined at /usr/local/lib/python3.7/dist-packages/jupyterhub/handlers/base.py:796> exception=HTTPError()>
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/jupyterhub/handlers/base.py", line 997, in spawn_single_user
timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
tornado.util.TimeoutError: Timeout
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/jupyterhub/handlers/base.py", line 1033, in spawn_single_user
% (status, spawner._log_name),
tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=1]. The logs for mndiaye may contain details.)
[W 2021-05-26 11:00:26.507 JupyterHub user:743] mndiaye's server never showed up at http://127.0.0.1:35879/user/mndiaye/ after 30 seconds. Giving up
[D 2021-05-26 11:00:26.507 JupyterHub user:790] Stopping mndiaye
[D 2021-05-26 11:00:26.516 JupyterHub user:816] Deleting oauth client jupyterhub-user-mndiaye
[D 2021-05-26 11:00:26.519 JupyterHub user:819] Finished stopping mndiaye
[E 2021-05-26 11:00:26.525 JupyterHub gen:624] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.7/dist-packages/jupyterhub/handlers/base.py:900> exception=TimeoutError("Server at http://127.0.0.1:35879/user/mndiaye/ didn't respond in 30 seconds")> after timeout
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/tornado/gen.py", line 618, in error_callback
future.result()
File "/usr/local/lib/python3.7/dist-packages/jupyterhub/handlers/base.py", line 907, in finish_user_spawn
await spawn_future
File "/usr/local/lib/python3.7/dist-packages/jupyterhub/user.py", line 719, in spawn
await self._wait_up(spawner)
File "/usr/local/lib/python3.7/dist-packages/jupyterhub/user.py", line 766, in _wait_up
raise e
File "/usr/local/lib/python3.7/dist-packages/jupyterhub/user.py", line 734, in _wait_up
http=True, timeout=spawner.http_timeout, ssl_context=ssl_context
File "/usr/local/lib/python3.7/dist-packages/jupyterhub/utils.py", line 241, in wait_for_http_server
timeout=timeout,
File "/usr/local/lib/python3.7/dist-packages/jupyterhub/utils.py", line 184, in exponential_backoff
raise TimeoutError(fail_message)
TimeoutError: Server at http://127.0.0.1:35879/user/mndiaye/ didn't respond in 30 seconds
I hope it’s gonna help u because I saw no additionnal infos 
Does that directory exist?