Jupyterhub 403 forbidden even with allow_all: true

JupyterHub
1

I deployed Jupyterhub with the following setting:

c.Authenticator.allow_all = True

however when I try to access it I see 403: Forbidden

2

You need to provide more details like your JupyterHub config and logs (in DEBUG) mode for others to help you.

3

I am deploying via binderhub. This is the jupyterhub config:

jupyterhub:
  debug:
    enabled: true
  proxy:
    service:
      type: ClusterIP
  hub:
    baseUrl: /jupyterhub/
    config:
      Authenticator:
        allow_all: true
4

Can you also provide your debug logs as requested earlier?

5

could you please clarify what kind of debug logs I should provide? the logs I provided for both the hub and binder pod are already with debug enabled, e.g.:

jupyterhub:
  debug:
    enabled: true

config:
  BinderHub: 
    debug: true

are you asking for the entire startup logs for the hub pod? if so:

[D 2025-03-07 19:13:04.605 JupyterHub application:908] Looking for /usr/local/etc/jupyterhub/jupyterhub_config in /srv/jupyterhub
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
Loading extra config: 0-binderspawnermixin
Loading extra config: 00-binder
[D 2025-03-07 19:13:05.181 JupyterHub application:929] Loaded config file: /usr/local/etc/jupyterhub/jupyterhub_config.py
[I 2025-03-07 19:13:05.192 JupyterHub app:3346] Running JupyterHub version 5.2.1
[I 2025-03-07 19:13:05.192 JupyterHub app:3376] Using Authenticator: jupyterhub.auth.NullAuthenticator-5.2.1
[I 2025-03-07 19:13:05.192 JupyterHub app:3376] Using Spawner: builtins.BinderSpawner
[I 2025-03-07 19:13:05.192 JupyterHub app:3376] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.2.1
[D 2025-03-07 19:13:05.195 JupyterHub app:1998] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2025-03-07 19:13:05.219 JupyterHub orm:1477] Stamping empty database with alembic revision 4621fec11365
[I 2025-03-07 19:13:05.222 alembic.runtime.migration migration:207] Context impl SQLiteImpl.
[I 2025-03-07 19:13:05.222 alembic.runtime.migration migration:210] Will assume non-transactional DDL.
[I 2025-03-07 19:13:05.235 alembic.runtime.migration migration:618] Running stamp_revision  -> 4621fec11365
[D 2025-03-07 19:13:05.236 alembic.runtime.migration migration:826] new branch insert 4621fec11365
[D 2025-03-07 19:13:05.251 JupyterHub orm:1477] Stamping empty database with alembic revision 4621fec11365
[I 2025-03-07 19:13:05.253 alembic.runtime.migration migration:207] Context impl SQLiteImpl.
[I 2025-03-07 19:13:05.253 alembic.runtime.migration migration:210] Will assume non-transactional DDL.
[D 2025-03-07 19:13:05.514 JupyterHub app:2338] Loading roles into database
[D 2025-03-07 19:13:05.515 JupyterHub app:2347] Loading role jupyterhub-idle-culler
[D 2025-03-07 19:13:05.518 JupyterHub app:2347] Loading role binder
[I 2025-03-07 19:13:05.624 JupyterHub roles:197] Role jupyterhub-idle-culler added to database
[I 2025-03-07 19:13:05.625 JupyterHub roles:197] Role binder added to database
[I 2025-03-07 19:13:05.661 JupyterHub app:2919] Creating service jupyterhub-idle-culler without oauth.
[I 2025-03-07 19:13:05.674 JupyterHub app:2919] Creating service binder without oauth.
[I 2025-03-07 19:13:05.695 JupyterHub app:2655] Adding API token for service: binder
[D 2025-03-07 19:13:05.708 JupyterHub app:2685] Purging expired APITokens
[D 2025-03-07 19:13:05.711 JupyterHub app:2685] Purging expired OAuthCodes
[D 2025-03-07 19:13:05.714 JupyterHub app:2685] Purging expired Shares
[D 2025-03-07 19:13:05.717 JupyterHub app:2685] Purging expired ShareCodes
[D 2025-03-07 19:13:05.720 JupyterHub app:2459] Loading role assignments from config
[D 2025-03-07 19:13:05.772 JupyterHub app:2970] Initializing spawners
[D 2025-03-07 19:13:05.787 JupyterHub app:3120] Loaded users:
    
[I 2025-03-07 19:13:05.787 JupyterHub app:3416] Initialized 0 spawners in 0.015 seconds
[I 2025-03-07 19:13:05.795 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2025-03-07 19:13:05.796 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2025-03-07 19:13:05.797 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.thirty_days
[I 2025-03-07 19:13:05.798 JupyterHub app:3703] Not starting proxy
[D 2025-03-07 19:13:05.798 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-03-07 19:13:05.805 JupyterHub proxy:996] Omitting non-jupyterhub route '/'
[I 2025-03-07 19:13:05.806 JupyterHub app:3739] Hub API listening on http://:8081/jupyterhub/hub/
[I 2025-03-07 19:13:05.806 JupyterHub app:3741] Private Hub API connect url http://hub:8081/jupyterhub/hub/
[I 2025-03-07 19:13:05.806 JupyterHub app:3615] Starting managed service jupyterhub-idle-culler
[I 2025-03-07 19:13:05.806 JupyterHub service:423] Starting service 'jupyterhub-idle-culler': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/jupyterhub/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10', '--cull-users']
[I 2025-03-07 19:13:05.808 JupyterHub service:136] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/jupyterhub/hub/api --timeout=3600 --cull-every=600 --concurrency=10 --cull-users
[D 2025-03-07 19:13:05.809 JupyterHub spawner:1475] Polling subprocess every 30s
[I 2025-03-07 19:13:05.810 JupyterHub app:3624] Adding external service binder
[D 2025-03-07 19:13:05.810 JupyterHub proxy:389] Fetching routes to check
[D 2025-03-07 19:13:05.810 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-03-07 19:13:05.813 JupyterHub proxy:996] Omitting non-jupyterhub route '/'
[D 2025-03-07 19:13:05.813 JupyterHub proxy:392] Checking routes
[I 2025-03-07 19:13:05.814 JupyterHub proxy:477] Adding route for Hub: /jupyterhub/ => http://hub:8081
[D 2025-03-07 19:13:05.814 JupyterHub proxy:925] Proxy: Fetching POST http://proxy-api:8001/api/routes/jupyterhub
[I 2025-03-07 19:13:05.819 JupyterHub app:3772] JupyterHub is now running, internal Hub API at http://hub:8081/jupyterhub/hub/
[D 2025-03-07 19:13:05.820 JupyterHub app:3339] It took 1.228 seconds for the Hub to start
[D 2025-03-07 19:13:06.137 JupyterHub base:366] Recording first activity for <APIToken('b518...', service='jupyterhub-idle-culler', client_id='jupyterhub')>
[I 2025-03-07 19:13:06.146 JupyterHub log:192] 200 GET /jupyterhub/hub/api/ (jupyterhub-idle-culler@127.0.0.1) 10.84ms
[D 2025-03-07 19:13:06.151 JupyterHub scopes:1010] Checking access to /jupyterhub/hub/api/users via scope list:users
[I 2025-03-07 19:13:06.169 JupyterHub log:192] 200 GET /jupyterhub/hub/api/users?state=[secret] (jupyterhub-idle-culler@127.0.0.1) 20.47ms
[D 2025-03-07 19:13:06.173 JupyterHub scopes:1010] Checking access to /jupyterhub/hub/api/users via scope list:users
[I 2025-03-07 19:13:06.191 JupyterHub log:192] 200 GET /jupyterhub/hub/api/users?state=[secret] (jupyterhub-idle-culler@127.0.0.1) 20.21ms
[D 2025-03-07 19:13:06.396 JupyterHub log:192] 200 GET /jupyterhub/hub/health (@10.92.1.1) 0.87ms
[D 2025-03-07 19:13:08.396 JupyterHub log:192] 200 GET /jupyterhub/hub/health (@10.92.1.1) 0.80ms
....
....
[W 2025-03-07 19:15:21.586 JupyterHub web:1873] 403 GET /jupyterhub/hub/api/users/paololazzari-binder-redis-python-p6hxng6u (::ffff:10.92.1.109): Missing or invalid credentials.
[W 2025-03-07 19:15:21.587 JupyterHub log:192] 403 GET /jupyterhub/hub/api/users/paololazzari-binder-redis-python-p6hxng6u (@::ffff:10.92.1.109) 1.86ms

if you’re asking for the proxy pod logs then here they are:

18:58:39.408 [ConfigProxy] debug: PROXY WEB /jupyterhub/hub/api/users/paololazzari-binder-redis-python-sntjd61g to http://hub:8081
18:58:39.416 [ConfigProxy] debug: Not recording activity for status 403 on /jupyterhub
6

Thanks for the logs. Yes, we are looking for hub logs like the ones you posted in your last comment.

Could you please try with DummyAuthenticator and share the logs right from the beginning like you did in this comment? I would suggest to access JupyterHub in an incognito session just to ensure we start on clean slate.

7

Here are the startup logs for the hub pod deployed with this configuration:

jupyterhub:
  debug:
    enabled: true
  proxy:
    service:
      type: ClusterIP
  hub:
    baseUrl: /jupyterhub/
    config:
      Authenticator:
        allow_all: true
      DummyAuthenticator:
        password: foobar
      JupyterHub:
        authenticator_class: dummy
config:
  BinderHub: 
    debug: true
    use_registry: true
    base_url: /binder/
    image_prefix: us-central1-docker.pkg.dev/myprojectid/binderhub/binder
    hub_url: http://mywebsite.com/jupyterhub/

[D 2025-03-08 09:32:37.981 JupyterHub application:908] Looking for /usr/local/etc/jupyterhub/jupyterhub_config in /srv/jupyterhub
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
Loading extra config: 0-binderspawnermixin
Loading extra config: 00-binder
[D 2025-03-08 09:32:38.554 JupyterHub application:929] Loaded config file: /usr/local/etc/jupyterhub/jupyterhub_config.py
[I 2025-03-08 09:32:38.565 JupyterHub app:3346] Running JupyterHub version 5.2.1
[I 2025-03-08 09:32:38.565 JupyterHub app:3376] Using Authenticator: jupyterhub.auth.DummyAuthenticator-5.2.1
[I 2025-03-08 09:32:38.565 JupyterHub app:3376] Using Spawner: builtins.BinderSpawner
[I 2025-03-08 09:32:38.565 JupyterHub app:3376] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.2.1
[D 2025-03-08 09:32:38.568 JupyterHub app:1998] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2025-03-08 09:32:38.590 JupyterHub orm:1477] Stamping empty database with alembic revision 4621fec11365
[I 2025-03-08 09:32:38.593 alembic.runtime.migration migration:207] Context impl SQLiteImpl.
[I 2025-03-08 09:32:38.593 alembic.runtime.migration migration:210] Will assume non-transactional DDL.
[I 2025-03-08 09:32:38.605 alembic.runtime.migration migration:618] Running stamp_revision  -> 4621fec11365
[D 2025-03-08 09:32:38.605 alembic.runtime.migration migration:826] new branch insert 4621fec11365
[D 2025-03-08 09:32:38.621 JupyterHub orm:1477] Stamping empty database with alembic revision 4621fec11365
[I 2025-03-08 09:32:38.623 alembic.runtime.migration migration:207] Context impl SQLiteImpl.
[I 2025-03-08 09:32:38.623 alembic.runtime.migration migration:210] Will assume non-transactional DDL.
[D 2025-03-08 09:32:38.839 JupyterHub app:2338] Loading roles into database
[D 2025-03-08 09:32:38.839 JupyterHub app:2347] Loading role jupyterhub-idle-culler
[D 2025-03-08 09:32:38.842 JupyterHub app:2347] Loading role binder
[I 2025-03-08 09:32:38.947 JupyterHub roles:197] Role jupyterhub-idle-culler added to database
[I 2025-03-08 09:32:38.948 JupyterHub roles:197] Role binder added to database
[W 2025-03-08 09:32:38.953 JupyterHub auth:1508] Using testing authenticator DummyAuthenticator! This is not meant for production!
[I 2025-03-08 09:32:38.982 JupyterHub app:2919] Creating service jupyterhub-idle-culler without oauth.
[I 2025-03-08 09:32:38.992 JupyterHub app:2919] Creating service binder without oauth.
[I 2025-03-08 09:32:39.011 JupyterHub app:2655] Adding API token for service: binder
[D 2025-03-08 09:32:39.024 JupyterHub app:2685] Purging expired APITokens
[D 2025-03-08 09:32:39.028 JupyterHub app:2685] Purging expired OAuthCodes
[D 2025-03-08 09:32:39.031 JupyterHub app:2685] Purging expired Shares
[D 2025-03-08 09:32:39.034 JupyterHub app:2685] Purging expired ShareCodes
[D 2025-03-08 09:32:39.037 JupyterHub app:2459] Loading role assignments from config
[D 2025-03-08 09:32:39.087 JupyterHub app:2970] Initializing spawners
[D 2025-03-08 09:32:39.101 JupyterHub app:3120] Loaded users:
    
[I 2025-03-08 09:32:39.101 JupyterHub app:3416] Initialized 0 spawners in 0.015 seconds
[I 2025-03-08 09:32:39.110 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2025-03-08 09:32:39.111 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2025-03-08 09:32:39.112 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.thirty_days
[I 2025-03-08 09:32:39.112 JupyterHub app:3703] Not starting proxy
[D 2025-03-08 09:32:39.113 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-03-08 09:32:39.120 JupyterHub proxy:996] Omitting non-jupyterhub route '/'
[I 2025-03-08 09:32:39.121 JupyterHub app:3739] Hub API listening on http://:8081/jupyterhub/hub/
[I 2025-03-08 09:32:39.121 JupyterHub app:3741] Private Hub API connect url http://hub:8081/jupyterhub/hub/
[I 2025-03-08 09:32:39.122 JupyterHub app:3615] Starting managed service jupyterhub-idle-culler
[I 2025-03-08 09:32:39.122 JupyterHub service:423] Starting service 'jupyterhub-idle-culler': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/jupyterhub/hub/api', '--timeout=3600', '--cull-every=600', '--concurrency=10', '--cull-users']
[I 2025-03-08 09:32:39.124 JupyterHub service:136] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/jupyterhub/hub/api --timeout=3600 --cull-every=600 --concurrency=10 --cull-users
[D 2025-03-08 09:32:39.125 JupyterHub spawner:1475] Polling subprocess every 30s
[I 2025-03-08 09:32:39.125 JupyterHub app:3624] Adding external service binder
[D 2025-03-08 09:32:39.126 JupyterHub proxy:389] Fetching routes to check
[D 2025-03-08 09:32:39.126 JupyterHub proxy:925] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2025-03-08 09:32:39.129 JupyterHub proxy:996] Omitting non-jupyterhub route '/'
[D 2025-03-08 09:32:39.129 JupyterHub proxy:392] Checking routes
[I 2025-03-08 09:32:39.129 JupyterHub proxy:477] Adding route for Hub: /jupyterhub/ => http://hub:8081
[D 2025-03-08 09:32:39.129 JupyterHub proxy:925] Proxy: Fetching POST http://proxy-api:8001/api/routes/jupyterhub
[I 2025-03-08 09:32:39.133 JupyterHub app:3772] JupyterHub is now running, internal Hub API at http://hub:8081/jupyterhub/hub/
[D 2025-03-08 09:32:39.134 JupyterHub app:3339] It took 1.168 seconds for the Hub to start
[D 2025-03-08 09:32:39.458 JupyterHub base:366] Recording first activity for <APIToken('537a...', service='jupyterhub-idle-culler', client_id='jupyterhub')>
[I 2025-03-08 09:32:39.467 JupyterHub log:192] 200 GET /jupyterhub/hub/api/ (jupyterhub-idle-culler@127.0.0.1) 10.98ms
[D 2025-03-08 09:32:39.472 JupyterHub scopes:1010] Checking access to /jupyterhub/hub/api/users via scope list:users
[I 2025-03-08 09:32:39.490 JupyterHub log:192] 200 GET /jupyterhub/hub/api/users?state=[secret] (jupyterhub-idle-culler@127.0.0.1) 20.02ms
[D 2025-03-08 09:32:39.494 JupyterHub scopes:1010] Checking access to /jupyterhub/hub/api/users via scope list:users
[I 2025-03-08 09:32:39.509 JupyterHub log:192] 200 GET /jupyterhub/hub/api/users?state=[secret] (jupyterhub-idle-culler@127.0.0.1) 17.52ms
[D 2025-03-08 09:32:39.810 JupyterHub log:192] 200 GET /jupyterhub/hub/health (@10.92.1.1) 1.17ms
[D 2025-03-08 09:32:41.810 JupyterHub log:192] 200 GET /jupyterhub/hub/health (@10.92.1.1) 1.14ms
...
...
[W 2025-03-08 09:33:07.248 JupyterHub web:1873] 403 GET /jupyterhub/hub/api/users/paololazzari-binder-redis-python-3ipdrg1h (::ffff:10.92.1.126): Missing or invalid credentials.
[W 2025-03-08 09:33:07.249 JupyterHub log:192] 403 GET /jupyterhub/hub/api/users/paololazzari-binder-redis-python-3ipdrg1h (@::ffff:10.92.1.126) 1.31ms

with this configuration I can login mywebsite.com/jupyterhub/hub with any user and with password foobar, but whenever I try to launch a repository from binder I see the Missing or invalid credentials error.

it looks like binder does not have permissions to call the hub api?

8

Did you take a look at the docs? I think you need more config to make JupyterHub as Oauth provider for Binderhub

9

Yes, I’ve read the docs. I want anyone to be able to launch a repository from my binder. Like how mybinder.org works.

If I configure jupyterhub with allow_all I would expect binder to be able to launch a user. Why does binder need permissions if jupyter is configured so?

10

I dont have experience with running BinderHub. Looking at the docs and source code, I understand that Binder service is run as JupyterHub service. In that case, you will need to configure a JupyterHub service definition for binder and specify roles and scopes for user to access binder service. In the docs you have example of how service and roles need to be configured. Can you try configuring services and loadRoles in the hub config?

1 Like
11

I tried adding this but it still doesn’t work:

    services:
      binder:
        oauth_client_id: service-binderhub
        oauth_no_confirm: true
        oauth_redirect_uri: "https://mywebsite.com/binder/oauth_callback"
    loadRoles:
      user:
        scopes:
          - self
          - "access:services!service=binder"

but I don’t understand why I would need this? why would I need oauth_redirect_uri if I don’t want any authentication?