404 on service with jupyterhub 4.1

JupyterHub
1

I’m trying to migrate my jupyterhub from 4.0.2 to 4.1 but my service returns:
WARNING:tornado.access:404 GET /services/myservice

c.JupyterHub.services = [
    {
        'name': 'myservice',
        'url': 'http://127.0.0.1:8887',
        'command': [sys.executable, str(HERE.joinpath("service.py"))],
        'environment': dict(os.environ),
        'oauth_no_confirm': True,
        'oauth_roles': ['user'],
    },
]

c.JupyterHub.load_roles = [
    {
        "name": "myservice",
        "services": ["myservice"],
        "scopes": ["admin:users", "admin:servers"],
    },
]

I tried to replace oauth_roles by oauth_client_allowed_scopes but without success.

Does anyone have an idea ?

The problem seems to appear at this commit : forward-port 4.1.0 · jupyterhub/jupyterhub@83ce6d3 · GitHub

Before this commit, the service works well.

Regards

2

Could you try to access your service with trailing slash? Like /services/myservice/

3

This returns a 404 error too.

4

Are you sure your service is running? Could you share JupyerHub and your service logs?

5

Yes the service is running:

[I 2024-04-18 15:04:33.570 JupyterHub app:3560] Hub API listening on http://:8081/hub/
[I 2024-04-18 15:04:33.571 JupyterHub app:3562] Private Hub API connect url http://jupyterhub:8081/hub/
[I 2024-04-18 15:04:33.571 JupyterHub app:3446] Starting managed service myservice at http://127.0.0.1:8887
[I 2024-04-18 15:04:33.571 JupyterHub service:415] Starting service 'myservice': ['/usr/bin/python3', '/usr/local/lib/python3.10/dist-packages/myservice/myservice.py']
[I 2024-04-18 15:04:33.574 JupyterHub service:136] Spawning /usr/bin/python3 /usr/local/lib/python3.10/dist-packages/myservice/myservice.py
/usr/local/lib/python3.10/dist-packages/jupyterhub/roles.py:114: UserWarning: Not expanding 'self' scope for owner <jupyterhub.orm.Service object at 0x7faf1b904d90> which is not a User
  return scopes.expand_scopes(roles_to_scopes(roles), owner=owner)
[I 2024-04-18 15:04:37.990 JupyterHub log:192] 200 GET /hub/api/users (myservice@127.0.0.1) 105.50ms
WARNING:tornado.access:404 GET /services/myservice (127.0.0.1) 1.05ms
15:04:38.034 [ConfigProxy] info: 200 GET /api/routes 
[I 2024-04-18 15:04:38.034 JupyterHub proxy:477] Adding route for Hub: / => http://jupyterhub:8081
[W 2024-04-18 15:04:38.035 JupyterHub proxy:445] Adding missing route for myservice (Server(url=http://127.0.0.1:8887/services/myservice/, bind_url=http://127.0.0.1:8887/services/myservice/))
[I 2024-04-18 15:04:38.037 JupyterHub proxy:312] Adding service myservice to proxy /services/myservice/ => http://127.0.0.1:8887
15:04:38.044 [ConfigProxy] info: Adding route / -> http://jupyterhub:8081
15:04:38.047 [ConfigProxy] info: Route added / -> http://jupyterhub:8081
15:04:38.049 [ConfigProxy] info: 201 POST /api/routes/ 
15:04:38.055 [ConfigProxy] info: Adding route /services/myservice -> http://127.0.0.1:8887
15:04:38.057 [ConfigProxy] info: Route added /services/myservice -> http://127.0.0.1:8887
[I 2024-04-18 15:04:38.059 JupyterHub app:3601] JupyterHub is now running at http://:8000
15:04:38.060 [ConfigProxy] info: 201 POST /api/routes/services/myservice
6

Is the source code for your service available, or alternatively are you able to modify your service so you can share a minimal example?

Can you share the logs for a failed user request?

Is your service similar to

?

7

unfortunately the source code is not available. I will try to make a minimal example soon.
I don’t think the problem is the same as for service-whoami.
Indeed, the logs during a user request may be a good lead.

15:17:33.286 [ConfigProxy] info: 201 POST /api/routes/services/myservice 
[I 2024-04-19 15:17:33.286 JupyterHub app:3601] JupyterHub is now running at http://:8000
[I 2024-04-19 15:17:33.883 JupyterHub log:192] 200 GET /hub/api/ (idle-culler@172.17.1.118) 37.52ms
[I 2024-04-19 15:17:33.957 JupyterHub log:192] 200 GET /hub/api/users?state=[secret] (idle-culler@172.17.1.118) 65.91ms
[I 2024-04-19 15:17:43.509 JupyterHub log:192] 200 GET /hub/app/projects (jupyter@172.17.1.136) 441.50ms
[I 2024-04-19 15:17:44.720 JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=service-myservice&redirect_uri=%2Fservices%2Fmyservice%2Foauth_callback&response_type=code&state=[secret] -> /services/myservice/oauth_callback?code=[secret]&state=[secret] (jupyter@172.17.1.136) 183.45ms
ERROR:tornado.application:Uncaught exception GET /services/myservice/oauth_callback?code=YhzrebNkw1ATobTcXHW6SwN5PVlgJ2&state=xY9jKvPwaYZu0hVvGtgaMQ (127.0.0.1)
HTTPServerRequest(protocol='http', host='jupyterapps.dev.intra.logilab.fr', method='GET', uri='/services/myservice/oauth_callback?code=YhzrebNkw1ATobTcXHW6SwN5PVlgJ2&state=xY9jKvPwaYZu0hVvGtgaMQ', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/tornado/web.py", line 1786, in _execute
    result = await result
  File "/usr/local/lib/python3.10/dist-packages/jupyterhub/services/auth.py", line 1539, in get
    self.hub_auth.clear_oauth_state_cookies(self)
  File "/usr/local/lib/python3.10/dist-packages/jupyterhub/services/auth.py", line 1205, in clear_oauth_state_cookies
    self._clear_cookie(
TypeError: HubOAuth._clear_cookie() missing 1 required positional argument: 'cookie_name'
ERROR:tornado.access:500 GET /services/myservice/oauth_callback?code=YhzrebNkw1ATobTcXHW6SwN5PVlgJ2&state=xY9jKvPwaYZu0hVvGtgaMQ (127.0.0.1) 6.40ms
WARNING:tornado.application:Detected unused OAuth state cookies
[I 2024-04-19 15:17:45.238 JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=service-myservice&redirect_uri=%2Fservices%2Fmyservice%2Foauth_callback&response_type=code&state=[secret] -> /services/myservice/oauth_callback?code=[secret]&state=[secret] (jupyter@172.17.1.136) 214.96ms
ERROR:tornado.application:Uncaught exception GET /services/myservice/oauth_callback?code=BKotn25aveRdxTUn2AjkjKL522PYDX&state=O4-bw5LBHNQam3R7ZkyWHA (127.0.0.1)
HTTPServerRequest(protocol='http', host='jupyterapps.dev.intra.logilab.fr', method='GET', uri='/services/myservice/oauth_callback?code=BKotn25aveRdxTUn2AjkjKL522PYDX&state=O4-bw5LBHNQam3R7ZkyWHA', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/tornado/web.py", line 1786, in _execute
    result = await result
  File "/usr/local/lib/python3.10/dist-packages/jupyterhub/services/auth.py", line 1539, in get
    self.hub_auth.clear_oauth_state_cookies(self)
  File "/usr/local/lib/python3.10/dist-packages/jupyterhub/services/auth.py", line 1205, in clear_oauth_state_cookies
    self._clear_cookie(
TypeError: HubOAuth._clear_cookie() missing 1 required positional argument: 'cookie_name'
ERROR:tornado.access:500 GET /services/myservice/oauth_callback?code=BKotn25aveRdxTUn2AjkjKL522PYDX&state=O4-bw5LBHNQam3R7ZkyWHA (127.0.0.1) 3.94ms

Thanks @mahendrapaipuri @manics for your help

8

Which version of JupyterHub are you using? The line numbers don’t match up with 4.1.5 nor main

9

Sory, You are right, I was on an intermediate commit. I based it on jupyterhub 4.1.5 and I no longer have the previous problem, however I still have my 404 error.

The real logs are:

[I 2024-04-22 07:31:48.432 JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=service-myservice&redirect_uri=%2Fservices%2Fmyservice%2Foauth_callback&response_type=code&state=[secret] -> /services/myservice/oauth_callback?code=[secret]&state=[secret] (jupyter@172.17.1.136) 121.69ms
[I 2024-04-22 07:31:48.632 JupyterHub log:192] 200 POST /hub/api/oauth2/token (myservice@172.17.1.136) 130.15ms
[I 2024-04-22 07:31:48.735 JupyterHub log:192] 200 GET /hub/api/user (jupyter@172.17.1.136) 66.91ms
[I 2024-04-22 07:31:49.252 JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=service-myservice&redirect_uri=%2Fservices%2Fmyservice%2Foauth_callback&response_type=code&state=[secret] -> /services/myservice/oauth_callback?code=[secret]&state=[secret] (jupyter@172.17.1.136) 120.83ms
[I 2024-04-22 07:31:49.532 JupyterHub log:192] 200 POST /hub/api/oauth2/token (myservice@172.17.1.136) 149.75ms
[I 2024-04-22 07:31:49.640 JupyterHub log:192] 200 GET /hub/api/user (jupyter@172.17.1.136) 71.43ms
[I 2024-04-22 07:31:49.945 JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=service-myservice&redirect_uri=%2Fservices%2Fmyservice%2Foauth_callback&response_type=code&state=[secret] -> /services/myservice/oauth_callback?code=[secret]&state=[secret] (jupyter@172.17.1.136) 150.79ms
[I 2024-04-22 07:31:50.171 JupyterHub log:192] 200 POST /hub/api/oauth2/token (myservice@172.17.1.136) 140.57ms
[I 2024-04-22 07:31:50.275 JupyterHub log:192] 200 GET /hub/api/user (jupyter@172.17.1.136) 63.11ms
[I 2024-04-22 07:31:51.247 JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=service-myservice&redirect_uri=%2Fservices%2Fmyservice%2Foauth_callback&response_type=code&state=[secret] -> /services/myservice/oauth_callback?code=[secret]&state=[secret] (jupyter@172.17.1.136) 120.30ms
[I 2024-04-22 07:31:51.560 JupyterHub log:192] 200 POST /hub/api/oauth2/token (myservice@172.17.1.136) 177.22ms
[I 2024-04-22 07:31:51.669 JupyterHub log:192] 200 GET /hub/api/user (jupyter@172.17.1.136) 69.93ms
[I 2024-04-22 07:31:51.916 JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=service-myservice&redirect_uri=%2Fservices%2Fmyservice%2Foauth_callback&response_type=code&state=[secret] -> /services/myservice/oauth_callback?code=[secret]&state=[secret] (jupyter@172.17.1.136) 128.33ms
[I 2024-04-22 07:31:52.234 JupyterHub log:192] 200 POST /hub/api/oauth2/token (myservice@172.17.1.136) 160.89ms
[I 2024-04-22 07:31:52.339 JupyterHub log:192] 200 GET /hub/api/user (jupyter@172.17.1.136) 73.68ms
[I 2024-04-22 07:31:52.893 JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=service-myservice&redirect_uri=%2Fservices%2Fmyservice%2Foauth_callback&response_type=code&state=[secret] -> /services/myservice/oauth_callback?code=[secret]&state=[secret] (jupyter@172.17.1.136) 107.03ms
[I 2024-04-22 07:31:53.194 JupyterHub log:192] 200 POST /hub/api/oauth2/token (myservice@172.17.1.136) 158.30ms
[I 2024-04-22 07:31:53.318 JupyterHub log:192] 200 GET /hub/api/user (jupyter@172.17.1.136) 75.62ms
[I 2024-04-22 07:31:53.592 JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=service-myservice&redirect_uri=%2Fservices%2Fmyservice%2Foauth_callback&response_type=code&state=[secret] -> /services/myservice/oauth_callback?code=[secret]&state=[secret] (jupyter@172.17.1.136) 124.12ms
[I 2024-04-22 07:31:53.895 JupyterHub log:192] 200 POST /hub/api/oauth2/token (myservice@172.17.1.136) 164.83ms
[I 2024-04-22 07:31:54.035 JupyterHub log:192] 200 GET /hub/api/user (jupyter@172.17.1.136) 93.52ms
[I 2024-04-22 07:31:55.850 JupyterHub log:192] 302 GET / -> /hub/ (@172.17.1.136) 2.44ms
[I 2024-04-22 07:31:55.868 JupyterHub log:192] 302 GET /hub/ -> /hub/app/ (@172.17.1.136) 2.34ms
[I 2024-04-22 07:31:55.938 JupyterHub log:192] 302 GET /hub/app/ -> /hub/login?next=%2Fhub%2Fapp%2F (@172.17.1.136) 2.43ms
[I 2024-04-22 07:31:56.005 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:yGBDL2Xy9dNs5TdxLRvT6C76_8ralnvf-ip8oaEKQ8U=' {'path': '/hub/', 'max_age': 3600}
[I 2024-04-22 07:31:56.014 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2Fapp%2F (@172.17.1.136) 9.71ms
[I 2024-04-22 07:32:06.997 JupyterHub log:192] 200 GET /hub/api/users (myservice@127.0.0.1) 38.38ms
WARNING:tornado.access:404 GET /services/myservice/ (127.0.0.1) 1.34ms
07:32:08.762 [ConfigProxy] info: 200 GET /api/routes
10

Can you try the JupyterHub 5.0.0b1 beta? [ANN] JupyterHub 5.0 beta

If that works we can try and work out why it doesn’t in 4.x- please feel free to open a bug report though we’ll probably need a minimal example of your service.

11

I have exactly the same error with JupyterHub 5.0 beta.
Thanks for your help, I will create a minimal example of my service.

1 Like
12

In firefox develment tools I see in console:

GET
https://jupyterapps.dev.intra.logilab.fr/services/myservice/oauth_callback?code=Y3JaJBIqHTWkImrdkgcAmqZRwyvPju&state=eyJ1dWlkIjogIjU2MWNlZDhhMjE3ZjQ4ZjM4ZTdiNDQ1OTI5MjQ0NzkxIiwgIm5leHRfdXJsIjogIi9zZXJ2aWNlcy9yZXBvMmltZy9hcGkvdXNlcnMvY3VycmVudCJ9

NS_ERROR_REDIRECT_LOOP
13

You could try the XSRF suggestions in

2 Likes
14

Thank @maniac this is the right direction.

with

class BaseHandler(HubOAuthenticated, web.RequestHandler):
    def check_xsrf_cookie(self):
        pass

The service works.

But self.headers is None, which probably explains the authentication problems.

Thank again, I’ll keep digging .