Jupyterhub with dockerspawner; error 404 when spawning single-user container

Dear all,
I have deployed jupyterhub with docker-compose setup described https://github.com/jupyterhub/jupyterhub-deploy-docker.
On two servers where I used PAM authentication, I have no problem. On a new server, I needed to use keycloak for authentication with GenericOAuthenticator. On this new server also authentication works fine, but after authentication I get 404 Error on spawning of the single-user image. I have tried multiple single images. In all cases, I get the following log for my single-user image.

[I 00:37:01.039 NotebookApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 00:37:01.041 NotebookApp] Serving notebooks from local directory: /home/pyiron
[I 00:37:01.042 NotebookApp] Jupyter Notebook 6.1.4 is running at:
[I 00:37:01.042 NotebookApp] http://5137103405f0:8888/?token=9c4321f5e60c9ebe56e54d4eee89351ad3d5c432a1abe9d1
[I 00:37:01.042 NotebookApp]  or http://127.0.0.1:8888/?token=9c4321f5e60c9ebe56e54d4eee89351ad3d5c432a1abe9d1
[I 00:37:01.042 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 00:37:01.044 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/pyiron/.local/share/jupyter/runtime/nbserver-6-open.html
    Or copy and paste one of these URLs:
        http://5137103405f0:8888/?token=9c4321f5e60c9ebe56e54d4eee89351ad3d5c432a1abe9d1
     or http://127.0.0.1:8888/?token=9c4321f5e60c9ebe56e54d4eee89351ad3d5c432a1abe9d1
[I 00:37:01.046 NotebookApp] 302 GET /user/muhammad/ (172.17.0.5) 0.63ms
[I 00:37:01.366 NotebookApp] 302 GET /user/muhammad/ (172.17.0.5) 0.50ms
[W 00:37:01.587 NotebookApp] 404 GET /user/muhammad (172.17.0.5) 18.40ms 

Has anyone any idea, what the reason for such an error could be?

Hey, @mhassani we ran into this too back in the day, try using 0.0.0.0 instead of 127.0.0.1 in your configs. Since the Notebook runs in a container the localhost address doesn’t resolve to the IPv4 address that corresponds to the docker container. Another trick for testing is to remove token auth with the NotebookApp.token='' setting. So if you test locally with docker run ... it would look something like docker run -d -p 8888:8888 jupyter/scipy-notebook:latest jupyter notebook --ip=.0.0.0.0 --port=8888 NotebookApp.token=''. With the JupyterHub it’s probably best to set up the configs using DockerSpawner instead of overriding the upstream default configs provided by the jupyter/docker-stacks images. I’m not as much of an expert or as experienced as the official maintaners, but it worked for us. Cheers.

Unfortunately, setting c.Jupyterhub.ip=‘0.0.0.0’ did not help. Is this what you meant?
I am actually using DockerSpawner with one of the images from jupyter/docker-stack. But still I am getting 404 error.
Here is a part of jupyterhub log:

[I 2020-11-23 09:32:06.523 JupyterHub log:181] 302 GET /static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 -> /hub/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@192.12.81.1) 0.75ms
[W 2020-11-23 09:32:06.692 JupyterHub log:181] 404 GET /hub/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@192.12.81.1) 0.46ms
[I 2020-11-23 09:32:06.859 JupyterHub log:181] 302 GET /static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec -> /hub/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@192.12.81.1) 0.72ms
[W 2020-11-23 09:32:07.028 JupyterHub log:181] 404 GET /hub/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@192.12.81.1) 0.60ms
[I 2020-11-23 09:32:07.200 JupyterHub log:181] 302 GET /static/components/jquery/jquery.min.js?v=20201123003745 -> /hub/static/components/jquery/jquery.min.js?v=20201123003745 (@192.12.81.1) 0.58ms
[I 2020-11-23 09:32:07.218 JupyterHub log:181] 302 GET /static/base/images/logo.png?v=a2a176ee3cee251ffddf5fa21fe8e43727a9e5f87a06f9c91ad7b776d9e9d3d5e0159c16cc188a3965e00375fb4bc336c16067c688f5040c0c2d4bfdb852a9e4 -> /hub/static/base/images/logo.png?v=a2a176ee3cee251ffddf5fa21fe8e43727a9e5f87a06f9c91ad7b776d9e9d3d5e0159c16cc188a3965e00375fb4bc336c16067c688f5040c0c2d4bfdb852a9e4 (@192.12.81.1) 0.77ms
[W 2020-11-23 09:32:07.380 JupyterHub log:181] 404 GET /hub/static/components/jquery/jquery.min.js?v=20201123003745 (@192.12.81.1) 0.74ms
[W 2020-11-23 09:32:07.400 JupyterHub log:181] 404 GET /hub/static/base/images/logo.png?v=a2a176ee3cee251ffddf5fa21fe8e43727a9e5f87a06f9c91ad7b776d9e9d3d5e0159c16cc188a3965e00375fb4bc336c16067c688f5040c0c2d4bfdb852a9e4 (@192.12.81.1) 0.74ms
[I 2020-11-23 09:32:13.061 JupyterHub log:181] 302 GET /tree -> /hub/tree (@192.12.81.1) 0.80ms
[W 2020-11-23 09:32:13.233 JupyterHub log:181] 404 GET /hub/tree (muhammad@192.12.81.1) 2.35ms
[D 2020-11-23 09:32:13.511 JupyterHub log:181] 304 GET /hub/logo (@192.12.81.1) 0.81ms
[D 2020-11-23 09:32:14.843 JupyterHub dockerspawner:777] Getting container 'jupyter-muhammad'
[D 2020-11-23 09:32:14.846 JupyterHub dockerspawner:762] Container 9de76ef status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '0001-01-01T00:00:00Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 442854,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2020-11-23T00:37:44.827562899Z',
     'Status': 'running'}

Looks to me like your end-user container is starting up ok, the problem looks like a path issue with some customized react components. But that’s kind of a swag.

In jupyterhub_config.py, for authenticator class I changed from GenericOAuthenticator, which were set to get token from keycloak, to PAM authenticator as a test. With PAM single-user container were spawned. Therefore, I assume my configuration for keycloak oauth has not been correct. In below, you can find the details of the config:

c.JupyterHub.authenticator_class = 'oauthenticator.generic.GenericOAuthenticator'
c.OAuthenticator.oauth_callback_url ='https://example.com/hub/oauth_callback'
c.OAuthenticator.client_id='some_name'
c.OAuthenticator.client_secret = 'some_key'
c.GenericOAuthenticator.token_url ='https://sso.example.com/auth/realms/some_realm/protocol/openid-connect/token'
c.GenericOAuthenticator.userdata_url ='https://sso.example.com/auth/realms/some_realm/protocol/openid-connect/userinfo'
c.GenericOAuthenticator.userdata_method = 'GET'
c.GenericOAuthenticator.userdata_params = {"state": "state"}
#c.GenericOAuthenticator.username_key = 'sub'
c.LocalAuthenticator.create_system_users = True
c.GenericOAuthenticator.username_key = 'preferred_username'

plus the two environmental variable:

ENV OAUTH2_TOKEN_URL https://sso.example.com/auth/realms/some_realm/protocol/openid-connect/token
ENV OAUTH2_AUTHORIZE_URL https://sso.example.com/auth/realms/some_realm/protocol/openid-connect/auth

I would be grateful if anyone can help me to correct the above configuration for keycloak oauth?

There should be no requests to /static/anything, that means a misconfigured extension or base_url in the single-user server. I’m still trying to understand how this can happen.