Debugging Jupyterhub Authentication

JupyterHub
, ,
1

I am trying to use a custom jupyterhub setup and really struggling. I am using jupyterhub 2.3.0 and I also have jupyterlab 3.4.5 installed on the user servers. I set this all up on a single compute instance, made sure everything worked and then distributed some of these parts around in containers.

I am using a custom authentication that is hardcoded to return 'test_user". That is all the auth does until I can isolate this.
I am using this for spawner: GitHub - uktrade/fargatespawner: Spawns JupyterHub single user servers in Docker containers running in AWS Fargate

When I was testing this on a single compute instance w/ local Spawner I saw the cookies and everything worked fine. When I have jupyterhub running in a separate container from the jupyterhub-singleuser instances, users can never access the server.
In the Jupyterhub log I see the server get created and the proxy create the routes:
User logged in: test_user
Not recording activity for status 302 on /jupyterhub
302 POST /jupyterhub/hub/login?next=/jupyterhub/hub → /jupyterhub/hub
PROXY WEB /jupyterhub/user/test_user to http://10.x.x.x/
PROXY WEB /jupyterhub/user/test_user /api/kernels to http://10.x.x.x/

Then in the single user jupyter logs I see basically this:
200 GET /jupyterhub/user/test_user
No user identified
Using services/contents
GET /jupyterhub/user/test_user

Any ideas on what the problem could be? I think the cookies are missing for some reason. They were working fine on a single node, do I need to handle that somewhere? Do I need to explicitly set user cookies even though I am using the jupyterhub-singleuser launch? I tried to set cookie_secret file to a shared location but jupyterhub-singleuser cookie_secret_file is not recognized so not too sure. I’d really appreciate your thoughts. Thanks!

2

For JupyterLab - I think you might need to use “jupyter-labhub” instead of the ordinary server. I don’t really know what jupyter-labhub is, but I think it’s some kind of wrapper around JupyterServer and JupyterLab that facilitates communication with the Hub. Although I don’t know what it is, it does appear to work well.

This is our working Dockerfile:

FROM jupyter/scipy-notebook:latest

USER root

RUN apt-get update -y

RUN apt-get -y install \
    net-tools \
    dnsutils

USER 1000

RUN pip3 install --no-cache \
    jupyterhub==2.3.1

CMD ["/opt/conda/bin/jupyter-labhub", "--port=8888", "--collaborative"]

The CMD isn’t actually necessary, as this can be specified as an override in the Spawner.

I installed net-tools and dnsutils for debugging - not needed. Installing jupyterhub 2.3.1 allows you to use the new authorization features.

1 Like
3

Can you provide pip freeze in your user and hub images? Actual logs from the single-user container would help a lot, too. There’s lots of useful info there that is missing from a summary.

JupyterHub doesn’t use shared-cookies for auth. The user arrives at the single-user server with no credentials, and the cookie for the server is set after completion of oauth with the Hub. If oauth doesn’t complete, a cookie on the /jupyterhub/user/test_user/ path shouldn’t be expected.