oauth-azure login's 'hang' after token is returned

I have jupyterhub set up on an azure server and working own port 22000.
I can get other authenticators to work but to make life easier for students I and am trying to get oauth-azure working.
At the moment when I try to log in it sends me through to our main systems for authentication.
Looking at their logs it seems the login attempts are successful.
However the process then just hangs and eventually returns a browser error saying the server can’t be reached.

(I’ve changed the actual domain name to MYDOMAIN and the code, star and session_state in the message)

Safari can’t open the page “https://MYDOMAIN:22000/hub/oauth_callback?code=AAAAA&state=BBBBBB&session_state=CCCCCC” because the server where this page is located isn’t responding.

Help needed please: How can I find out where the process is stalled?

Notes:

Our systems are set up with the callback_url set to my https://MYDOMAIN:22000
(MYDOMAIN is set to the appropriate url with a redirect to the IP address as well).

I’ve registered my domain and installed the certificates, and checked with different authenticators that the https part works fine.

jupyter hub is installed in /opt/jupyterhub following these instructions:
https://jupyterhub.readthedocs.io/en/stable/installation-guide-hard.html
and works with the native authenticator and the 'dummy authenticator.

it is configured to listen on port 22000 and with the azure config options hard coded using. these Ines in the jupyterhub_config.py:
c.JupyterHub.bind_url = ‘https://:22000’

from oauthenticator.azuread import AzureAdOAuthenticator
c.JupyterHub.authenticator_class = AzureAdOAuthenticator

c.Application.log_level = ‘DEBUG’

c.AzureAdOAuthenticator.tenant_id = ‘XXXXX’ #os.environ.get(‘AAD_TENANT_ID’)

I’ve hard coded this rather than exporting the environment variable outside this file

c.AzureAdOAuthenticator.oauth_callback_url = ‘https://MYDOMAIN:22000/hub/oauth_callback
c.AzureAdOAuthenticator.client_id = ‘YYYYYYY’ #’{AAD-APP-CLIENT_ID}’
c.AzureAdOAuthenticator.client_secret = ‘ZZZZZZZ’ # ‘{AAD-APP-CLIENT-SECRET}’

I’ve tried setting the AAD_TENANT variable in the script that registers the service and also running running from the command line, prefixed by sudo and explicitly setting those three environments variables but that makes no difference.

Could you try turning on debug logging in JupyterHub, and show us the logs during and before the attempt to login?

starting from the command line:
sudo /opt/jupyterhub/bin/jupyterhub --debug -f /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py

[D 2021-01-26 11:12:33.353 JupyterHub application:730] Looking for /opt/jupyterhub/etc/jupyterhub/jupyterhub_config in /opt/jupyterhub

[D 2021-01-26 11:12:33.354 JupyterHub application:752] Loaded config file: /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py

[I 2021-01-26 11:12:33.355 JupyterHub app:2349] Running JupyterHub version 1.3.0

[I 2021-01-26 11:12:33.355 JupyterHub app:2379] Using Authenticator: oauthenticator.azuread.AzureAdOAuthenticator-0.12.3

[I 2021-01-26 11:12:33.355 JupyterHub app:2379] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-1.3.0

[I 2021-01-26 11:12:33.355 JupyterHub app:2379] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.3.0

[D 2021-01-26 11:12:33.356 JupyterHub app:2310] Could not load pycurl: No module named ‘pycurl’

pycurl is recommended if you have a large number of users.

[D 2021-01-26 11:12:33.357 JupyterHub app:1460] Generating new cookie_secret

[I 2021-01-26 11:12:33.357 JupyterHub app:1465] Writing cookie_secret to /opt/jupyterhub/jupyterhub_cookie_secret

[D 2021-01-26 11:12:33.357 JupyterHub app:1587] Connecting to db: sqlite:///jupyterhub.sqlite

[D 2021-01-26 11:12:33.367 JupyterHub orm:784] Stamping empty database with alembic revision 4dc2d5a8c53c

[I 2021-01-26 11:12:33.371 alembic.runtime.migration migration:155] Context impl SQLiteImpl.

[I 2021-01-26 11:12:33.371 alembic.runtime.migration migration:158] Will assume non-transactional DDL.

[I 2021-01-26 11:12:33.399 alembic.runtime.migration migration:517] Running stamp_revision -> 4dc2d5a8c53c

[D 2021-01-26 11:12:33.399 alembic.runtime.migration migration:727] new branch insert 4dc2d5a8c53c

[I 2021-01-26 11:12:33.560 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN

[I 2021-01-26 11:12:33.584 JupyterHub app:1725] Not using allowed_users. Any authenticated user will be allowed.

[D 2021-01-26 11:12:33.611 JupyterHub app:1877] Purging expired APITokens

[D 2021-01-26 11:12:33.613 JupyterHub app:1877] Purging expired OAuthAccessTokens

[D 2021-01-26 11:12:33.614 JupyterHub app:1877] Purging expired OAuthCodes

[D 2021-01-26 11:12:33.623 JupyterHub app:2004] Initializing spawners

[D 2021-01-26 11:12:33.624 JupyterHub app:2137] Loaded users:

[I 2021-01-26 11:12:33.625 JupyterHub app:2416] Initialized 0 spawners in 0.002 seconds

[I 2021-01-26 11:12:33.627 JupyterHub proxy:666] Starting proxy @ https://:22000

[D 2021-01-26 11:12:33.627 JupyterHub proxy:667] Proxy cmd: [‘configurable-http-proxy’, ‘–ip’, ‘’, ‘–port’, ‘22000’, ‘–api-ip’, ‘127.0.0.1’, ‘–api-port’, ‘8001’, ‘–error-target’, ‘http://127.0.0.1:8081/hub/error’, ‘–ssl-key’, ‘/etc/letsencrypt/live/csctcloud.uwe.ac.uk/privkey.pem’, ‘–ssl-cert’, ‘/etc/letsencrypt/live/csctcloud.uwe.ac.uk/fullchain.pem’]

[D 2021-01-26 11:12:33.629 JupyterHub proxy:574] Writing proxy pid file: jupyterhub-proxy.pid

11:12:34.051 [ConfigProxy] info: Proxying https://*:22000 to (no default)

11:12:34.053 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes

[D 2021-01-26 11:12:34.198 JupyterHub proxy:702] Proxy started and appears to be up

[D 2021-01-26 11:12:34.201 JupyterHub proxy:795] Proxy: Fetching GET http://127.0.0.1:8001/api/routes

11:12:34.207 [ConfigProxy] info: 200 GET /api/routes

[I 2021-01-26 11:12:34.208 JupyterHub app:2664] Hub API listening on http://127.0.0.1:8081/hub/

[D 2021-01-26 11:12:34.208 JupyterHub proxy:314] Fetching routes to check

[D 2021-01-26 11:12:34.208 JupyterHub proxy:795] Proxy: Fetching GET http://127.0.0.1:8001/api/routes

11:12:34.210 [ConfigProxy] info: 200 GET /api/routes

[I 2021-01-26 11:12:34.210 JupyterHub proxy:319] Checking routes

[I 2021-01-26 11:12:34.210 JupyterHub proxy:399] Adding default route for Hub: / => http://127.0.0.1:8081

[D 2021-01-26 11:12:34.210 JupyterHub proxy:795] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/

11:12:34.212 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081

11:12:34.212 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081

11:12:34.213 [ConfigProxy] info: 201 POST /api/routes/

[I 2021-01-26 11:12:34.213 JupyterHub app:2739] JupyterHub is now running at https://:22000

[D 2021-01-26 11:12:34.213 JupyterHub app:2342] It took 0.865 seconds for the Hub to start

then trying to connect via a browser pointed str localhost:22000 (there’s a ssh tunnel from my local port 22000 t port 22000 on the server) generates this extra logging (I’ve removed the specific client and tenth ids and domain from the GET command and replaced them with “MYxxx”

[I 2021-01-26 11:14:27.128 JupyterHub log:181] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:127.0.0.1) 23.03ms

[I 2021-01-26 11:14:28.439 JupyterHub log:181] 302 GET / -> /hub/ (@::ffff:127.0.0.1) 0.78ms

[I 2021-01-26 11:14:28.524 JupyterHub log:181] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:127.0.0.1) 0.64ms

[I 2021-01-26 11:14:28.602 JupyterHub log:181] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:127.0.0.1) 1.11ms

[I 2021-01-26 11:14:35.033 JupyterHub oauth2:104] OAuth redirect: ‘https://csctcloud.uwe.ac.uk:22000/hub/oauth_callback

[D 2021-01-26 11:14:35.034 JupyterHub base:521] Setting cookie oauthenticator-state: {‘httponly’: True, ‘secure’: True, ‘expires_days’: 1}

[I 2021-01-26 11:14:35.034 JupyterHub log:181] 302 GET /hub/oauth_login?next=%2Fhub%2F -> Sign in to your account (@::ffff:127.0.0.1) 1.41ms

Could that be the problem? The hub is trying to redirect you to

but since you’re connected through a tunnel your browser is unable to access that host:port.

Hi,
thanks for getting back to me.
I think you are correct- or at least if we open that port directly to the internet, and connect to it, then we get a different bunch of problems around local certificate issuers.

Do you happen to know if there is a way of configuring what our ITS people would like:
users create ssh tunnel to azure VM which then runs oauth talking to a different machine?

If everyone is always connecting in the same way (port forwarding to localhost:2222) you might be able to configure the hub oauth callback url to http://localhost:22000/hub/oauth_callback. This would have to be done in the jupyterhub config and probably also in the Azure application settings.

Another alttenative is to use ssh to provide a SOCKS proxy for your browser, effectively giving you a basic VPN, see for example

Thanks very much for the help and pointers.