Jupyter(hub) notebooks are not showing any output

[scott-anderson-wc]

This symptoms are pretty much the same as: https://discourse.jupyter.org/t/help-my-jupyter-notebook-is-not-showing-any-outputs/8122/17, but I am using a brand-new installation of Jupyterhub (which I did, thanks to help from this forum), so the trouble is probably due to a mistake in my installation.

I read that other post, and the test notebook does have expressions that evaluate to something, such as 3+4. I’ve restarted the kernel. In Chrome, the kernel seems to say “initializing” for quite some time and then says “disconnected”. Here’s a screenshot of the browser:

I don’t see any errors in the logs (took me some effort to set up logging and maybe they are empty because they aren’t set up correctly). The “journalctl -u jupyterhub” says:

Aug 26 15:39:08 cs.wellesley.edu systemd[1]: Started JupyterHub.
Aug 26 15:39:58 cs.wellesley.edu sudo[684484]: jupyterhub : PWD=/home/jupyterhub ; USER=cs299 ; COMMAND=/home/jupyterhub/env39/bin/sudospawner

And that’s it. It just doesn’t look like anything is executing. In fact, I looked at the raw first.ipynb file and the cells look like:

"cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65d13271-2ffb-4e8a-a30b-59826f0bfd62",
   "metadata": {},
   "outputs": [],
   "source": [
    "4+5"
   ]
  },

so the execution_count being null seems bad.

But the spawning seems to be successful. After logging in as “cs299”, I see the following processes have been created:

cs299     684487       1  0 15:39 ?        00:00:00 /usr/lib/systemd/systemd --user
cs299     684489  684487  0 15:39 ?        00:00:00 (sd-pam)
cs299     684506       1  0 15:39 ?        00:00:00 /home/jupyterhub/env39/bin/python /home/jupyterhub/env39/bin/sudospawner
cs299     684507  684506  0 15:39 ?        00:00:06 /home/jupyterhub/env39/bin/python /home/jupyterhub/env39/bin/jupyterhub-singleuser
cs299     684584  684507  0 15:40 ?        00:00:00 /home/jupyterhub/env39/bin/python -m ipykernel_launcher -f /home/cs299/.local/share/jupyter/runtime/kernel-3828591a-fcf4-4bd3-9a1b-945cc9601ac9.json

What have I missed?

Thanks,

Scott Anderson

[manics]

Can you share your browser console logs?

If you see any websocket errors this usually means there’s an incorrectly configured proxy, or a firewall/antivirus scanner, that’s blocking websocket connections.

[scott-anderson-wc]

Thanks for responding. I’m happy to share my browser console logs. Most of the connections are successful:

The one that doesn’t succeed says:

Firefox can’t establish a connection to the server at wss://cs.wellesley.edu/jhub/user/cs299/api/kernels/3828591a-fcf4-4bd3-9a1b-945cc9601ac9/channels?session_id=e680b4e2-2e08-4269-ab33-ab63c6f84a6c.

That doesn’t mean anything to me. What do we need to look at next?

I did notice that “wss:” means an encrypted socket connection. I haven’t installed any security certificates in Jupyterhub. I’m letting Apache handle the SSL, and using a reverse proxy to transfer to Jupyterhub. Could encryption be the issue?

Thanks,

Scott

[manics]

Assuming your certiticates are correctly installed it shouldn’t be a problem.

It sounds like something (e.g. Apache, firewall, etc) between JupyterHub and your browser is blocking websocket connections:

Troubleshooting

This is outside the control of JupyterHub.

[scott-anderson-wc]

Thanks for your followup. I was testing from outside the firewall, so I started the VPN and tried again. I’m afraid I got the same result. Apparently WSS uses port 443, which wouldn’t have been blocked by the firewall anyhow. I looked for my IP in the Apache logs, and I saw only the successes. The error_log did not seem to have any requests from the Jupyterhub client.

I looked in the Network tab of the browser dev tools, and the failed request says:

Status 400 Bad Request
Version HTTP/1.1
Transferred 302 B (0 B size)
DNS Resolution System

So, that doesn’t look like a firewall issue (though I could be wrong).

I double-checked the Apache config. I have the following:

# Because /jhub is easier to spell and type than jupyterhub
# Following code copied from
# https://jupyterhub.readthedocs.io/en/latest/howto/configuration/config-proxy.html
# changed the port from 8000 to 9389
RewriteRule /jhub/(.*) ws://127.0.0.1:9389/jhub/$1 [P,L]
RewriteRule /jhub/(.*) http://127.0.0.1:9389/jhub/$1 [P,L]

ProxyPass /jhub/ http://127.0.0.1:9389/jhub/
ProxyPassReverse /jhub/  http://127.0.0.1:9389/jhub/

Should that ws above actually be wss? I tried changing it to wss but the same error persisted.

It should be possible to find out why something was blocked if it was blocked. And (thinking about the http 400 error) the browser shouldn’t be creating bad requests. And the server should report, somewhere, the receipt of a bad request and why it was considered bad. The two logfiles I set up have had nothing written to them since yesterday. Is there someplace else I should look for logfiles? Something user-specific? I looked in the user’s account and I don’t see any logfiles there.

Following https://jupyterhub.readthedocs.io/en/latest/faq/troubleshooting.html#troubleshooting-commands I added --debug to the main command, but I’m not seeing any additional debug output anywhere.

Thanks again,

Scott

[manics]

Can you try connecting to JupyterHub directly, without encryption, bypassing Apache? That will help narrow down where the problem is.

Note that firewalls don’t only block ports, more advanced ones can block selected protocols such as websockets.

[scott-anderson-wc]

I tried your suggestion of connecting to Jupyterhub directly. My browsers (Firefox and Chrome) both failed, but Firefox I know was insisting on HTTPS and I think Chrome did the same. Rather than go down a rabbit hole of trying to convince them to make an unencrypted connection, I set up an SSH tunnel to the port that Jupyterhub is listening on. That also failed but in an interesting way. I clearly got a response from Jupyterhub:

Then, just to be sure I hadn’t messed up the SSH tunnel (though I’ve done that in other contexts many times), I tried using GET and Curl to access the page. And I get the same thing! Namely, I get the Jupyterhub response page that says:

  <div class="error">
         <h1>404 : Not Found</h1>
    <p>Jupyter has lots of moons, but this is not one...</p>
 </div>

9389 is the correct port (I think) because the config file has:

c.JupyterHub.bind_url = 'http://0.0.0.0:9389/jhub/'

Netstat shows:

netstat -ntlp | grep 9389
tcp        0      0 0.0.0.0:9389            0.0.0.0:*               LISTEN      1013972/python      

and that process is:

ps -f 1013972
UID          PID    PPID  C STIME TTY      STAT   TIME CMD
jupyter+ 1013972 1013964  0 11:04 ?        Ssl    0:00 /home/jupyterhub/env39/bin/python /home/jupyterhub/env39/bin/configurable-http-proxy --ip 0.0.0.0 --port 9389 --api-ip 127.0.0.1 --api-port 8001 

So, I’m not sure what, if anything, I’m doing wrong. I double-checked that connecting via the reverse proxy still works, and it does (though the cells are still not evaluated).

I appreciate your help so far, but I still need more.

Thanks,

Scott

PS: Thanks for the info about firewalls. My experiments above were inside the campus firewall, so that should not be an issue. And I also shut off the server’s firewall for the experiments above:

systemctl stop firewalld.service
firewall-cmd --state
not running

[manics]

You’ve specified a /jhub prefix for JupyterHub, so you’ll need to go to http://localhost:PORT/jhub/

If for some reason that doesn’t work try a minimal jupyterhub_config.py file with no prefix or other customisations.

[scott-anderson-wc]

D’oh! I forgot about the /jhub. Thanks. Having done that, I can report wild success. I was able to login (different user this time, just to be sure there wasn’t a conflict) and started a notebook and cells are evaluated:

So, I guess this tells us that there’s something involved with the handoff from Apache that is messing things up. The firewall is still off. What else can I check?

Thanks for your patience!

Scott

PS: I repeated the experiment of talking directly to Jupyterhub but with the firewall running and it still worked, so I think the firewall is not the culprit.

PPS: I looked in the web console for the working version, and I saw this message:

Starting WebSocket: ws://localhost:8080/jhub/user/cs304node/api/kernels/73e49043-52d7-4246-a9d3-960a56ed3ca1

comparing that to the earlier connection that failed:

Firefox can’t establish a connection to the server at wss://cs.wellesley.edu/jhub/user/cs299/api/kernels/3828591a-fcf4-4bd3-9a1b-945cc9601ac9/channels?session_id=e680b4e2-2e08-4269-ab33-ab63c6f84a6c.

I notice the change in the protocol (ws: versus wss:), the change in the hostname:port (localhost:8080 is the SSH tunnel I set up to the jupyterhub server). But the path is entirely different, so maybe I’m comparing the wrong two URLs. The current reverse proxy rewrite rules are:

RewriteRule /jhub/(.*) wss://127.0.0.1:9389/jhub/$1 [P,L]
RewriteRule /jhub/(.*) http://127.0.0.1:9389/jhub/$1 [P,L]

[manics]

I don’t have much experience with Apache, but the example in

Using a reverse proxy

has

  # Use RewriteEngine to handle WebSocket connection upgrades
  RewriteEngine On
  RewriteCond %{HTTP:Connection} Upgrade [NC]
  RewriteCond %{HTTP:Upgrade} websocket [NC]
  RewriteRule /(.*) ws://127.0.0.1:8000/$1 [P,L]

[scott-anderson-wc]

Thanks. I did indeed miss those lines of Apache configuration. My bad. I added them, without success.

You mentioned I should try again with a more default configuration, so I did that. I’ve put the change of port on hold. My current jupyterhub_config.py file differs from the default in only a few ways:

diff jupyterhub_config.py jupyterhub_config.py.orig 
193d192
< c.JupyterHub.bind_url = 'http://127.0.0.1:8000/jhub/'
809d807
< c.JupyterHub.spawner_class = 'sudospawner.SudoSpawner'
1553d1550
< c.Authenticator.allow_all = True
1845d1841
< c.SudoSpawner.sudospawner_path = '/home/jupyterhub/env39/bin/sudospawner'

The first line is copied directly from https://jupyterhub.readthedocs.io/en/latest/howto/configuration/config-proxy.html#apache

And my Apache conf file is now the following:

RewriteEngine On
RewriteCond %{HTTP:Connection} Upgrade [NC]
RewriteCond %{HTTP:Upgrade} websocket [NC]

ProxyPreserveHost on
RequestHeader     set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}

RewriteRule /jhub/(.*) ws://127.0.0.1:8000/jhub/$1 [P,L]
RewriteRule /jhub/(.*) http://127.0.0.1:8000/jhub/$1 [P,L]

ProxyPass /jhub/ http://127.0.0.1:8000/jhub/
ProxyPassReverse /jhub/  http://127.0.0.1:8000/jhub/

I believe those are exactly from the docs. I’ve checked and double-checked. I shut the firewall off, restarted Apache, and tried again. The error is identical:

Firefox can’t establish a connection to the server at 
wss://cs.wellesley.edu/jhub/user/cs304flask/api/kernels/c0823302-4e4c-4d54-b265-3c4026c1a90a/channels?session_id=4edf80fd-42fc-4c23-9380-8a9b1062c5e9.

I note again that the error is for wss:// and the rewrite rules are for ws://; could that difference be the problem? I tried changing that rewrite rule to specify wss:// but no luck.

Let me back up a bit. Am I trying to install Jupyterhub in a non-standard way? I thought it was designed to allow multiple users on a single system to use Jupyter Notebooks, which is exactly what I’m trying to do. Is there a more tried-and-true way of achieving my goal? I don’t mind starting all over from scratch if there’s a well-trodden path to success. Right now, it feels like I’m way off the beaten path.

Thanks again,

Scott

[manics]

@scott-anderson-wc

I note again that the error is for wss:// and the rewrite rules are for ws://; could that difference be the problem?

No. The encrypted websocket should be handled by Apache, which proxies it as an unencrypted websocket to JupyterHub, so it still sounds like Apache is the problem.

@scott-anderson-wc

Let me back up a bit. Am I trying to install Jupyterhub in a non-standard way?

According to your earlier post JupyterHub was working fine, and it’s the Apache proxy that’s the problem. If you think JupyterHub isn’t working then let’s go back and fix it before dealing with the proxy!

I’ve tested JupyterHub with an Apache proxy in Docker, running JupyterHub on a custom port 9000 and prefix /jhub, using a self-signed SSL certificate in Apache, and everything works fine for me, JupyterHub is acessible on https://localhost:8443/jhub/:

https://gist.github.com/manics/9fcfcd545e328ad1f3d4008a0aa7a2ce

Can you try that httpd.conf and disable all other Apache configuration? If that works then try adding the rest of your configuration back one bit at a time.

[scott-anderson-wc]

Thanks for that suggestion. The server is actively used for lots of stuff, so I can’t really disable all other Apache configuration. However, I’ll talk to the IT folks and see if I can temporarily get an identical spare server and start with a bare Apache configuration.

Thanks again!

Scott