Unable to connect consistently from Macs

I manage a JupyterHub instance running on CentOS on a local multi-core VM, which is used a few months of the year. I’ve just applied the latest OS and JupyterHub updates in preparation for the upcoming term, and Mac clients are no longer able to connect. (Since both I and the course leader use a Mac, we thought it was completely broken, until we contacted our IT support, who could access it fine from Windows.)

Access from Windows and Linux works as normal. Access from Mac (not browser specific—this includes from Safari, Firefox, and Brave) gives intermittent connection resets and timeouts. It is possible to get to the login page with repeated refreshing, but after logging in the JupyterLab is unusable as it claims that the server is not running (presumably because the JavaScript running on the client side tries to talk to the server and gets a timeout or dropped connection rather than the heartbeat response it expects).

The only clue I have is that the Mac connection looks like the following in logs:

[W 2022-02-04 15:35:37.888 SingleUserLabApp log:189] 403 GET /user/ed/api/sessions?XXX (@XXX) 1.12ms
[W 2022-02-04 15:36:14.786 SingleUserLabApp handlers:542] Forbidden
[W 2022-02-04 15:36:14.787 SingleUserLabApp log:189] 403 GET /user/ed/api/sessions?XXX (@XXX) 1.05ms
[W 2022-02-04 15:36:38.985 SingleUserLabApp handlers:542] Forbidden
[W 2022-02-04 15:36:38.985 SingleUserLabApp log:189] 403 GET /user/ed/api/kernels?XXX (@XXX) 1.08ms
[W 2022-02-04 15:36:38.996 SingleUserLabApp handlers:542] Forbidden
[W 2022-02-04 15:36:38.997 SingleUserLabApp log:189] 403 GET /user/ed/api/sessions?XXX (@XXX) 0.74ms
[W 2022-02-04 15:36:51.069 SingleUserLabApp handlers:542] Forbidden
[W 2022-02-04 15:36:51.069 SingleUserLabApp log:189] 403 GET /user/ed/api/sessions?XXX (@XXX) 1.06ms
[W 2022-02-04 15:37:51.546 SingleUserLabApp handlers:542] Forbidden
[W 2022-02-04 15:37:51.546 SingleUserLabApp log:189] 403 GET /user/ed/api/kernelspecs?XXX (@XXX) 1.06ms
[W 2022-02-04 15:37:59.595 SingleUserLabApp handlers:542] Forbidden
[W 2022-02-04 15:37:59.595 SingleUserLabApp log:189] 403 GET /user/ed/api/sessions?XXX (@XXX) 1.08ms

Successful connections from other OSes do not have the lines ending Forbidden. (Does this refer to line 542 of handlers/base.py? On my installation this line is the docstring for _set_cookie, which doesn’t seem relevant.)

Has anyone else seen this? And/or, does anyone have any tips on how to debug/fix this, please?

Many thanks

If everything’s working fine on Windows and Linux I suspect it’s a problem with Macs rather than JupyterHub.

Is your Mac managed by your IT department or is it your own, and does it have any anti-virus/firewalling/proxying configured? Have you changed any certificates recently? Do you see network problems when connecting with making requests with e.g. curl? If you don’t can you show us your browser console logs?

Many thanks for your reply. I was sure that because I’d seen the issue with three different Macs, one of which not mine, that it was common to all Macs, but I’ve just tried my partner’s Mac and indeed it doesn’t have a problem. (I also tested my iPhone, which also has the same problem.)

No antivirus or firewalling beyond what is built into macOS. I’ve tried via a SOCKS5 proxy via a machine on the same network as the VM, but also see the problem without any proxy. I renewed the TLS certificate from LetsEncrypt a month or two ago, but didn’t test the deployment thoroughly afterwards.

My organisation doesn’t specifically manage Macs in the same way it does Windows devices, but the Macs in question are all signed into the Exchange server in Mail (and Calendar, and possibly other locations as well) so it’s possible that that is having some effect maybe.

In curl, the first request succeeds (and gets a 302 to /hub/), then the next request times out. Ctrl+Cing and repeating times out, so it seems like there is some kind of filtering on repeated requests—either to the host, or this specific port. But only from certain Macs, for some reason.

I’m not seeing this in any software other than JupyterHub, so I assumed it was a weird interaction between Macs and JupyterHub, but then I’m also not connecting to HTTP services on this VM infrastructure for any other software. I’ll dig some more and see if I can find something more specific.

Thanks again.