One of my teammates cloned an Ubuntu 16.04 server into 2 servers in Azure, and since then our Jupyter Hub has been having problems. If I connect to Jupyter Hub on http://localhost:8000/ (over an ssh tunnel) it says simply “Not found” and nothing more in the browser.
I -think- we’re using Jupyter Hub 1.0.0, but jupyterhub --version outputs that version-looking string, and then a SystemExit traceback. ISTR that’s considered benign.
Checking /var/log with a recursive grep didn’t prove informative.
If I strace the process listening on tcp/8000, I can see:
[pid 30739] read(20, “GET / HTTP/1.1\r\nHost: localhost:8000\r\nConnection: keep-alive\r\nPragma: no-cache\r\nCache-Control: no-cache\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36\r\nSec-Fetch-User: ?1\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3\r\nSec-Fetch-Site: none\r\nSec-Fetch-Mode: navigate\r\nAccept-Encoding: gzip, deflate, br\r\nAccept-Language: en-US,en;q=0.9\r\n\r\n”, 65536) = 521
[pid 30739] write(1, “00:18:40.279 [ConfigProxy] \33[31merror\33[39m: 404 GET / \n”, 55) = 55
[pid 30739] socket(PF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 22
[pid 30739] connect(22, {sa_family=AF_INET, sin_port=htons(8081), sin_addr=inet_addr(“127.0.0.1”)}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 30739] epoll_ctl(13, EPOLL_CTL_ADD, 22, {EPOLLOUT, {u32=22, u64=22}}) = 0
[pid 30739] epoll_wait(13, [{EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=22, u64=22}}], 1024, 719) = 1
[pid 30739] getsockopt(22, SOL_SOCKET, SO_ERROR, [111], [4]) = 0
[pid 30739] epoll_ctl(13, EPOLL_CTL_DEL, 22, 0x7fffe88bc610) = 0
[pid 30739] close(22) = 0
[pid 30739] write(1, “00:18:40.281 [ConfigProxy] \33[31merror\33[39m: Failed to get custom error page: Error: connect ECONNREFUSED 127.0.0.1:8081\n at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1129:14) {\n errno: ‘ECONNREFUSED’,\n code: ‘ECONNREFUSED’,\n syscall: ‘connect’,\n address: ‘127.0.0.1’,\n port: 8081\n}\n”, 300) = 300
[pid 30739] writev(20, [{“HTTP/1.1 404 Not Found\r\nDate: Wed, 11 Dec 2019 00:18:40 GMT\r\nConnection: keep-alive\r\nTransfer-Encoding: chunked\r\n\r\n9”, 116}, {“\r\n”, 2}, {“Not Found”, 9}, {“\r\n”, 2}, {“0\r\n\r\n”, 5}], 5) = 134
I googled for services that would listen on tcp/8081 (found nothing interesting), and I grep’d for 8081 in /etc/services (found tproxy). tproxy doesn’t feel very relevant, in a quick google.
What is this mysterious service that Jupyter Hub wants to connect to, and how can I start it?
Or more generally, if the tcp/8081 thing is a red herring: how can I get Jupyter Hub happy again?
Thanks!
