JupyterHubSingleUser - Failed to connect to my Hub at http://127.0.0.1:8081/hub/api

krishnav · November 20, 2023, 12:17pm

After updating the base notebook image to Quay (base-notebook:2023-10-31) getting these connection errors in single user server spawn. Earlier used version jupyter/base-notebook:2023-04-14 this working without any errors. (python3.10 version used in working docker image)

Environment:
Used helm chart version “3.0.3” to deploy the 4.0.2 version of application on AWS EKS Cluster. hub and proxy pods runs on same EC2 Node. Single user server pod runs on different node. However, all core and user server pods in same namespace. (This is same architecture in working image as well)

Python 3.11 (updated python version to 3.11 in new docker image)

Logs:
notebook [I 2023-11-20 07:00:26.945 ServerApp] jupyter_server_terminals | extension was successfully linked.
notebook [I 2023-11-20 07:00:26.946 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 4.0.2
notebook [I 2023-11-20 07:00:26.946 JupyterHubSingleUser] Using default url from environment $JUPYTERHUB_DEFAULT_URL: /lab
notebook [I 2023-11-20 07:00:29.849 ServerApp] jupyter_server_terminals | extension was successfully loaded.
notebook [E 2023-11-20 07:00:29.855 JupyterHubSingleUser] Failed to connect to my Hub at http://127.0.0.1:8081/hub/api (attempt 1/5). Is it running?
notebook Traceback (most recent call last):
notebook File “/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/extension.py”, line 336, in check_hub_version
notebook resp = await client.fetch(self.hub_auth.api_url)
notebook ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
notebook ConnectionRefusedError: [Errno 111] Connection refused
notebook [I 2023-11-20 07:00:31.774 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
notebook [E 2023-11-20 07:00:31.862 JupyterHubSingleUser] Failed to connect to my Hub at http://127.0.0.1:8081/hub/api (attempt 2/5). Is it running?
notebook Traceback (most recent call last):
notebook File “/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/extension.py”, line 336, in check_hub_version
notebook resp = await client.fetch(self.hub_auth.api_url)
notebook ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
notebook ConnectionRefusedError: [Errno 111] Connection refused
notebook [E 2023-11-20 07:00:35.868 JupyterHubSingleUser] Failed to connect to my Hub at http://127.0.0.1:8081/hub/api (attempt 3/5). Is it running?
notebook Traceback (most recent call last):
notebook File “/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/extension.py”, line 336, in check_hub_version
notebook resp = await client.fetch(self.hub_auth.api_url)
notebook ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
notebook ConnectionRefusedError: [Errno 111] Connection refused
notebook [E 2023-11-20 07:00:43.876 JupyterHubSingleUser] Failed to connect to my Hub at http://127.0.0.1:8081/hub/api (attempt 4/5). Is it running?
notebook Traceback (most recent call last):
notebook File “/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/extension.py”, line 336, in check_hub_version
notebook resp = await client.fetch(self.hub_auth.api_url)
notebook ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
notebook ConnectionRefusedError: [Errno 111] Connection refused
notebook [E 2023-11-20 07:00:59.884 JupyterHubSingleUser] Failed to connect to my Hub at http://127.0.0.1:8081/hub/api (attempt 5/5). Is it running?
notebook Traceback (most recent call last):
notebook File “/opt/conda/lib/python3.11/site-packages/jupyterhub/singleuser/extension.py”, line 336, in check_hub_version
notebook resp = await client.fetch(self.hub_auth.api_url)
notebook ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
notebook ConnectionRefusedError: [Errno 111] Connection refused

mahendrapaipuri · November 20, 2023, 1:38pm

From your logs, it seems that single user server is attempting to connect to hub running on localhost. If they are running on different nodes, you need to expose hub on an IP that single user server can connect to. Can you try to set c.JupyterHub.hub_ip to an IP that your single user servers can connect to?

krishnav · November 20, 2023, 2:19pm

@mahendrapaipuri Thanks for quick reply.
Updated my values.yaml file with hub_ip: "0.0.0.0" and redeployed the hub.

config:
    JupyterHub:
      authenticator_class: nullauthenticator.NullAuthenticator
      # only serve the hub's API, not full UI
      hub_routespec: "/hub/api/"
      redirect_to_server: True
      # hub_connect_url: "http://hub:8081"
      hub_ip: "0.0.0.0"

Still the same connection issue in single user server startup.

FYI, I have overrided jupyter_server_config.py file in docker image used for single user server. Do I need to add c.JupyterHub.hub_ip there or hub helm values.yaml fie.
Can you explain the difference of these two configs. Thanks!

mahendrapaipuri · November 20, 2023, 2:38pm

Can you try with c.JupyterHub.hub_connect_url and set it to public one?

jupyter_server_config.py is the config file for single user servers, ie., JupyterLab/Notebook.
jupyterhub_config.py is the config file for JupyterHub.

All the hub related config will go into jupyterhub_config.py and all the JupyterLab/Notebook related config goes to jupyter_server_config.py

krishnav · November 20, 2023, 2:52pm

Already tried with this, no luck.
hub_connect_url: "http://hub:8081"

And one more thing, I can see this JUPYTERHUB_API_URL: http://hub:8081/hub/api environment variable in single user server pod. But, single user server not taking this and trying to connect Hub at http://127.0.0.1:8081/hub/api

mahendrapaipuri · November 20, 2023, 3:24pm

That is strange. If the env var is set, API URL should be set to the one in env var.

How are you sure that single user server pod has this env var set?

krishnav · November 20, 2023, 3:42pm

In 2 ways, I’m finding this variable value.

kubectl describe pod my-user-server-pod resulted this value.

2.Connected to my-user-server-pod before it gets terminated and ran env command to list all the environment variables. Resulted this. JUPYTERHUB_API_URL=http://hub:8081/hub/api

krishnav · November 20, 2023, 4:02pm

I have observed this in my testing with 2 different version of base image tag. I’m not sure is this relevant or not. Please have a look.

The below commit to jupyterhub repo on 02/14/2023 has the fix.

Jupterhub tag before this fix. i.e., 3.1.1 version on 01/27/2023 was working for me with Jupyterlab version 3.6.3 and jupyterhub version 3.1.1 which got installed using below base-notebook version

).

After, I updated my notebook image tag to below, it installed Jupyterlab version 4.0.7 and jupyterhub version 4.0.2

Tag Reference: Release 3.1.1 · jupyterhub/jupyterhub · GitHub

mahendrapaipuri · November 20, 2023, 4:35pm

Can you share your full JupyterHub config?

And try setting JUPYTERHUB_SINGLEUSER_EXTENSION=0 for your single user pod and see if it helps.

krishnav · November 21, 2023, 6:33am

@mahendrapaipuri JUPYTERHUB_SINGLEUSER_EXTENSION=0 option worked for me. Thanks for the help.

Can you explain, what does it mean and do I need to set this every time from now on wards?

mahendrapaipuri · November 21, 2023, 8:35am

Well, starting from JupyterHub 4, single user server app has been implemented as an extension to jupyter_server. It is mostly an implementation detail and it is supposed to be completely transparent for users. Basically using JUPYTERHUB_SINGLEUSER_EXTENSION=0 will use the single user server app implementation that is used in JupyterHub 3.x and not the new extension implementation.

In any case, both implementations read the hub API URL from the same env var JUPYTERHUB_API_URL. It is very strange that it works for one case and not for other.

Can you share your complete hub and single user config, please?

krishnav · November 21, 2023, 10:58am

hub chart values.yaml (Removed proxy and scheduling to minimize the config)

singleuser:
defaultUrl: “/lab”
image:
name: “my-docker-image”
tag: “latest”
storage:
type: “none”

hub:
initContainers:
# custom-templates config volume mount config here
config:
JupyterHub:
authenticator_class: nullauthenticator.NullAuthenticator
# only serve the hub’s API, not full UI
hub_routespec: “/hub/api/”
redirect_to_server: True

extraConfig:
templates: |
c.JupyterHub.template_paths = [‘mount-path’]
tornado_cors_settings: |
origin = ‘xxx’
c.JupyterHub.tornado_settings = {
‘headers’: {
‘Access-Control-Allow-Origin’: origin,
‘Access-Control-Allow-Methods’: ‘GET, POST, OPTIONS, PUT, DELETE’
},
}
lifecycle_hooks: |
c.KubeSpawner.lifecycle_hooks = {
“postStart”: {
“exec”: {
“command”: [“/bin/sh”, “xxx.sh”]
}
},
}
spawner_add_env: |
  from kubespawner import KubeSpawner
  

  class AbcSpawner(KubeSpawner):

    def _build_common_labels(self, extra_labels):
      labels = super()._build_common_labels(extra_labels)
      del labels['hub.jupyter.org/username']
      return labels        

    def get_env(self):
      env = super().get_env()
      env['JUPYTERHUB_SINGLEUSER_EXTENSION'] = '0'
      # custom code
      return env

  c.JupyterHub.spawner_class = AbcSpawner

jupyter_server_config.py

c = get_config()
c.ContentsManager.allow_hidden = True
c.FileContentsManager.allow_hidden = True
c.FileContentsManager.always_delete_dir = True
c.FileContentsManager.delete_to_trash = False

c.ServerApp.allow_origin_pat=‘reg-ex to my domain’
c.ServerApp.tornado_settings = {‘cookie_options’: {‘SameSite’: ‘None’, ‘Secure’: True}}
c.ServerApp.cookie_options = {‘SameSite’: ‘None’, ‘Secure’: True}
c.JupyterHub.tornado_settings = {‘cookie_options’: {‘SameSite’: ‘None’, ‘Secure’: True}}

krishnav · December 7, 2023, 7:05am

Could you check my config and help me in correcting it, if I miss anything to work single user server app as extension.

mahendrapaipuri · December 7, 2023, 9:21am

No, I dont see any obvious in your config. Not sure why server extension auth does not work for you. But it is not an issue for you. This is mostly an implementation detail and should not have any differences in functionality.

Topic		Replies	Views
500: Failed to connect to my Hub at http://hub:8081/hub/hub/api General community , jupyterhub , how-to , help-wanted	3	3201	February 14, 2022
Single user server failed to connect hub API JupyterHub jupyterhub	0	231	March 5, 2024
JupyterHub not loading Recent Jupyter Docker-Stacks JupyterHub	2	714	August 18, 2020
Why are hub and user jupyterhub versions mismatched in the z2jh Helm chart? Zero to JupyterHub on Kubernetes jupyterhub , question , docker	2	809	June 15, 2022
JupyterHub with DockerSpawner: Failed to connect to Hub API JupyterHub help-wanted	5	3388	March 25, 2021

JupyterHubSingleUser - Failed to connect to my Hub at http://127.0.0.1:8081/hub/api

Related topics