Issue with hub-managed service on Kubernetes: 503 ECONNREFUSED

JupyterHub
,
1

Has anyone experience with hub-managed services in JupyterHub?

I’m trying to make a deployment of JupyterHub (5.2.1) that includes the hub-managed service jhub-apps (2025.2.1) on Azure Kubernetes (1.30.9). Everything works well (authentication, singleuser servers etc…), but not the jhub-apps service. Here is a link to the their repo (GitHub - nebari-dev/jhub-apps: Application creator and general launcher for JupyterHub).

Deployment is installed using helm with the JupyterHub helm chart version 4.1.0. ‘jhub-apps’ is installed in a custom hub image, which is derived from jupyterhub/k8s-hub:4.1.0 (the default image for that helm chart). I’m turning on ‘jhub-apps’ using the ‘extraConfig’ of the helm deployment yaml file (see below). The deployment works fine, if I’m skipping the ‘install_jhub_apps’ and the reconfiguration of the ‘theme_template_parts’ part at the end

The hub pod logs (see below) show that everything was started correctly. However, when the home screen /hub/home is supposed to come up, an error 503 seems to sent out. The proxy pod logs show this more clearly with 503 messages: ERRCONNREFUSED. (see below, too)

What am I doing wrong? I already tried things like turning off network policies that might block important ports (like port 10202), and even tried with a earlier versions of JupyterHub (4.0.2).

Any help with troubleshooting is greatly appreciated!

These are the logs of the proxy pod:
It looks like if the proxy pod does not route to the japps service correctly. I think it should be routing to http:// hub:10202, not to https:// 127.0.0.1:10202 ? But How to change that?

21:34:49.951 [ConfigProxy] info: Adding route / -> http://hub:8081
21:34:50.018 [ConfigProxy] info: Proxying https://*:8443 to http://hub:8081
21:34:50.018 [ConfigProxy] info: Proxying https://*:8443 to http://hub:8081
21:34:50.018 [ConfigProxy] info: Proxy API at http://*:8001/api/routes
21:34:50.020 [ConfigProxy] info: Added HTTP to HTTPS redirection from 8000 to 443
21:34:50.021 [ConfigProxy] info: Route added / -> http://hub:8081
21:35:04.995 [ConfigProxy] info: 200 GET /api/routes
21:35:08.113 [ConfigProxy] info: 200 GET /api/routes
21:35:08.118 [ConfigProxy] info: Adding route / -> http://hub:8081
21:35:08.118 [ConfigProxy] info: Route added / -> http://hub:8081
21:35:08.119 [ConfigProxy] info: 201 POST /api/routes/
21:35:08.120 [ConfigProxy] info: Adding route /services/japps -> http://127.0.0.1:10202
21:35:08.121 [ConfigProxy] info: Route added /services/japps -> http://127.0.0.1:10202
21:35:08.122 [ConfigProxy] info: 201 POST /api/routes/services/japps
21:36:05.891 [ConfigProxy] error: 503 GET /services/japps/static/css/index.css connect ECONNREFUSED 127.0.0.1:10202
21:36:05.894 [ConfigProxy] error: 503 GET /services/japps/static/js/index.js connect ECONNREFUSED 127.0.0.1:10202
21:36:08.127 [ConfigProxy] info: 200 GET /api/routes
21:36:35.366 [ConfigProxy] error: 503 GET /services/japps/ connect ECONNREFUSED 127.0.0.1:10202

Deployment yaml file (for helm deployment):

scheduling:
  corePods:
    nodeAffinity:
      matchNodePurpose: prefer
  userScheduler:
    enabled: false
  podPriority:
    enabled: true
  userPlaceholder:
    enabled: false

cull:
  enabled: false

singleuser:
  networkPolicy:
    egressAllowRules:
      privateIPs: true
  defaultUrl: "/lab"
  image:
    name: quay.io/jupyter/base-notebook
    tag: 2025-03-14
  storage:
    capacity: 5Gi

proxy:
  service:
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      service.beta.kubernetes.io/azure-load-balancer-ipv4: "<ip-address>"
  chp:
    networkPolicy:
      enabled: false
  traefik:
    networkPolicy:
      enabled: false
  https:
    enabled: true
    hosts:
      - <dns-address>
    type: secret
    secret: 
      name: the-cert

hub:
  image:
    name: >registry-name>.azurecr.io/<hub-image-name>
    tag: <hub-tag>
  networkPolicy:
    enabled: false
  config:
    Authenticator:
      admin_users:
        - the admin
    AzureAdOAuthenticator:
      client_id: "<client-id>"
      client_secret: "<client-secret>"
      oauth_callback_url: "https://<dns-address>/hub/oauth_callback"
      tenant_id: "<tenant-id>"
    JupyterHub:
      authenticator_class: azuread
  extraConfig:
    my_config: |
      c.Authenticator.allow_all = True
      c.KubeSpawner.default_url = "/lab"
      from jhub_apps.configuration import install_jhub_apps
      from jhub_apps import theme_template_paths
      from kubespawner import KubeSpawner
      c.JupyterHub.bind_url = "https://<dns-address>"      
      c.JupyterHub.default_url = "/hub/home"
      c.JAppsConfig.allowed_frameworks = ["jupyterlab"]
      c.JAppsConfig.jupyterhub_config_path = "/usr/local/etc/jupyterhub/jupyterhub_config.py"
      c.JupyterHub.template_paths = theme_template_paths
      c = install_jhub_apps(c, spawner_to_subclass=KubeSpawner)

These are the logs of the hub pod:

Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
Loading extra config: my_config
[I 2025-03-15 21:35:04.620 JupyterHub app:3346] Running JupyterHub version 5.2.1
[I 2025-03-15 21:35:04.621 JupyterHub app:3376] Using Authenticator: oauthenticator.azuread.AzureAdOAuthenticator-17.3.0
[I 2025-03-15 21:35:04.621 JupyterHub app:3376] Using Spawner: jhub_apps.spawner.spawner_creation.JHubSpawner
[I 2025-03-15 21:35:04.621 JupyterHub app:3376] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.2.1
/usr/local/lib/python3.12/site-packages/jupyter_events/schema.py:68: JupyterEventsVersionWarning: The `version` property of an event schema must be a string. It has been type coerced, but in a future version of this library, it will fail to validate. Please update schema: https://schema.jupyter.org/jupyterhub/events/server-action
  validate_schema(_schema)
[I 2025-03-15 21:35:04.825 JupyterHub app:2881] Creating service japps with oauth_client_id=service-japps
[I 2025-03-15 21:35:04.829 JupyterHub provider:663] Updating oauth client service-japps
[I 2025-03-15 21:35:04.920 JupyterHub app:2919] Creating service japps-initialize-startup-apps without oauth.
[I 2025-03-15 21:35:04.978 JupyterHub app:3416] Initialized 0 spawners in 0.011 seconds
[I 2025-03-15 21:35:04.985 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2025-03-15 21:35:04.986 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.seven_days
[I 2025-03-15 21:35:04.987 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.thirty_days
[I 2025-03-15 21:35:04.988 JupyterHub app:3703] Not starting proxy
[I 2025-03-15 21:35:04.995 JupyterHub app:3739] Hub API listening on http://:8081/hub/
[I 2025-03-15 21:35:04.995 JupyterHub app:3741] Private Hub API connect url http://hub:8081/hub/
[I 2025-03-15 21:35:04.996 JupyterHub app:3615] Starting managed service japps at http://127.0.0.1:10202
[I 2025-03-15 21:35:04.996 JupyterHub service:423] Starting service 'japps': ['python', '-m', 'uvicorn', 'jhub_apps.service.app:app', '--port=10202', '--host=0.0.0.0', '--workers=2']
[I 2025-03-15 21:35:04.997 JupyterHub service:136] Spawning python -m uvicorn jhub_apps.service.app:app --port=10202 --host=0.0.0.0 --workers=2
INFO:     Uvicorn running on http://0.0.0.0:10202 (Press CTRL+C to quit)
INFO:     Started parent process [12]
INFO:     Started server process [14]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [15]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     127.0.0.1:34084 - "GET /services/japps HTTP/1.1" 307 Temporary Redirect
[I 2025-03-15 21:35:08.108 JupyterHub app:3615] Starting managed service japps-initialize-startup-apps
[I 2025-03-15 21:35:08.108 JupyterHub service:423] Starting service 'japps-initialize-startup-apps': ['/bin/sh', '-c', 'python -m jhub_apps.tasks.commands.initialize_startup_apps && tail -f /dev/null']
[I 2025-03-15 21:35:08.110 JupyterHub service:136] Spawning /bin/sh -c 'python -m jhub_apps.tasks.commands.initialize_startup_apps && tail -f /dev/null'
[I 2025-03-15 21:35:08.113 JupyterHub proxy:477] Adding route for Hub: / => http://hub:8081
[W 2025-03-15 21:35:08.114 JupyterHub proxy:445] Adding missing route for japps (Server(url=http://127.0.0.1:10202/services/japps/, bind_url=http://127.0.0.1:10202/services/japps/))
[I 2025-03-15 21:35:08.114 JupyterHub proxy:312] Adding service japps to proxy /services/japps/ => http://127.0.0.1:10202
[I 2025-03-15 21:35:08.122 JupyterHub app:3772] JupyterHub is now running, internal Hub API at http://hub:8081/hub/
2025-03-15 21:35:10,693      INFO jhub_apps.service.utils:  45: event='Getting JHub config from file: /usr/local/etc/jupyterhub/jupyterhub_config.py' view=None peer=None
2025-03-15 21:35:11,351      INFO jhub_apps.service.utils:  50: event="JHub Apps config: {'allowed_frameworks': ['jupyterlab'], 'jupyterhub_config_path': '/usr/local/etc/jupyterhub/jupyterhub_config.py', 'app_icon': 'https://jupyter.org/assets/homepage/main-logo.svg', 'app_title': 'JHub Apps Launcher', 'apps_auth_type': 'oauth', 'blocked_frameworks': None, 'conda_envs': [], 'hub_host': '127.0.0.1', 'python_exec': 'python', 'service_workers': 2, 'startup_apps': [], 'config':{...}, 'JAppsConfig': {...}}, 'log': <_FixedFindCallerLogger traitlets (INFO)>, 'parent': None}" view=None peer=None
2025-03-15 21:35:11,351      INFO __main__:  51: event='Finished instantiating startup apps' view=None peer=None
Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
Loading extra config: my_config
[I 2025-03-15 21:36:00.967 JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:10.100.200.36) 1.18ms
[W 2025-03-15 21:36:01.011 JupyterHub base:482] Invalid or expired cookie token
[I 2025-03-15 21:36:01.012 JupyterHub log:192] 302 GET /hub/ -> /hub/home (@::ffff:10.100.200.36) 1.74ms
[I 2025-03-15 21:36:01.060 JupyterHub log:192] 302 GET /hub/home -> /hub/login?next=%2Fhub%2Fhome (@::ffff:10.100.200.36) 1.24ms
[I 2025-03-15 21:36:01.099 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'70ce983bbab043bb9e73de95cef3af96:Vv6f8Mq9y2qwph-Gea583S4T972AIajjH9jRZu7cMDw=' {'path': '/hub/', 'max_age': 3600} 
[I 2025-03-15 21:36:01.162 JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2Fhome (@::ffff:10.100.200.36) 64.61ms
[I 2025-03-15 21:36:03.832 JupyterHub oauth2:126] OAuth redirect: https://<dns-address>/hub/oauth_callback
[I 2025-03-15 21:36:03.833 JupyterHub log:192] 302 GET /hub/oauth_login?next=%2Fhub%2Fhome -> https://login.microsoftonline.com/<....>
[I 2025-03-15 21:36:05.747 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'70ce983bbab043bb9e73de95cef3af96:7cd0de7dd8e046cab0aeeb2cc5d0c76b' {'path': '/hub/'}
[I 2025-03-15 21:36:05.747 JupyterHub base:973] User logged in: the admin
[I 2025-03-15 21:36:05.748 JupyterHub log:192] 302 GET /hub/oauth_callback?code=[secret]&state=[secret]&session_state=[secret] -> /hub/home (user name@::ffff:10.100.200.36) 256.11ms
[I 2025-03-15 21:36:05.826 JupyterHub log:192] 200 GET /hub/home (the admin@::ffff:10.100.200.36) 39.54ms
[I 2025-03-15 21:36:05.894 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:d_vj8HY38q0MrtNc84UHiW7g3ty5C9wR20bsl4_urTI=' {'path': '/hub/', 'max_age': 3600}
[I 2025-03-15 21:36:05.902 JupyterHub log:192] 200 GET /hub/error/503?url=%2Fservices%2Fjapps%2Fstatic%2Fcss%2Findex.css%3Fv%3D20250315213504 (@10.100.200.58) 8.66ms
[I 2025-03-15 21:36:05.903 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:d_vj8HY38q0MrtNc84UHiW7g3ty5C9wR20bsl4_urTI=' {'path': '/hub/', 'max_age': 3600}
[I 2025-03-15 21:36:05.905 JupyterHub log:192] 200 GET /hub/error/503?url=%2Fservices%2Fjapps%2Fstatic%2Fjs%2Findex.js%3Fv%3D20250315213504 (@10.100.200.58) 2.28ms
INFO:     127.0.0.1:50250 - "GET /services/japps HTTP/1.1" 307 Temporary Redirect
[I 2025-03-15 21:36:35.367 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:d_vj8HY38q0MrtNc84UHiW7g3ty5C9wR20bsl4_urTI=' {'path': '/hub/', 'max_age': 3600}
[I 2025-03-15 21:36:35.369 JupyterHub log:192] 200 GET /hub/error/503?url=%2Fservices%2Fjapps%2F (@10.100.200.58) 2.52ms
2

Try accessing your service via JupyterHub instead, under /services/service name/. This is also passed to the service as the JUPYTERHUB_SERVICE_PREFIX environment variables

If you want to access the service on a custom external port I think you’ll need to run it as an external service in it’s own pod

You could try to customise your hub pod and kubernetes service to expose your additional service, but given this is kubernetes I think running it separately fits better with the overall architecture of Z2JH. An added benefit is you no longer have to customise your hub image since your service will be decoupled.

3

@manics: Thank for your help. You’re right, it could be a better idea to run the service in a another pod. The only real downside I see, is that the deployment of the service would not be within the standard helm deployment, but that 's manageable. I will try that.

Nevertheless, it should be possible to run the service within the hub pod. I just stumbled upon this thread: Running dash app as a Hub Service in EKS. In that case the service is indeed running as a hub-managed service within the same pod. So maybe it’s rather a configuration problem, or a bug in the ‘jhub-apps’ service itself.

If I’m not succeeding with ‘jhub-apps’ in a another pod, I will try to deploy one of the simpler services, like this one jupyterhub/examples/service-whoami-flask at main · jupyterhub/jupyterhub · GitHub as a hub-managed service.