JupyterHub stuck after the first login on Kubernetes (with NGINX Ingress Controller)

nuka137 · August 11, 2021, 6:24am

I’m trying to install JupyterHub on Kubernetes by following the helm installation.
I’m also trying to use the ingress to access JupyterHub by the sub-directry /jupyterhub . (The full base URL likes https://secure-ingress.com/jupyterhub/hub)

To meet this requirement, I changed values.yaml whose original version is downloaded from chart repository (chart version is 1.1.2).

--- jupyterhub.orig/jupyterhub/values.yaml      2021-08-05 21:38:58.000000000 +0000
+++ jupyterhub/values.yaml      2021-08-11 06:09:14.206337343 +0000
@@ -30,6 +30,8 @@
     JupyterHub:
       admin_access: true
       authenticator_class: dummy
+    Authenticator:
+      admin_users: [admin]
   service:
     type: ClusterIP
     annotations: {}
@@ -37,7 +39,7 @@
       nodePort:
     extraPorts: []
     loadBalancerIP:
-  baseUrl: /
+  baseUrl: /jupyterhub
   cookieSecret:
   initContainers: []
   fsGid: 1000
@@ -174,7 +176,7 @@
     rollingUpdate:
   # service relates to the proxy-public service
   service:
-    type: LoadBalancer
+    type: ClusterIP
     labels: {}
     annotations: {}
     nodePorts:
@@ -537,12 +541,22 @@
       pullSecrets: []
 
 ingress:
-  enabled: false
-  annotations: {}
-  hosts: []
+  enabled: true
+  annotations: {
+  }
+  hosts: [
+    "secure-ingress.com"
+  ]
   pathSuffix:
   pathType: Prefix
-  tls: []
+  tls: [
+    {
+      "hosts": [
+        "secure-ingress.com"
+      ],
+      "secretName": "k8s-jupyterhub-ingress-tls-secret"
+    }
+  ]
 
 # cull relates to the jupyterhub-idle-culler service, responsible for evicting
 # inactive singleuser pods.

And then, I run the below command.

helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
helm repo update
helm install jupyterhub jupyterhub/jupyterhub -n jupyterhub --create-namespace --values values.yaml

Above commands did not output any errors and I could access to https://secure-ingress.com/jupyterhub/hub without errors.

However, JupyterHub stuck after the first admin login (only displays Your server is starting up.).
Is there any mistakes in my installation process of JupyterHub on Kubernetes Ingress?

manics · August 11, 2021, 11:27am

Hi! Could you give us more information about your Kubenetes cluster, e.g. is it on public cloud, did you deploy it yourself on your own servers, etc.

The Z2JH guide has some debuggging advice: Debugging — Zero to JupyterHub with Kubernetes documentation

Could you try those steps, and if you’re still stuck show us the logs from your hub and singleuser pods? Thanks!

nuka137 · August 11, 2021, 11:13pm

@manics

My kubernetes environment is deployed on the own on-premise server behind the proxy.

Thanks for sharing the debugging method.
I enabled the debug option and output the related message.
But I could not find the solutions yet.

$ kubectl get po -n jupyterhub 
NAME                              READY   STATUS    RESTARTS   AGE
continuous-image-puller-9hl9k     1/1     Running   0          5m18s
continuous-image-puller-rzb5r     1/1     Running   0          5m18s
hub-7bccb984c7-scrb8              1/1     Running   0          5m18s
jupyter-admin                     1/1     Running   0          4m48s
proxy-859664c6f7-jmt74            1/1     Running   0          5m18s
user-scheduler-599d6dbf87-b4b8g   1/1     Running   0          5m18s
user-scheduler-599d6dbf87-sjbvh   1/1     Running   0          5m18s

$ kubectl logs -n jupyterhub hub-7bccb984c7-scrb8 | tail -n 20
[D 2021-08-11 23:10:06.457 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.111) 1.26ms
[D 2021-08-11 23:10:08.394 JupyterHub reflector:357] pods watcher timeout
[D 2021-08-11 23:10:08.394 JupyterHub reflector:282] Connecting pods watcher
[D 2021-08-11 23:10:08.457 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.111) 2.05ms
[D 2021-08-11 23:10:08.458 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.111) 2.09ms
[D 2021-08-11 23:10:08.679 JupyterHub reflector:357] events watcher timeout
[D 2021-08-11 23:10:08.679 JupyterHub reflector:282] Connecting events watcher
[D 2021-08-11 23:10:10.456 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.111) 1.23ms
[D 2021-08-11 23:10:12.456 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.111) 1.35ms
[D 2021-08-11 23:10:14.456 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.111) 1.25ms
[D 2021-08-11 23:10:16.456 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.111) 1.24ms
[D 2021-08-11 23:10:18.405 JupyterHub reflector:357] pods watcher timeout
[D 2021-08-11 23:10:18.405 JupyterHub reflector:282] Connecting pods watcher
[D 2021-08-11 23:10:18.456 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.111) 2.03ms
[D 2021-08-11 23:10:18.457 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.111) 2.06ms
[D 2021-08-11 23:10:18.697 JupyterHub reflector:357] events watcher timeout
[D 2021-08-11 23:10:18.697 JupyterHub reflector:282] Connecting events watcher
[D 2021-08-11 23:10:20.456 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.111) 1.28ms
[D 2021-08-11 23:10:22.463 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.111) 3.15ms
[D 2021-08-11 23:10:24.457 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.111) 1.26ms

$ kubectl logs -n jupyterhub jupyter-admin  | tail -n 20
        /home/jovyan/.jupyter/jupyter_notebook_config.json
[I 2021-08-11 23:04:21.627 SingleUserNotebookApp notebookapp:1593] Authentication of /metrics is OFF, since other authentication is disabled.
[W 2021-08-11 23:04:22.203 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-08-11 23:04:22.203 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-08-11 23:04:22.203 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-08-11 23:04:22.203 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[D 2021-08-11 23:04:22.212 LabApp] Config changed: {'NotebookApp': {'open_browser': False, 'nbserver_extensions': <LazyConfigValue {'update': {'jupyterlab': True, 'nbgitpuller': True}}>}, 'ServerApp': {'ip': '0.0.0.0', 'port': 8888}, 'FileContentsManager': {'delete_to_trash': False}, 'SingleUserNotebookApp': {'ip': '0.0.0.0', 'port': 8888}, 'Application': {'log_level': 10}}
[I 2021-08-11 23:04:22.213 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.9/site-packages/jupyterlab
[I 2021-08-11 23:04:22.213 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
Patching auth into jupyter_server.base.handlers.JupyterHandler(jupyter_server.base.handlers.AuthenticatedHandler) -> JupyterHandler(jupyterhub.singleuser.mixins.HubAuthenticatedHandler, jupyter_server.base.handlers.AuthenticatedHandler)
[I 2021-08-11 23:04:22.221 SingleUserNotebookApp mixins:576] Starting jupyterhub-singleuser server version 1.4.2
[D 2021-08-11 23:04:22.230 SingleUserNotebookApp _version:76] jupyterhub and jupyterhub-singleuser both on version 1.4.2
[I 2021-08-11 23:04:22.231 SingleUserNotebookApp notebookapp:2302] Serving notebooks from local directory: /home/jovyan
[I 2021-08-11 23:04:22.231 SingleUserNotebookApp notebookapp:2302] Jupyter Notebook 6.4.0 is running at:
[I 2021-08-11 23:04:22.231 SingleUserNotebookApp notebookapp:2302] http://jupyter-admin:8888/jupyterhub/user/admin/
[I 2021-08-11 23:04:22.231 SingleUserNotebookApp notebookapp:2303] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2021-08-11 23:04:22.245 SingleUserNotebookApp mixins:556] Updating Hub with activity every 300 seconds
[D 2021-08-11 23:04:22.246 SingleUserNotebookApp mixins:518] Notifying Hub of activity 2021-08-11T23:04:21.626437Z
[I 2021-08-11 23:04:22.488 SingleUserNotebookApp log:189] 302 GET /jupyterhub/user/admin/ -> /jupyterhub/user/admin/tree? (@10.233.117.50) 1.37ms
[D 2021-08-11 23:09:29.719 SingleUserNotebookApp mixins:518] Notifying Hub of activity 2021-08-11T23:04:21.626437Z

If you need more information, please tell me.

manics · August 12, 2021, 9:21am

Hi! Could you show us the logs for the jupyter-admin pod from the beginning?

nuka137 · August 12, 2021, 9:36pm

@manics

Here is the full logs of jupyter-admin pod.

[D 2021-08-12 21:34:05.265 SingleUserNotebookApp application:164] Searching ['/home/jovyan', '/home/jovyan/.jupyter', '/opt/conda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2021-08-12 21:34:05.265 SingleUserNotebookApp application:730] Looking for jupyter_config in /etc/jupyter
[D 2021-08-12 21:34:05.265 SingleUserNotebookApp application:730] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2021-08-12 21:34:05.265 SingleUserNotebookApp application:730] Looking for jupyter_config in /opt/conda/etc/jupyter
[D 2021-08-12 21:34:05.265 SingleUserNotebookApp application:730] Looking for jupyter_config in /home/jovyan/.jupyter
[D 2021-08-12 21:34:05.265 SingleUserNotebookApp application:730] Looking for jupyter_config in /home/jovyan
[D 2021-08-12 21:34:05.266 SingleUserNotebookApp application:730] Looking for jupyter_notebook_config in /etc/jupyter
[D 2021-08-12 21:34:05.266 SingleUserNotebookApp application:752] Loaded config file: /etc/jupyter/jupyter_notebook_config.py
[D 2021-08-12 21:34:05.266 SingleUserNotebookApp application:730] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 2021-08-12 21:34:05.266 SingleUserNotebookApp application:730] Looking for jupyter_notebook_config in /opt/conda/etc/jupyter
[D 2021-08-12 21:34:05.266 SingleUserNotebookApp application:752] Loaded config file: /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2021-08-12 21:34:05.266 SingleUserNotebookApp application:730] Looking for jupyter_notebook_config in /home/jovyan/.jupyter
[D 2021-08-12 21:34:05.266 SingleUserNotebookApp application:730] Looking for jupyter_notebook_config in /home/jovyan
[W 2021-08-12 21:34:05.267 SingleUserNotebookApp configurable:190] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[D 2021-08-12 21:34:05.275 SingleUserNotebookApp config_manager:96] Paths used for configuration of jupyter_notebook_config: 
        /etc/jupyter/jupyter_notebook_config.json
[D 2021-08-12 21:34:05.275 SingleUserNotebookApp config_manager:96] Paths used for configuration of jupyter_notebook_config: 
        /usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2021-08-12 21:34:05.275 SingleUserNotebookApp config_manager:96] Paths used for configuration of jupyter_notebook_config: 
        /opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
        /opt/conda/etc/jupyter/jupyter_notebook_config.d/nbgitpuller.json
        /opt/conda/etc/jupyter/jupyter_notebook_config.json
[D 2021-08-12 21:34:05.276 SingleUserNotebookApp config_manager:96] Paths used for configuration of jupyter_notebook_config: 
        /home/jovyan/.jupyter/jupyter_notebook_config.json
[I 2021-08-12 21:34:05.283 SingleUserNotebookApp notebookapp:1593] Authentication of /metrics is OFF, since other authentication is disabled.
[W 2021-08-12 21:34:05.875 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-08-12 21:34:05.875 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-08-12 21:34:05.875 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-08-12 21:34:05.875 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[D 2021-08-12 21:34:05.885 LabApp] Config changed: {'NotebookApp': {'open_browser': False, 'nbserver_extensions': <LazyConfigValue {'update': {'jupyterlab': True, 'nbgitpuller': True}}>}, 'ServerApp': {'ip': '0.0.0.0', 'port': 8888}, 'FileContentsManager': {'delete_to_trash': False}, 'SingleUserNotebookApp': {'ip': '0.0.0.0', 'port': 8888}, 'Application': {'log_level': 10}}
[I 2021-08-12 21:34:05.887 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.9/site-packages/jupyterlab
[I 2021-08-12 21:34:05.887 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
Patching auth into jupyter_server.base.handlers.JupyterHandler(jupyter_server.base.handlers.AuthenticatedHandler) -> JupyterHandler(jupyterhub.singleuser.mixins.HubAuthenticatedHandler, jupyter_server.base.handlers.AuthenticatedHandler)
[I 2021-08-12 21:34:05.894 SingleUserNotebookApp mixins:576] Starting jupyterhub-singleuser server version 1.4.2
[D 2021-08-12 21:34:05.903 SingleUserNotebookApp _version:76] jupyterhub and jupyterhub-singleuser both on version 1.4.2
[I 2021-08-12 21:34:05.903 SingleUserNotebookApp notebookapp:2302] Serving notebooks from local directory: /home/jovyan
[I 2021-08-12 21:34:05.903 SingleUserNotebookApp notebookapp:2302] Jupyter Notebook 6.4.0 is running at:
[I 2021-08-12 21:34:05.903 SingleUserNotebookApp notebookapp:2302] http://jupyter-admin:8888/jupyterhub/user/admin/
[I 2021-08-12 21:34:05.903 SingleUserNotebookApp notebookapp:2303] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2021-08-12 21:34:05.912 SingleUserNotebookApp mixins:556] Updating Hub with activity every 300 seconds
[D 2021-08-12 21:34:05.913 SingleUserNotebookApp mixins:518] Notifying Hub of activity 2021-08-12T21:34:05.283354Z
[I 2021-08-12 21:34:10.227 SingleUserNotebookApp log:189] 302 GET /jupyterhub/user/admin/ -> /jupyterhub/user/admin/tree? (@10.233.117.56) 1.59ms

nuka137 · August 16, 2021, 1:27am

I also confirmed that JupyterHub works correctly when it runs without ingress on my environment.
So, I think ingress configuration of JupyterHub may cause this problem.

Also, the access to the API (such as /hub/api/users) results 403 (Forbidden) error.
This 403 error can be seen at progress bar from web browser’s developer tools.
I’m not sure if this is related or not.

manics · August 16, 2021, 8:25pm

Would you mind explaining how did you tested this?

Are you sure your ingress controller is correctly setup? Note this is not the same as the ingress- Z2JH creates an ingress specification, but it’s up to the ingress controller to implement it. Since you’re running your own cluster it’s possible your ingress controller requires additional configuration or annotations.

nuka137 · August 16, 2021, 9:52pm

@manics

In this case, I only disable ingress in values.yaml.

i.e.

--- jupyterhub.orig/jupyterhub/values.yaml      2021-08-05 21:38:58.000000000 +0000
+++ jupyterhub/values.yaml      2021-08-16 06:33:43.637657836 +0000
@@ -30,6 +30,8 @@
     JupyterHub:
       admin_access: true
       authenticator_class: dummy
+    Authenticator:
+      admin_users: [admin]
   service:
     type: ClusterIP
     annotations: {}
@@ -37,6 +39,7 @@
       nodePort:
     extraPorts: []
     loadBalancerIP:
   baseUrl: /
   cookieSecret:
   initContainers: []
@@ -174,11 +177,12 @@
     rollingUpdate:
   # service relates to the proxy-public service
   service:
     type: LoadBalancer
     labels: {}
     annotations: {}
     nodePorts:
-      http:
+      http: 30100
       https:
     disableHttpPort: false
     extraPorts: []

Yes, I have already tested my ingress controller by setting up kubernetes dashboard and accessing it without any issues.
I think the ingress controller setup correctly.

BTW, I used NGINX Ingress Controller now.
The configuration of values.yaml of the first post is correct?

nuka137 · August 16, 2021, 11:41pm

Regarding this, I checked the log from hub pod.
There is 403 error.
I think the stuck after the first login comes from the failure of this API call.
But I’m not sure why this 403 error is raised when I accessed JupyterHub via the ingress controller.

[D 2021-08-16 23:34:09.398 JupyterHub user:317] Creating <class 'kubespawner.spawner.KubeSpawner'> for admin:
[I 2021-08-16 23:34:09.402 JupyterHub log:189] 302 GET /jupyterhub/hub/ -> /jupyterhub/hub/spawn (admin@10.233.119.0) 14.96ms
[D 2021-08-16 23:34:09.451 JupyterHub pages:217] Triggering spawn with default options for admin
[D 2021-08-16 23:34:09.451 JupyterHub base:880] Initiating spawn for admin
[D 2021-08-16 23:34:09.451 JupyterHub base:884] 0/64 concurrent spawns
[D 2021-08-16 23:34:09.451 JupyterHub base:889] 0 active servers
[I 2021-08-16 23:34:09.483 JupyterHub provider:574] Creating oauth client jupyterhub-user-admin
[D 2021-08-16 23:34:09.512 JupyterHub user:631] Calling Spawner.start for admin
[I 2021-08-16 23:34:09.514 JupyterHub spawner:2344] Attempting to create pvc claim-admin, with timeout 3
[I 2021-08-16 23:34:09.516 JupyterHub log:189] 302 GET /jupyterhub/hub/spawn -> /jupyterhub/hub/spawn-pending/admin (admin@10.233.119.0) 70.28ms
[I 2021-08-16 23:34:09.530 JupyterHub spawner:2361] PVC claim-admin already exists, so did not create new pvc.
[I 2021-08-16 23:34:09.536 JupyterHub spawner:2302] Attempting to create pod jupyter-admin, with timeout 3
[I 2021-08-16 23:34:09.637 JupyterHub pages:402] admin is pending spawn
[I 2021-08-16 23:34:09.638 JupyterHub log:189] 200 GET /jupyterhub/hub/spawn-pending/admin (admin@10.233.119.0) 11.91ms
[D 2021-08-16 23:34:09.820 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.74) 0.97ms
[W 2021-08-16 23:34:09.865 JupyterHub base:57] Blocking Cross Origin API request.  Referer: https://10.24.78.77:31340/jupyterhub/hub/spawn-pending/admin, Host: secure-ingress.com/jupyterhub/hub/
[W 2021-08-16 23:34:09.866 JupyterHub log:189] 403 GET /jupyterhub/hub/api/users/admin/server/progress (@10.233.119.0) 5.68ms
[D 2021-08-16 23:34:11.530 JupyterHub reflector:357] events watcher timeout
[D 2021-08-16 23:34:11.530 JupyterHub reflector:282] Connecting events watcher
[D 2021-08-16 23:34:11.532 JupyterHub reflector:357] pods watcher timeout
[D 2021-08-16 23:34:11.532 JupyterHub reflector:282] Connecting pods watcher
[D 2021-08-16 23:34:11.821 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.74) 1.90ms
[D 2021-08-16 23:34:11.821 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.74) 1.62ms
[I 2021-08-16 23:34:12.685 JupyterHub log:189] 200 GET /jupyterhub/hub/api (@10.233.99.75) 1.12ms
[D 2021-08-16 23:34:12.701 JupyterHub base:283] Recording first activity for <APIToken('b6fe...', user='admin')>
[D 2021-08-16 23:34:12.723 JupyterHub users:778] Activity for user admin: 2021-08-16T23:34:12.094719Z
[D 2021-08-16 23:34:12.724 JupyterHub users:796] Activity on server admin/: 2021-08-16T23:34:12.094719Z
[I 2021-08-16 23:34:12.734 JupyterHub log:189] 200 POST /jupyterhub/hub/api/users/admin/activity (admin@10.233.99.75) 38.25ms
[D 2021-08-16 23:34:13.820 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.74) 1.02ms
[D 2021-08-16 23:34:14.628 JupyterHub spawner:2592] pod jupyterhub/jupyter-admin events before launch: 2021-08-16T23:34:09.553340Z [Normal] Successfully assigned jupyterhub/jupyter-admin to node-worker-01
    2021-08-16T23:34:10Z [Normal] Container image "jupyterhub/k8s-network-tools:1.1.2" already present on machine
    2021-08-16T23:34:10Z [Normal] Created container block-cloud-metadata
    2021-08-16T23:34:10Z [Normal] Started container block-cloud-metadata
    2021-08-16T23:34:11Z [Normal] Container image "jupyterhub/k8s-singleuser-sample:1.1.2" already present on machine
    2021-08-16T23:34:11Z [Normal] Created container notebook
    2021-08-16T23:34:11Z [Normal] Started container notebook
[D 2021-08-16 23:34:14.647 JupyterHub spawner:1179] Polling subprocess every 30s
[D 2021-08-16 23:34:14.654 JupyterHub utils:225] Server at http://10.233.99.75:8888/jupyterhub/user/admin/ responded with 302
[D 2021-08-16 23:34:14.654 JupyterHub _version:76] jupyterhub and jupyterhub-singleuser both on version 1.4.2
[I 2021-08-16 23:34:14.654 JupyterHub base:909] User admin took 5.203 seconds to start
[I 2021-08-16 23:34:14.654 JupyterHub proxy:285] Adding user admin to proxy /jupyterhub/user/admin/ => http://10.233.99.75:8888
[D 2021-08-16 23:34:14.654 JupyterHub proxy:832] Proxy: Fetching POST http://proxy-api:8001/api/routes/jupyterhub/user/admin
[D 2021-08-16 23:34:15.821 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.74) 1.16ms
[D 2021-08-16 23:34:17.820 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.74) 1.02ms
[D 2021-08-16 23:34:19.820 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.74) 1.02ms
[D 2021-08-16 23:34:20.220 JupyterHub proxy:832] Proxy: Fetching GET http://proxy-api:8001/api/routes
[D 2021-08-16 23:34:20.225 JupyterHub proxy:905] Omitting non-jupyterhub route '/'
[I 2021-08-16 23:34:20.246 JupyterHub proxy:347] Checking routes
[D 2021-08-16 23:34:21.573 JupyterHub reflector:357] pods watcher timeout
[D 2021-08-16 23:34:21.574 JupyterHub reflector:282] Connecting pods watcher
[D 2021-08-16 23:34:21.576 JupyterHub reflector:357] events watcher timeout
[D 2021-08-16 23:34:21.576 JupyterHub reflector:282] Connecting events watcher
[D 2021-08-16 23:34:21.821 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.74) 1.80ms
[D 2021-08-16 23:34:21.821 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.74) 1.79ms
[D 2021-08-16 23:34:23.820 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.74) 1.08ms
[D 2021-08-16 23:34:25.820 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.74) 1.01ms
[D 2021-08-16 23:34:27.820 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.74) 1.12ms
[D 2021-08-16 23:34:29.820 JupyterHub log:189] 200 GET /jupyterhub/hub/health (@10.24.78.74) 1.04ms

manics · August 17, 2021, 7:28pm

I think your ingress config is OK.

Have you checked your ingress controller is correctly setting or forwarding all headers?

Also, from your earlier post:

Is this a separate proxy in front of your K8s cluster?

nuka137 · August 18, 2021, 1:28am

Is this mean use-forwarded-headers in NGINX ingress controller?
I configured this, but results are same.

I will show you the detail installation command of NGINX ingress controller.

$ helm install ingress-nginx ingress-nginx/ingress-nginx --set controller.config.use-forward-headers=true

Yes, that’s right.
This is an our company’s proxy server separated from k8s cluster.

manics · August 18, 2021, 6:26pm

Does JupyterHub work if you connect directly to the K8S ingress, bypassing your company’s proxy server?

nuka137 · August 18, 2021, 9:22pm

No, JuyterHub is not working even if I directly connect to the ingress.

nuka137 · September 8, 2021, 12:53am

@manics

Is there any other idea about this issue?
I could not solve this issue still now.

manics · September 8, 2021, 3:33pm

That error suggests there’s a mismatched host header somewhere. It might be worth looking at your nginx ingress logs to see if there’s anything informative in them?

nuka137 · February 15, 2022, 1:49am

Sorry for answering this question.

It might be worth looking at your nginx ingress logs to see if there’s anything informative in them?

Yes. Your advice makes me solve the issue.
The main reason of this issue comes from the mismatch between host header and URL.
I changed host header by using browser plugin, and this made the mismatch between them.

Topic		Replies	Views
Trouble configuring Ingress for Helm Chart Zero to JupyterHub on Kubernetes how-to	4	2157	July 9, 2020
Jupyterhub behind ingress controller Zero to JupyterHub on Kubernetes how-to , help-wanted	4	881	March 23, 2023
I am unable to create a gitlab integration with the recommended jupyterhub config Zero to JupyterHub on Kubernetes jupyterhub , how-to , help-wanted	3	1283	March 3, 2022
Connectivity Issue with JupyterHub on Kubernetes: Cannot Access Services by Internal DNS Name Zero to JupyterHub on Kubernetes	4	96	March 26, 2024
Advice Regarding JupyterHub using Kubernetes Authentication Zero to JupyterHub on Kubernetes jupyterhub , how-to	1	450	October 3, 2020

JupyterHub stuck after the first login on Kubernetes (with NGINX Ingress Controller)

Related Topics