Issue access ipykernel with Jupyterhub behind Nginx Ingress

JupyterHub Zero to JupyterHub on Kubernetes
1

Hi Team,
We are having problem accessing the ipykernel for any user on JupyterHub.
The Python 3 (ipykernel) Kernel status goes into connecting state and remains “disconnected”.
This happens only when we host it behind a ingress object (ingress-nginx).
When we remove the ingress object and convert the service object to type NodePort, it works as expected and kernel status goes to “connected”

Existing Setup:
We have configured the jupyterhub application behind a service object of type Cluster IP and Ingress Object

  1. Configuration of service object of type Cluster IP.
    Configuration: `
    spec:
    clusterIP: 10.228.144.188
    clusterIPs:
  • 10.228.144.188
    internalTrafficPolicy: Cluster
    ipFamilies:
  • IPv4
    ipFamilyPolicy: SingleStack
    ports:
  • port: 8000
    protocol: TCP
    targetPort: http
    selector:
    app: jupyterhub
    component: proxy
    release: jupyterhub
    sessionAffinity: None
    type: ClusterIP`
  1. Configuration of ingress object.
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
    nginx.org/websocket-services: proxy-public
  name: test-jupyterhub-ingress
  namespace: jupyterhub
spec:
  ingressClassName: nginx
  rules:
  - host: kubeingress-ml.test.eng.com
    http:
      paths:
      - backend:
          service:
            name: proxy-public
            port:
              number: 8000
        path: /hub
        pathType: ImplementationSpecific
      - backend:
          service:
            name: proxy-public
            port:
              number: 8000
        path: /user
        pathType: ImplementationSpecific

Any pointer or guidance would be helpful and appreciated as we are blocked here.

2

Perhaps your deployed Nginx ingress (controller) has an incorrect configuration. Can you check the logs for your Nginx pods and share them here? Are you using the default Nginx ingress Helm chart configuration, or have you customised it?

3

Hey Manics,
Thank you for responding,
Yeah we are using the same ingress config that was part of the jupyterhub helm chart.


ingress:
  enabled: true
  annotations: {}
  ingressClassName:
  hosts: [kubeingress-ml.test.eng.com]
  pathSuffix: hub
  pathType: Prefix
  tls: []

Here we are using pathSuffix as hub since our webhook configuration does not allow us to set “/” as path in ingress. also, we had to add one more path for “/user” since kubeingress-ml.test.eng.com/hub/login would get redirected to kubeingress-ml.test.eng.com/user/admin/ and would fail with 403 not found.

Please find the logs below for ingress-nginx and jupyter-admin pod logs.
Also, let us know if we are missing any annotations in ingress configs.
INGRESS NGINX LOGS

10.45.181.144 - - [01/Aug/2024:17:31:10 +0000] "GET /hub/static/components/font-awesome/fonts/fontawesome-webfont.woff2?v=4.7.0 HTTP/1.1" 200 77160 "https://kubeingress-ml.test.eng.com/hub/static/css/style.min.css?v=01598a5386176f0279952a3b9632a07e7fce9a12aa53108973c83be9ec3473e7a59354876fab64bfeb01892eb503870183707aa03f207d7a94845ca7980c3819" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1137 0.006 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 77160 0.006 200 959a74adaf0d4218e4e3fa791cb12020
10.244.72.192 - - [01/Aug/2024:17:31:10 +0000] "POST /hub/login?next=%2Fhub%2F HTTP/1.1" 302 0 "https://kubeingress-ml.test.eng.com/hub/login?next=%2Fhub%2F" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1495 0.015 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 0 0.015 302 b76e92baa4e4b5e295fbefed62cea121
10.244.72.192 - - [01/Aug/2024:17:31:10 +0000] "GET /hub/ HTTP/1.1" 302 0 "https://kubeingress-ml.test.eng.com/hub/login?next=%2Fhub%2F" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1377 0.052 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 0 0.052 302 f23239b5de7f1e08c83232bab301d537
10.244.72.192 - - [01/Aug/2024:17:31:10 +0000] "GET /user/admin/ HTTP/1.1" 302 0 "https://kubeingress-ml.test.eng.com/hub/login?next=%2Fhub%2F" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 927 0.005 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 0 0.005 302 1235e8739016bbcfea9a9e8ec981f872
10.244.72.192 - - [01/Aug/2024:17:31:11 +0000] "GET /user/admin/lab? HTTP/1.1" 302 0 "https://kubeingress-ml.test.eng.com/hub/login?next=%2Fhub%2F" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 931 0.007 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 0 0.006 302 a363df4d33460bca5d9f9ee9d7be7e54
10.244.72.192 - - [01/Aug/2024:17:31:11 +0000] "GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogImI0ZjU0YTVjOGUyYTQyOGU5MDljMGQ3OTY2OTc0YzhlIiwgIm5leHRfdXJsIjogIi91c2VyL3BzaGFoL2xhYj8ifQ HTTP/1.1" 302 0 "https://kubeingress-ml.test.eng.com/hub/login?next=%2Fhub%2F" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1603 0.058 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 0 0.058 302 d028b1be579dd3631271c0f964a9c7b6
10.244.72.192 - - [01/Aug/2024:17:31:11 +0000] "GET /user/admin/oauth_callback?code=ZVmoLIqqxmCpRXdloYUP0Chi3wKLDz&state=eyJ1dWlkIjogImI0ZjU0YTVjOGUyYTQyOGU5MDljMGQ3OTY2OTc0YzhlIiwgIm5leHRfdXJsIjogIi91c2VyL3BzaGFoL2xhYj8ifQ HTTP/1.1" 302 0 "https://kubeingress-ml.test.eng.com/hub/login?next=%2Fhub%2F" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1384 0.192 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 0 0.191 302 9ce03cbb938ac65349172684ce322a9d
10.244.72.192 - - [01/Aug/2024:17:31:12 +0000] "GET /user/admin/lab? HTTP/1.1" 200 3872 "https://kubeingress-ml.test.eng.com/hub/login?next=%2Fhub%2F" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1422 0.015 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 3872 0.016 200 0e9b0b820a6313bca87b2d25fa207edb
10.244.72.192 - - [01/Aug/2024:17:31:16 +0000] "GET /user/admin/static/lab/812.d5d36066012c892c6271.js?v=d5d36066012c892c6271 HTTP/1.1" 200 11956 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1259 0.005 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 11956 0.005 200 b717822f3e62ec0cd6aa78aa2dc92fe5
10.244.72.192 - - [01/Aug/2024:17:31:17 +0000] "GET /user/admin/lab/extensions/jupyterlab_pygments/static/747.67662283a5707eeb4d4c.js?v=67662283a5707eeb4d4c HTTP/1.1" 200 8351 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1290 0.007 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 8351 0.007 200 2eb6ca0ad8e397f9891560e00b8c3aee
10.45.181.144 - - [01/Aug/2024:17:31:17 +0000] "GET /user/admin/lab/extensions/jupyterlab_pygments/static/568.1e2faa2ba0bbe59c4780.js?v=1e2faa2ba0bbe59c4780 HTTP/1.1" 200 224 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1290 0.008 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 224 0.008 200 f530819c01bd4ee7c3fa712d669b0b8e
10.45.181.144 - - [01/Aug/2024:17:31:17 +0000] "GET /user/admin/static/favicons/favicon.ico HTTP/1.1" 200 32038 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1285 0.006 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 32038 0.006 200 2b949428df636ed17999de94a02bfb2a
10.45.181.144 - - [01/Aug/2024:17:31:17 +0000] "GET /user/admin/api/kernelspecs?1722533477246 HTTP/1.1" 200 531 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1629 0.008 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 531 0.008 200 4db3f2bc024336091495623b4e2c8787
10.244.72.192 - - [01/Aug/2024:17:31:17 +0000] "GET /user/admin/api/me?1722533477247 HTTP/1.1" 200 149 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1620 0.011 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 149 0.011 200 417206caa71bafa0bc508bb64948cc10
10.244.100.192 - - [01/Aug/2024:17:31:17 +0000] "GET /user/admin/lab/api/settings?1722533477251 HTTP/1.1" 200 115014 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1630 0.088 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 115014 0.088 200 811d223dd74b4e04062c17b20c2095c1
10.45.181.145 - - [01/Aug/2024:17:31:17 +0000] "GET /user/admin/api/sessions?1722533477257 HTTP/1.1" 200 2944 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1626 0.080 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 2944 0.080 200 ad47f8d8ec858404afbf3e969d441149
10.244.100.192 - - [01/Aug/2024:17:31:17 +0000] "GET /user/admin/api/kernels?1722533477257 HTTP/1.1" 200 1312 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1625 0.080 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 1312 0.081 200 dd85da2f965e9d3083f3aa2889ce4b5e
10.45.181.144 - - [01/Aug/2024:17:31:17 +0000] "GET /user/admin/api/terminals?1722533477258 HTTP/1.1" 200 2 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1627 0.058 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 2 0.058 200 9881041fd578282e6579180a8b751ace
10.45.181.144 - - [01/Aug/2024:17:31:17 +0000] "GET /user/admin/api/me?1722533477517 HTTP/1.1" 200 149 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1620 0.007 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 149 0.007 200 5a344e51b080adaf39544f9d83167b3d
10.244.72.192 - - [01/Aug/2024:17:31:17 +0000] "GET /user/admin/api/kernelspecs?1722533477519 HTTP/1.1" 200 531 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1629 0.009 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 531 0.009 200 fcd1d8a33ce644f2bd432d838b2b4564
10.244.100.192 - - [01/Aug/2024:17:31:18 +0000] "GET /user/admin/lab/api/translations/default?1722533477920 HTTP/1.1" 200 63 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1642 0.030 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 63 0.029 200 0db71c7e199508bcd8feed9250b9e601
10.244.100.192 - - [01/Aug/2024:17:31:18 +0000] "GET /user/admin/api/config/jupyterlabapputilsextensionannouncements?1722533478224 HTTP/1.1" 200 2 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1665 0.006 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 2 0.006 200 4067c02284a901000738401292e00597
10.45.181.144 - - [01/Aug/2024:17:31:18 +0000] "GET /user/admin/lab/api/build?1722533478249 HTTP/1.1" 200 35 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1627 0.006 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 35 0.006 200 ca48dfa169da479b794894e08ba6ce8b
10.244.72.192 - - [01/Aug/2024:17:31:18 +0000] "GET /user/admin/lsp/status?1722533478284 HTTP/1.1" 200 9854 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1624 0.007 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 9854 0.006 200 4e6a7c6639ae657f99aa6b9cda841c0f
10.45.181.144 - - [01/Aug/2024:17:31:18 +0000] "GET /user/admin/lab/api/settings/@jupyterlab/codemirror-extension:plugin?1722533478332 HTTP/1.1" 200 541 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1670 0.017 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 541 0.017 200 54095de62c7b083d2f0c392bd876a89c
10.244.100.192 - - [01/Aug/2024:17:31:18 +0000] "GET /user/admin/lab/api/translations?1722533478295 HTTP/1.1" 200 84 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1634 0.053 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 84 0.053 200 32e7f7731962c99f4cc5f3b168be7436
10.45.181.145 - - [01/Aug/2024:17:31:18 +0000] "GET /user/admin/api/contents?content=1&hash=0&1722533478299 HTTP/1.1" 200 6544 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1643 0.061 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 6544 0.061 200 6ef0233f8ba4539d0ea85c5716fadd8b
10.244.100.192 - - [01/Aug/2024:17:31:18 +0000] "GET /user/admin/lab/api/themes/@jupyterlab/theme-light-extension/index.css HTTP/1.1" 200 16550 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1270 0.007 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 16550 0.007 200 cfe199077b5db897e65b18cce36c8481
10.45.181.144 - - [01/Aug/2024:17:31:18 +0000] "GET /user/admin/lab/api/settings/@jupyterlab/notebook-extension:panel?1722533478336 HTTP/1.1" 200 2583 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1667 0.016 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 2583 0.016 200 bd8e3a3d0c4a8e75acaeeb7c5c06972a
10.45.181.145 - - [01/Aug/2024:17:31:18 +0000] "GET /user/admin/lab/api/workspaces/default?1722533478673 HTTP/1.1" 200 2070 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1640 0.008 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 2070 0.008 200 842514e2f548df6cf1c66b1b01d1164b
10.244.100.192 - - [01/Aug/2024:17:31:18 +0000] "GET /user/admin/kernelspecs/python3/logo-svg.svg HTTP/1.1" 200 9605 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1290 0.007 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 9605 0.007 200 2e695309f4a537cc8ed97d62294d2e1b
10.45.181.144 - - [01/Aug/2024:17:31:18 +0000] "GET /user/admin/api/contents?content=1&hash=0&1722533478694 HTTP/1.1" 200 6544 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1643 0.023 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 6544 0.022 200 648ec118b43009149e5ddfc5811a62e6
10.244.100.192 - - [01/Aug/2024:17:31:19 +0000] "GET /user/admin/api/contents/Untitled17.ipynb?content=0&hash=0&1722533478949 HTTP/1.1" 200 286 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1660 0.006 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 286 0.006 200 c296d49be0e230258347a1c1beb3786c
10.45.181.144 - - [01/Aug/2024:17:31:19 +0000] "GET /user/admin/api/contents/Untitled13.ipynb?content=0&hash=0&1722533478949 HTTP/1.1" 200 286 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1660 0.015 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 286 0.015 200 24ed740cd871f76e56a250bcf9efaed9
10.45.181.145 - - [01/Aug/2024:17:31:19 +0000] "GET /user/admin/api/contents/Untitled12.ipynb?content=0&hash=0&1722533478948 HTTP/1.1" 200 286 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1660 0.008 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 286 0.008 200 a4e20cca0bf2fa393c74b0eaf85e6fed
10.244.100.192 - - [01/Aug/2024:17:31:19 +0000] "GET /user/admin/api/contents/Untitled14.ipynb?content=0&hash=0&1722533478949 HTTP/1.1" 200 286 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1660 0.017 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 286 0.017 200 9dcd3f78167f0363d1e061a306fe8664
10.244.72.192 - - [01/Aug/2024:17:31:19 +0000] "GET /user/admin/api/contents/Untitled15.ipynb?content=0&hash=0&1722533478949 HTTP/1.1" 200 286 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1660 0.018 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 286 0.018 200 e0d131cc8c1b9626eb06796ff46f7851
10.45.181.144 - - [01/Aug/2024:17:31:19 +0000] "GET /user/admin/api/contents/Untitled18.ipynb?content=0&hash=0&1722533478949 HTTP/1.1" 200 286 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1660 0.003 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 286 0.003 200 8409d7174deca5c46b7f1dcb7c847f95
10.244.100.192 - - [01/Aug/2024:17:31:19 +0000] "GET /user/admin/api/contents/Untitled20.ipynb?content=0&hash=0&1722533478950 HTTP/1.1" 200 286 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1660 0.004 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 286 0.005 200 ea058f5fcf1713b2a29a68bd1b30796e
10.45.181.145 - - [01/Aug/2024:17:31:19 +0000] "GET /user/admin/api/contents/Untitled19.ipynb?content=0&hash=0&1722533478949 HTTP/1.1" 200 286 "https://kubeingress-ml.test.eng.com/user/admin/lab?" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1660 0.007 [jupyterhub-proxy-public-http] [] 10.244.100.253:8000 286 0.007 200 2eb6667ceb61fe5667a52aa6202a1b05

JUPYTER-ADMIN.log

[I 2024-08-01 17:31:10.826 ServerApp] 302 GET /user/admin/ -> /user/admin/lab? (@10.244.72.192) 0.74ms
[I 2024-08-01 17:31:11.092 ServerApp] 302 GET /user/admin/lab? -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-admin&redirect_uri=%2Fuser%2Fadmin%2Foauth_callback&response_type=code&state=[secret] (@10.244.72.192) 2.88ms
[I 2024-08-01 17:31:11.867 ServerApp] Logged-in user {'admin': False, 'name': 'admin', 'kind': 'user', 'groups': [], 'session_id': '445247067f1042f69ba519963d6833d1', 'scopes': ['access:servers!server=admin/', 'read:users:groups!user=admin', 'read:users:name!user=admin']}
[I 2024-08-01 17:31:11.868 ServerApp] Setting new xsrf cookie for b'445247067f1042f69ba519963d6833d1:d78cc95ea6e7dc660e2f19e6720a0d5e84bb2dd119d8bb1052bc4365f6dfff84' {'path': '/user/admin/'}
[I 2024-08-01 17:31:11.870 ServerApp] 302 GET /user/admin/oauth_callback?code=[secret]&state=[secret] -> /user/admin/lab? (admin@10.244.72.192) 187.57ms
[I 2024-08-01 17:31:12.148 ServerApp] 200 GET /user/admin/lab? (admin@10.244.72.192) 9.95ms
[I 2024-08-01 17:31:17.412 ServerApp] 200 GET /user/admin/api/kernelspecs?1722533477246 (admin@10.45.181.144) 4.25ms
[I 2024-08-01 17:31:17.416 ServerApp] 200 GET /user/admin/api/me?1722533477247 (admin@10.244.72.192) 2.01ms
[I 2024-08-01 17:31:17.493 ServerApp] 200 GET /user/admin/lab/api/settings?1722533477251 (admin@10.244.100.192) 76.37ms
[I 2024-08-01 17:31:17.495 ServerApp] 200 GET /user/admin/api/sessions?1722533477257 (admin@10.45.181.145) 1.27ms
[I 2024-08-01 17:31:17.497 ServerApp] 200 GET /user/admin/api/kernels?1722533477257 (admin@10.244.100.192) 0.83ms
[I 2024-08-01 17:31:17.498 ServerApp] 200 GET /user/admin/api/terminals?1722533477258 (admin@10.45.181.144) 0.74ms
[I 2024-08-01 17:31:17.682 ServerApp] 200 GET /user/admin/api/me?1722533477517 (admin@10.45.181.144) 2.02ms
[I 2024-08-01 17:31:17.686 ServerApp] 200 GET /user/admin/api/kernelspecs?1722533477519 (admin@10.244.72.192) 2.49ms
[I 2024-08-01 17:31:18.106 ServerApp] 200 GET /user/admin/lab/api/translations/default?1722533477920 (admin@10.244.100.192) 25.50ms
[I 2024-08-01 17:31:18.385 ServerApp] 200 GET /user/admin/api/config/jupyterlabapputilsextensionannouncements?1722533478224 (admin@10.244.100.192) 2.69ms
[W 2024-08-01 17:31:18.409 LabApp] Could not determine jupyterlab build status without nodejs
[I 2024-08-01 17:31:18.409 ServerApp] 200 GET /user/admin/lab/api/build?1722533478249 (admin@10.45.181.144) 2.71ms
[I 2024-08-01 17:31:18.446 ServerApp] 200 GET /user/admin/lsp/status?1722533478284 (admin@10.244.72.192) 2.73ms
[I 2024-08-01 17:31:18.504 ServerApp] 200 GET /user/admin/lab/api/settings/@jupyterlab/codemirror-extension:plugin?1722533478332 (admin@10.45.181.144) 10.87ms
[I 2024-08-01 17:31:18.508 ServerApp] 200 GET /user/admin/lab/api/translations?1722533478295 (admin@10.244.100.192) 46.22ms
[I 2024-08-01 17:31:18.516 ServerApp] 200 GET /user/admin/api/contents?content=1&hash=0&1722533478299 (admin@10.45.181.145) 57.04ms
[I 2024-08-01 17:31:18.683 ServerApp] 200 GET /user/admin/lab/api/settings/@jupyterlab/notebook-extension:panel?1722533478336 (admin@10.45.181.144) 10.35ms
[I 2024-08-01 17:31:18.839 ServerApp] 200 GET /user/admin/lab/api/workspaces/default?1722533478673 (admin@10.45.181.145) 3.45ms
[I 2024-08-01 17:31:18.874 ServerApp] 200 GET /user/admin/api/contents?content=1&hash=0&1722533478694 (admin@10.45.181.144) 19.00ms
[I 2024-08-01 17:31:19.116 ServerApp] 200 GET /user/admin/api/contents/Untitled12.ipynb?content=0&hash=0&1722533478948 (admin@10.45.181.145) 3.16ms
[I 2024-08-01 17:31:19.121 ServerApp] 200 GET /user/admin/api/contents/Untitled17.ipynb?content=0&hash=0&1722533478949 (admin@10.244.100.192) 3.01ms
[I 2024-08-01 17:31:19.123 ServerApp] 200 GET /user/admin/api/contents/Untitled13.ipynb?content=0&hash=0&1722533478949 (admin@10.45.181.144) 3.95ms
[I 2024-08-01 17:31:19.126 ServerApp] 200 GET /user/admin/api/contents/Untitled14.ipynb?content=0&hash=0&1722533478949 (admin@10.244.100.192) 2.19ms
[I 2024-08-01 17:31:19.127 ServerApp] 200 GET /user/admin/api/contents/Untitled15.ipynb?content=0&hash=0&1722533478949 (admin@10.244.72.192) 3.45ms
[I 2024-08-01 17:31:19.131 ServerApp] 200 GET /user/admin/api/contents/Untitled18.ipynb?content=0&hash=0&1722533478949 (admin@10.45.181.144) 1.39ms
[I 2024-08-01 17:31:19.384 ServerApp] 200 GET /user/admin/api/contents/Untitled20.ipynb?content=0&hash=0&1722533478950 (admin@10.244.100.192) 1.46ms
[I 2024-08-01 17:31:19.387 ServerApp] 200 GET /user/admin/api/contents/Untitled19.ipynb?content=0&hash=0&1722533478949 (admin@10.45.181.145) 1.63ms
[I 2024-08-01 17:31:19.497 ServerApp] 200 GET /user/admin/api/contents?content=1&hash=0&1722533479284 (admin@10.45.181.144) 52.76ms
[I 2024-08-01 17:31:19.716 ServerApp] 200 GET /user/admin/lab/api/settings?ids_only=true&1722533479540 (admin@10.45.181.145) 12.17ms
[I 2024-08-01 17:31:19.776 ServerApp] 200 GET /user/admin/api/contents/Untitled14.ipynb?type=notebook&content=1&hash=1&1722533479554 (admin@10.244.72.192) 57.21ms
[I 2024-08-01 17:31:19.813 ServerApp] 200 GET /user/admin/api/contents/Untitled12.ipynb?type=notebook&content=1&hash=1&1722533479554 (admin@10.244.100.192) 94.29ms
[I 2024-08-01 17:31:19.818 ServerApp] 200 GET /user/admin/lab/api/update?1722533479542 (admin@10.244.100.192) 101.20ms
[I 2024-08-01 17:31:19.862 ServerApp] 200 GET /user/admin/api/contents/Untitled15.ipynb?type=notebook&content=1&hash=1&1722533479555 (admin@10.45.181.144) 47.35ms
[I 2024-08-01 17:31:19.945 ServerApp] 200 GET /user/admin/api/nbconvert?1722533479266 (admin@10.45.181.144) 517.33ms
[I 2024-08-01 17:31:20.018 ServerApp] 200 GET /user/admin/api/contents/Untitled17.ipynb?type=notebook&content=1&hash=1&1722533479555 (admin@10.45.181.145) 37.95ms
[I 2024-08-01 17:31:20.072 ServerApp] 200 GET /user/admin/api/contents/Untitled13.ipynb?type=notebook&content=1&hash=1&1722533479555 (admin@10.244.72.192) 41.10ms
[I 2024-08-01 17:31:20.125 ServerApp] 200 GET /user/admin/api/contents/Untitled20.ipynb?type=notebook&content=1&hash=1&1722533479555 (admin@10.244.100.192) 49.86ms
[I 2024-08-01 17:31:20.166 ServerApp] 200 GET /user/admin/api/contents/Untitled18.ipynb?type=notebook&content=1&hash=1&1722533479555 (admin@10.244.100.192) 88.23ms
[I 2024-08-01 17:31:20.205 ServerApp] 200 GET /user/admin/api/contents/Untitled19.ipynb?type=notebook&content=1&hash=1&1722533479555 (admin@10.45.181.144) 76.66ms
[I 2024-08-01 17:31:20.209 ServerApp] 200 GET /user/admin/api/contents/Untitled14.ipynb/checkpoints?1722533479883 (admin@10.45.181.144) 3.28ms
[I 2024-08-01 17:31:20.287 ServerApp] 200 GET /user/admin/api/contents/Untitled14.ipynb/checkpoints?1722533479883 (admin@10.45.181.145) 3.48ms
[I 2024-08-01 17:31:20.335 ServerApp] 200 GET /user/admin/api/contents/Untitled14.ipynb/checkpoints?1722533479885 (admin@10.244.72.192) 4.04ms
[I 2024-08-01 17:31:20.395 ServerApp] 200 GET /user/admin/api/contents/Untitled12.ipynb/checkpoints?1722533479921 (admin@10.244.100.192) 3.14ms
[I 2024-08-01 17:31:20.433 ServerApp] 200 GET /user/admin/api/contents/Untitled12.ipynb/checkpoints?1722533479921 (admin@10.244.100.192) 4.24ms
[I 2024-08-01 17:31:20.473 ServerApp] 200 GET /user/admin/api/contents/Untitled12.ipynb/checkpoints?1722533479922 (admin@10.45.181.144) 7.86ms
[I 2024-08-01 17:31:20.474 ServerApp] 200 GET /user/admin/api/contents/Untitled15.ipynb/checkpoints?1722533479972 (admin@10.45.181.144) 5.33ms
[I 2024-08-01 17:31:20.554 ServerApp] 200 GET /user/admin/api/contents/Untitled15.ipynb/checkpoints?1722533479972 (admin@10.45.181.145) 3.27ms
[I 2024-08-01 17:31:20.596 ServerApp] 200 GET /user/admin/api/contents/Untitled15.ipynb/checkpoints?1722533479973 (admin@10.244.72.192) 3.64ms
[I 2024-08-01 17:31:20.662 ServerApp] 204 PUT /user/admin/lab/api/workspaces/default?1722533480098 (admin@10.244.100.192) 1.69ms
4

Can you check your browser console messages and network tabs, and show them to us?

Which nginx-ingress controller have you deployed, ingress-nginx or Nginx ingress, they’re not the same. Did you use the default configuration, or something custom?

5

Hey,

Browser Consul error logs

WebSocket connection to 'wss://kubeingress-ml.test.eng.com/user/admin/api/kernels/0f0296…af54-b759bbbfc4a1/channels?session_id=35e7001a-eaee-49d2-9200-a87116dfc88e' failed: 
_createSocket	@	jlab_core.23e8a6b191…8a6b191d1fa15c1b7:1

jlab_core.23e8a6b191…8a6b191d1fa15c1b7:1 Connection lost, reconnecting in 24 seconds.
_reconnect	@	jlab_core.23e8a6b191…8a6b191d1fa15c1b7:1
_onWSClose	@	jlab_core.23e8a6b191…8a6b191d1fa15c1b7:1
c	@	jlab_core.23e8a6b191…8a6b191d1fa15c1b7:1

WebSocket connection to 'wss://kubeingress-ml.test.eng.com/user/admin/api/kernels/67ce6e…a096-43affa50ed53/channels?session_id=92225965-a000-4e64-a71c-9ab98f2f28f3' failed: 
_createSocket	@	jlab_core.23e8a6b191…8a6b191d1fa15c1b7:1

jlab_core.23e8a6b191…8a6b191d1fa15c1b7:1 WebSocket connection to 'wss://kubeingress-ml.test.eng.com/user/admin/api/kernels/c70bf7…959d-81b6c14d4527/channels?session_id=892ede12-5687-4d7c-bf1d-cc83108818a1' failed: 
_createSocket	@	jlab_core.23e8a6b191…8a6b191d1fa15c1b7:1

jlab_core.23e8a6b191…8a6b191d1fa15c1b7:1 Connection lost, reconnecting in 57 seconds.
_reconnect	@	jlab_core.23e8a6b191…8a6b191d1fa15c1b7:1
_onWSClose	@	jlab_core.23e8a6b191…8a6b191d1fa15c1b7:1
c	@	jlab_core.23e8a6b191…8a6b191d1fa15c1b7:1

Can you elaborate on what all details do we need from network tab of browser.?

We are using nginx-ingress-controller.

From the Jupyterhub helm chart, the only modifications that we have made is added /hub and /user as path prefix. this is added since our org webhook does not allow us to keep path: “/”

Here is the working ingress config

    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "300"

  name: test-jupyterhub-ingress
  namespace: jupyterhub
spec:
  ingressClassName: nginx
  rules:
  - host: kubeingress-ml.test.eng.com
    http:
      paths:
      - backend:
          service:
            name: proxy-public
            port:
              number: 8000
        path: /hub
        pathType: ImplementationSpecific
      - backend:
          service:
            name: proxy-public
            port:
              number: 8000
        path: /user
        pathType: ImplementationSpecific
6

It sounds like your ingress may be blocking websocket connections so perhaps nginx-ingress-controller needs additional configuration? Advanced configuration with Annotations | NGINX Ingress Controller has some websockets annotations you could try

7

Okay, let me try and come back.

8

Found out the issue, we are using a custom loadbalancer for our ingress and that was blocking the websocket connections.
Thank for all the support and quick response.