Redirect error when spawning notebook in Kubernetes

JupyterHub Zero to JupyterHub on Kubernetes
1

Hello everyone!

I am installing JupyterHub 0.9.0 on a custom Kubernetes deployment, which has its own certificate provider, but does not provide external IPs. Because of this, the proxy-public L4 Balancer does not have a public IP, so in order to access the Hub, we modify the jupyterhub L7 ingress to point directly to the /hub (and then we access to the specific hostname with HTTPs).

We are, then, able to access the login (configured with Github accounts), but, when the time comes for spawning the server/environment, we receive a HTTP_TOO_MANY_REDIRECTS error. Logs from the hub workload are:

[I 2020-06-22 07:50:53.775 JupyterHub base:707] User logged in: adr-arroyo
[I 2020-06-22 07:50:53.793 JupyterHub reflector:199] watching for pods with label selector='component=singleuser-server' in namespace adri-jupyter
[I 2020-06-22 07:50:53.887 JupyterHub reflector:199] watching for events with field selector='involvedObject.kind=Pod' in namespace adri-jupyter
[I 2020-06-22 07:50:53.889 JupyterHub log:174] 302 GET /hub/oauth_callback?code=[secret]&state=[secret] -> /hub/spawn (@X.X.X.X) 674.59ms
[I 2020-06-22 07:50:54.004 JupyterHub log:174] 200 GET /hub/spawn (adr-arroyo@X.X.X.X) 60.19ms
[I 2020-06-22 07:50:58.039 JupyterHub log:174] 200 GET /hub/health (@10.42.2.1) 1.01ms
[W 2020-06-22 07:50:59.741 JupyterHub base:950] User adr-arroyo is slow to start (timeout=0)
[I 2020-06-22 07:50:59.756 JupyterHub log:174] 302 POST /hub/spawn -> /hub/spawn-pending/adr-arroyo (adr-arroyo@X.X.X.X) 123.08ms
[I 2020-06-22 07:50:59.827 JupyterHub pages:347] adr-arroyo is pending spawn
[I 2020-06-22 07:50:59.831 JupyterHub log:174] 200 GET /hub/spawn-pending/adr-arroyo (adr-arroyo@X.X.X.X) 8.54ms
[I 2020-06-22 07:51:06.759 JupyterHub base:855] User adr-arroyo took 7.114 seconds to start
[I 2020-06-22 07:51:06.759 JupyterHub proxy:262] Adding user adr-arroyo to proxy /user/adr-arroyo/ => http://10.42.1.155:8888
[I 2020-06-22 07:51:06.761 JupyterHub log:174] 200 GET /hub/api (@10.42.1.155) 0.90ms
[I 2020-06-22 07:51:06.763 JupyterHub users:609] Server adr-arroyo is ready
[I 2020-06-22 07:51:06.764 JupyterHub log:174] 200 GET /hub/api/users/adr-arroyo/server/progress (adr-arroyo@X.X.X.X) 6765.71ms
[I 2020-06-22 07:51:06.831 JupyterHub log:174] 200 POST /hub/api/users/adr-arroyo/activity (adr-arroyo@10.42.1.155) 47.78ms
[I 2020-06-22 07:51:06.842 JupyterHub log:174] 302 GET /hub/spawn-pending/adr-arroyo -> /user/adr-arroyo/ (adr-arroyo@X.X.X.X) 9.73ms
[I 2020-06-22 07:51:06.891 JupyterHub log:174] 302 GET /user/adr-arroyo/ -> /hub/user/adr-arroyo/ (@X.X.X.X) 0.67ms
[I 2020-06-22 07:51:06.947 JupyterHub log:174] 302 GET /hub/user/adr-arroyo/ -> /user/adr-arroyo/ (adr-arroyo@X.X.X.X) 4.60ms
[I 2020-06-22 07:51:06.997 JupyterHub log:174] 302 GET /user/adr-arroyo/ -> /hub/user/adr-arroyo/ (@X.X.X.X) 0.63ms
[I 2020-06-22 07:51:07.048 JupyterHub log:174] 302 GET /hub/user/adr-arroyo/ -> /user/adr-arroyo/ (adr-arroyo@X.X.X.X) 4.30ms
[I 2020-06-22 07:51:07.099 JupyterHub log:174] 302 GET /user/adr-arroyo/ -> /hub/user/adr-arroyo/ (@X.X.X.X) 0.66ms
[I 2020-06-22 07:51:07.150 JupyterHub log:174] 302 GET /hub/user/adr-arroyo/ -> /user/adr-arroyo/ (adr-arroyo@X.X.X.X) 4.33ms
[I 2020-06-22 07:51:07.197 JupyterHub log:174] 302 GET /user/adr-arroyo/ -> /hub/user/adr-arroyo/ (@X.X.X.X) 0.62ms
[I 2020-06-22 07:51:07.246 JupyterHub log:174] 302 GET /hub/user/adr-arroyo/ -> /user/adr-arroyo/ (adr-arroyo@X.X.X.X) 4.21ms
[I 2020-06-22 07:51:07.293 JupyterHub log:174] 302 GET /user/adr-arroyo/ -> /hub/user/adr-arroyo/ (@X.X.X.X) 0.62ms
[I 2020-06-22 07:51:07.342 JupyterHub log:174] 302 GET /hub/user/adr-arroyo/ -> /user/adr-arroyo/ (adr-arroyo@X.X.X.X) 4.36ms
[I 2020-06-22 07:51:07.391 JupyterHub log:174] 302 GET /user/adr-arroyo/ -> /hub/user/adr-arroyo/ (@X.X.X.X) 0.62ms
[I 2020-06-22 07:51:07.441 JupyterHub log:174] 302 GET /hub/user/adr-arroyo/ -> /user/adr-arroyo/ (adr-arroyo@X.X.X.X) 4.76ms
[I 2020-06-22 07:51:07.489 JupyterHub log:174] 302 GET /user/adr-arroyo/ -> /hub/user/adr-arroyo/ (@X.X.X.X) 0.64ms
[I 2020-06-22 07:51:07.538 JupyterHub log:174] 302 GET /hub/user/adr-arroyo/ -> /user/adr-arroyo/ (adr-arroyo@X.X.X.X) 4.23ms
[I 2020-06-22 07:51:07.586 JupyterHub log:174] 302 GET /user/adr-arroyo/ -> /hub/user/adr-arroyo/ (@X.X.X.X) 0.63ms
[I 2020-06-22 07:51:07.636 JupyterHub log:174] 302 GET /hub/user/adr-arroyo/ -> /user/adr-arroyo/ (adr-arroyo@X.X.X.X) 4.34ms
.....

ending up in the TOO_MANY_REDIRECTS error I just mentioned. Logs from proxy-public workload are:

07:44:26.532 [ConfigProxy] info: Adding route / -> http://10.43.75.108:8081
07:44:26.533 [ConfigProxy] info: Route added / -> http://10.43.75.108:8081
07:44:26.533 [ConfigProxy] info: 201 POST /api/routes/
07:45:26.536 [ConfigProxy] info: 200 GET /api/routes
07:46:26.536 [ConfigProxy] info: 200 GET /api/routes
07:47:26.536 [ConfigProxy] info: 200 GET /api/routes
07:48:26.536 [ConfigProxy] info: 200 GET /api/routes
07:49:26.536 [ConfigProxy] info: 200 GET /api/routes
07:50:26.535 [ConfigProxy] info: 200 GET /api/routes
07:51:06.762 [ConfigProxy] info: Adding route /user/adr-arroyo -> http://10.42.1.155:8888
07:51:06.763 [ConfigProxy] info: Route added /user/adr-arroyo -> http://10.42.1.155:8888
07:51:06.763 [ConfigProxy] info: 201 POST /api/routes/user/adr-arroyo

Any idea how to address this problem?

2

Hello,

Just found out that disabling the proxy https configuration, but leaving the ingress tls configuration I avoid the error. But why is it?

Do the proxy and the ingress configuration get somehow mixed??

For the record, this is my current yaml (partially):

proxy:
  secretToken: ""
  # https:
  #   hosts:
  #     - jupyterhub-test.mydomain.com
  #   type: secret
  #   secret:
  #     name: jupyterhub-cert

ingress:
  enabled: true
  hosts:
    - jupyterhub-test.mydomain.com
  annotations:
    kubernetes.io/tls-acme: "true"
  tls:
   - hosts:
      - jupyterhub-test.mydomain.com
     secretName: jupyterhub-cert