Weird problem when deploying Z2JK version 0.10.6 to a GKE cluster managed by Rancher with ISTIO MTLS enabled between the ISTIO Ingress Gateway and the Configurable HTTP Proxy.
Problem occurs when the user tries to login to Jupyterhub. (Hub login page successfully displays)
- Login request is received by the ISTIO Ingress Gateway.
- ISTIO Ingress Gateway routes the request to the ISTIO side car container running in the Proxy pod.
- ISTIO side car container – running in the Proxy pod – routes the request to the CHP container.
- CHP Container routes the request to the Hub container running in the Hub pod. There is no ISTIO side car container installed in the Hub pod – just the Hub container is installed in the Hub pod.
- Hub container authenticates the user using an oauth spawner.
- Hub container spawns the Single User (jupyter lab) Server pod.
- Notebook container within the Single User (jupyter lab) server pod starts. See attached log file snippet from the spawned Single User Pod.
- Notebook container tries to send a redirect /user//lab? back to the browser.
- Browser NEVER gets the redirect request. Log file from CHP container reports a 503 socket hang-up error. See attached log file snippet from the CHP container.
I am also attaching the relevant log file snippet from the Hub container and an image of browser page with the 503 (notice that the url address bar does not show the redirected url from step 8.
Any help debugging this problem is greatly appreciated