Hi, I’m running the CHP separate(in another pod, k8s) from the hub. At initialization everything is OK,
but after a while CHP updates its config with wrong target info.
api response right after pod initialization, when everything is OK:
call:
curl -H "Authorization: token $CONFIGPROXY_AUTH_TOKEN" http://localhost:8001/api/routes
response:
{"/":{"hub":true,"target":"http://jupyterhub:8081","jupyterhub":true,"last_activity":"2020-01-30T13:29:43.150Z"}}
after ~10 minutes, it makes an api call like:
13:41:51.986 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
13:41:51.986 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
13:41:51.980 [ConfigProxy] info: 200 GET /api/routes
then, proxy starts route requests to itself,
curl -H "Authorization: token $CONFIGPROXY_AUTH_TOKEN" http://localhost:8001/api/routes
response,
{"/":{"hub":true,"target":"http://127.0.0.1:8081","jupyterhub":true,"last_activity":"2020-01-30T13:42:51.980Z"}}
command section of CHP container definition:
command:
- configurable-http-proxy
- --ip=0.0.0.0
- --api-ip=0.0.0.0
- --api-port=8001
- --default-target=http://jupyterhub:8081
- --error-target=http://jupyterhub:8081/hub/error
- --port=8000
- --log-level=debug
Any ideas?
What does your JupyterHub pod log show for that moment in time? What does the hub believe its IP/hostname is? I think the hub will repair routes in the proxy if it thinks that they aren’t pointing to the right place. It does so every once in a while which could explain why it works at first and then happens about 10min after startup.
Hi, thanks for your answer,
I noticed the same and I changed last_activity_interval config like below to get more frequent requests.
c.JupyterHub.last_activity_interval = 60
now, I’m getting same request in 60 second interval. It looks like hub sends correct address info but proxy uses 127.0.0.1 instead.
hub logs,
[W 2020-01-31 08:35:43.601 JupyterHub proxy:333] Updating default route http://127.0.0.1:8081 → http://jupyterhub:8081
[I 2020-01-31 08:35:43.601 JupyterHub proxy:399] Adding default route for Hub: / => http://jupyterhub:8081
[D 2020-01-31 08:35:43.601 JupyterHub proxy:765] Proxy: Fetching POST http://jupyterhub-proxy:8001/api/routes/
meanwhile proxy logs:
08:35:43.600 [ConfigProxy] info: 200 GET /api/routes
08:35:43.605 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
08:35:43.605 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
08:35:43.605 [ConfigProxy] info: 201 POST /api/routes/
08:36:43.601 [ConfigProxy] info: 200 GET /api/routes
08:36:43.607 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
08:36:43.607 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
08:36:43.608 [ConfigProxy] info: 201 POST /api/routes/
c.JupyterHub.cleanup_servers = False
c.JupyterHub.last_activity_interval = 60
c.JupyterHub.log_level = 'DEBUG'
c.JupyterHub.ip = 'jupyterhub'
c.JupyterHub.hub_connect_ip = 'jupyterhub'
c.JupyterHub.hub_connect_url = 'http://jupyterhub:8081'
c.ConfigurableHTTPProxy.should_start = False
c.ConfigurableHTTPProxy.auth_token = "******************"
c.ConfigurableHTTPProxy.api_url = "http://jupyterhub-proxy:8001"
I think the c.JupyterHub.ip field has to be an IP and for a kubernetes setup should be '0.0.0.0' instead of a hostname.
One thing to check is to log in (with kubectl exec) to the hub pod and see what jupyterhub resolves to. My guess is that it gives you 127.0.0.1
As a remark: we have z2jh.jupyter.org/ which is a whole project about setting up and maintaining a production JupyterHub on kubernetes. I think it would be a good idea to use that if you plan on running a hub for a longer time. Building something new by hand is a great way to learn how it all works though 
Hi,
At first c.JupyterHub.ip, c.JupyterHub.hub_connect_ip and c.JupyterHub.hub_connect_url was not set but the behavior of the system was same. I tried a lot of combinations of that configs, I will try to 0.0.0.0 again, to be sure.
To be precise, we have OpenShift and jupyterhub hostname handled by a service that pointing to the Hub pod. When I ping the hostname (both from Hub and Proxy pods), it resolved as the service’s IP as expected.
I’m well aware of the site, excellent source. I check out very often but we are not allowed to use helm with OpenShift -Corporate IT rules are whole another story
1 Like
Have you checked how we configure the CHP in the Z2JH chart (with an IP or a service)? I don’t really have a good idea left beyond comparing how things are setup in Z2JH and your setup. especially if jupyterhub resolves to the IP of the service in both pods then I don’t know how the hub gets the idea to set it to 127.0.0.1.
Sorry to hear that 
Have you tried the helm option to generate YAML (called something like “local render” or “render only”) and then kubectl apply -f that? Maybe even just temporarily to see what it does.
There is also Jupyter on OpenShift · GitHub which might help
Hi,
Mystery solved!
I did not mention here but we are extending our hub images from jupyter-on-openshift images and never actually deep dive in the config.
Recently I noticed a function in the config file (https://github.com/jupyter-on-openshift/jupyterhub-quickstart/blob/a6e0ccf2e567dbb76130883eea2cd236f03414eb/jupyterhub_config.py) like below, to fix some kind minishift problem!
Just removed the function and everything worked as expected 
@wrapt.patch_function_wrapper(‘jupyterhub.proxy’, ‘ConfigurableHTTPProxy.add_route’)
def _wrapper_add_route(wrapped, instance, args, kwargs):
def _extract_args(routespec, target, data, *_args, **_kwargs):
return (routespec, target, data, _args, _kwargs)
routespec, target, data, _args, _kwargs = _extract_args(*args, **kwargs)
old = 'http://%s:%s' % (c.JupyterHub.hub_connect_ip, c.JupyterHub.hub_port)
new = 'http://127.0.0.1:%s' % c.JupyterHub.hub_port
if target.startswith(old):
target = target.replace(old, new)
return wrapped(routespec, target, data, *_args, **_kwargs)