Separate CHP gets wrong target info

JupyterHub Zero to JupyterHub on Kubernetes
#1

Hi, I’m running the CHP separate(in another pod, k8s) from the hub. At initialization everything is OK,
but after a while CHP updates its config with wrong target info.

api response right after pod initialization, when everything is OK:
call:

curl -H "Authorization: token $CONFIGPROXY_AUTH_TOKEN" http://localhost:8001/api/routes

response:

{"/":{"hub":true,"target":"http://jupyterhub:8081","jupyterhub":true,"last_activity":"2020-01-30T13:29:43.150Z"}}

after ~10 minutes, it makes an api call like:

13:41:51.986 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
13:41:51.986 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
13:41:51.980 [ConfigProxy] info: 200 GET /api/routes

then, proxy starts route requests to itself,

curl -H "Authorization: token $CONFIGPROXY_AUTH_TOKEN" http://localhost:8001/api/routes

response,

{"/":{"hub":true,"target":"http://127.0.0.1:8081","jupyterhub":true,"last_activity":"2020-01-30T13:42:51.980Z"}}

command section of CHP container definition:

command:
  - configurable-http-proxy
  - --ip=0.0.0.0
  - --api-ip=0.0.0.0
  - --api-port=8001
  - --default-target=http://jupyterhub:8081
  - --error-target=http://jupyterhub:8081/hub/error
  - --port=8000
  - --log-level=debug

Any ideas?

#2

What does your JupyterHub pod log show for that moment in time? What does the hub believe its IP/hostname is? I think the hub will repair routes in the proxy if it thinks that they aren’t pointing to the right place. It does so every once in a while which could explain why it works at first and then happens about 10min after startup.

#3

Hi, thanks for your answer,

I noticed the same and I changed last_activity_interval config like below to get more frequent requests.

c.JupyterHub.last_activity_interval = 60

now, I’m getting same request in 60 second interval. It looks like hub sends correct address info but proxy uses 127.0.0.1 instead.

hub logs,

[W 2020-01-31 08:35:43.601 JupyterHub proxy:333] Updating default route http://127.0.0.1:8081 → http://jupyterhub:8081
[I 2020-01-31 08:35:43.601 JupyterHub proxy:399] Adding default route for Hub: / => http://jupyterhub:8081
[D 2020-01-31 08:35:43.601 JupyterHub proxy:765] Proxy: Fetching POST http://jupyterhub-proxy:8001/api/routes/

meanwhile proxy logs:

08:35:43.600 [ConfigProxy] info: 200 GET /api/routes
08:35:43.605 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
08:35:43.605 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
08:35:43.605 [ConfigProxy] info: 201 POST /api/routes/
08:36:43.601 [ConfigProxy] info: 200 GET /api/routes
08:36:43.607 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
08:36:43.607 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
08:36:43.608 [ConfigProxy] info: 201 POST /api/routes/
#4 
c.JupyterHub.cleanup_servers = False
c.JupyterHub.last_activity_interval = 60

c.JupyterHub.log_level = 'DEBUG'
c.JupyterHub.ip = 'jupyterhub'
c.JupyterHub.hub_connect_ip = 'jupyterhub'
c.JupyterHub.hub_connect_url = 'http://jupyterhub:8081'

c.ConfigurableHTTPProxy.should_start = False
c.ConfigurableHTTPProxy.auth_token = "******************"
c.ConfigurableHTTPProxy.api_url = "http://jupyterhub-proxy:8001"
#5

I think the c.JupyterHub.ip field has to be an IP and for a kubernetes setup should be '0.0.0.0' instead of a hostname.

One thing to check is to log in (with kubectl exec) to the hub pod and see what jupyterhub resolves to. My guess is that it gives you 127.0.0.1

As a remark: we have z2jh.jupyter.org/ which is a whole project about setting up and maintaining a production JupyterHub on kubernetes. I think it would be a good idea to use that if you plan on running a hub for a longer time. Building something new by hand is a great way to learn how it all works though :slight_smile:

#6

Hi,
At first c.JupyterHub.ip, c.JupyterHub.hub_connect_ip and c.JupyterHub.hub_connect_url was not set but the behavior of the system was same. I tried a lot of combinations of that configs, I will try to 0.0.0.0 again, to be sure.

To be precise, we have OpenShift and jupyterhub hostname handled by a service that pointing to the Hub pod. When I ping the hostname (both from Hub and Proxy pods), it resolved as the service’s IP as expected.

I’m well aware of the site, excellent source. I check out very often but we are not allowed to use helm with OpenShift -Corporate IT rules are whole another story :slight_smile:

1 Like
#7

Have you checked how we configure the CHP in the Z2JH chart (with an IP or a service)? I don’t really have a good idea left beyond comparing how things are setup in Z2JH and your setup. especially if jupyterhub resolves to the IP of the service in both pods then I don’t know how the hub gets the idea to set it to 127.0.0.1.

Sorry to hear that :frowning: Have you tried the helm option to generate YAML (called something like “local render” or “render only”) and then kubectl apply -f that? Maybe even just temporarily to see what it does.

There is also https://github.com/jupyter-on-openshift which might help

#8

Hi,

Mystery solved!

I did not mention here but we are extending our hub images from jupyter-on-openshift images and never actually deep dive in the config.

Recently I noticed a function in the config file (https://github.com/jupyter-on-openshift/jupyterhub-quickstart/blob/a6e0ccf2e567dbb76130883eea2cd236f03414eb/jupyterhub_config.py) like below, to fix some kind minishift problem!

Just removed the function and everything worked as expected :slight_smile:

@wrapt.patch_function_wrapper(‘jupyterhub.proxy’, ‘ConfigurableHTTPProxy.add_route’)
def _wrapper_add_route(wrapped, instance, args, kwargs):
def _extract_args(routespec, target, data, *_args, **_kwargs):
return (routespec, target, data, _args, _kwargs)

routespec, target, data, _args, _kwargs = _extract_args(*args, **kwargs)

old = 'http://%s:%s' % (c.JupyterHub.hub_connect_ip, c.JupyterHub.hub_port)
new = 'http://127.0.0.1:%s' % c.JupyterHub.hub_port

if target.startswith(old):
    target = target.replace(old, new)

return wrapped(routespec, target, data, *_args, **_kwargs)