Managed singleuser service on kubernetes with separated hub and proxy pods

JupyterHub Zero to JupyterHub on Kubernetes
,
1

I’m deploying a jupyterhub instance to run nbgrader with multiple courses using the Z2JH helm chart. On startup the hub starts a managed singleuser service for each course. The problem comes, when I want to make the services publicly available. There are 3 scenarios I’ve tried:

  1. Making the service listen on http://127.0.0.1:port
c.JupyterHub.services.append(
  {
    "name": course,
    "url": f"http://127.0.0.1:{port}",
    "command": ["jupyterhub-singleuser", f"--group=formgrade-{course}", f"--port={port}", "--debug"],
    "oauth_no_confirm": True,
  }
)
  • The first problem is, that the singleuser process is not accessible from the proxy pod, since it is listening only on 127.0.0.1.
  • The other problem is, that when starting the service, the proxy creates the following route.
    /services/course101 -> http://127.0.0.1:9000
    And since the hub and proxy pods are separated, they don’t have a shared localhost and all the traffic from /services/course101 gets routed to the proxy pod instead of the hub.
  1. Making the service listen on http://0.0.0.0:port
c.JupyterHub.services.append(
  {
    "name": course,
    "url": f"http://0.0.0.0:{port}",
    "command": ["jupyterhub-singleuser", f"--group=formgrade-{course}", f"--port={port}", "--debug"],
    "oauth_no_confirm": True,
  }
)
  • In this case the proxy can at least access the process on http://hub:port
  • But also the proxy creates the following route,
    /services/course101 -> http://hub-878b50d7d4-gzfpr:9000
    which is not working, since in kubernetes you can’t address pods by their names. It should route to the hub service instead.
  1. Making the service listen on http://hub:port
  • The service fails to start, since it cannot bind to hub:port, which represents the kubernetes service of the hub pod.

The desired route, created by the proxy would look like this:
/services/course101 -> http://hub:9000

So my question is, how can I make this work? Basically the problem is, that the “url” option in the service definition is used by the singleuser service as the URL to bind to and at the same time by the proxy to create the route. The only thing that kinda worked so far is to use the 2nd option and create the route manually (by making a call to the API of the proxy). Although this route gets periodically rewritten by the hub.

Thank you in advance for all your help or suggestions.

2

Dear @KrKO_o,

why do you need a singleuser service for each course?

The Z2JH Helm chart was design to have one hub service and one singleuser service. This is because of Kubernetes archtecture. Services will handle the domain redirection.

3

Hi @rgaiacs, thank you for the reply.

I have a separate service for each course, because “nbgrader” is documented that way.
https://nbgrader.readthedocs.io/en/stable/configuration/jupyterhub_config.html#example-use-case-multiple-classes

And also, I don’t really see a way to make it work with only one service, since the “nbgrader” extension itself is listing the classes by their service names.

But if there is a simpler way, please let me know.

The Z2JH Helm chart was design to have one hub service and one singleuser service. This is because of Kubernetes archtecture. Services will handle the domain redirection.

To be honest, I don’t really understand this argument. Why couldn’t you create multiple services? I don’t think I has anything to do with kubernetes, since all the managed services are running in the “hub” pod. And they also work as intended, only the proxy fails to route the traffic correctly.

4

If you’re using a hub managed service you don’t need the url parameter, if you omit it JupyterHub should automatically take care of the routing.

5

Hello @manics, thank you for the reply.

I tried to run in without the “url” parameter, but now the proxy doesn’t create the routes for the services. If I understand correctly, in the output of “https://<hub>/hub/api/routes” it should set the “prefix” parameter to something like “/services/<serviceName>”, but it doesn’t.

Response from https://<hub>/hub/api/routes

image
image404×640 26 KB

And also, now it seems, that the singleuser processes are bound to “localhost:<port>” which means, that they are not accessible from outside of the “hub” pod.

6

I managed to fix it by setting --ServerApp.ip=0.0.0.0 when launching singleuser. Now the process is bound to 0.0.0.0 and the proxy routes the traffic to http://course-svc:{port}, where course-svc is a headless kubernetes service connected to the “hub” pod.

c.JupyterHub.services.append(
  {
    "name": course,
    "url": f"http://course-svc:{port}",
    "command": ["jupyterhub-singleuser", f"--group=formgrade-{course}", f"--port={port}", "--debug", "--ServerApp.ip=0.0.0.0"],
    "oauth_no_confirm": True,
  }
)

apiVersion: v1
kind: Service
metadata:
  name: course-svc
spec:
  clusterIP: None
  selector:
    app: jupyterhub
    component: hub
7

Thank you for your post. I’m trying to deploy the same thing (z2jh with nbgrader) but I have also some problems with services.
regarding your solution, do you have to create this service manually before running jupyterhub?

apiVersion: v1
kind: Service
metadata:
  name: course-svc
spec:
  clusterIP: None
  selector:
    app: jupyterhub
    component: hub
8

Hi. Yes, you have to create the service manually.

Also… after I managed to successfully deploy a z2jh + nbgrader instance, I created a github repo with all the necessary configuration. There are a couple of other problems besides that service. I bundled everything to a helm chart, so it should pretty much work out of the box. Or at least you can take inspiration for your custom deployment :wink:

2 Likes
9

Hi,
Is this as expected that picking a service changes the user pod to hub? how does jupyterhub communicate with services in general?
best