JupyterHub on k8s with Traefik

JupyterHub Zero to JupyterHub on Kubernetes
1

Hi everyone,

I’m having issues setting up JupyterHub on kubernetes that sits behind traefik doing path based routing to services. I’d like to have a number of different services deployed including a number of jupyterhub deployments under different namespaces and reachable via different web paths.

I’m deploying jupyterhub using the helm chart with the following configuration:

hub:
  baseUrl: /longbow-workshop/
proxy:
  secretToken: "removed"
  service:
    type: ClusterIP
rbac:
  enabled: true
singleuser:
  image:
    name: ccpbiosim/longbow-workshop
    tag: v1.0
  storage:
    type: none
  memory:
    limit: 5.0G
    guarantee: 1.0G
  cpu:
    limit: 10.0
    guarantee: 0.2
cull:
  timeout: 2400
  every: 600
prePuller:
  continuous:
    enabled: true
auth:
  type: custom
  custom:
    className: tmpauthenticator.TmpAuthenticator

with the following command:

helm upgrade --install longbow-workshop jupyterhub/jupyterhub --namespace longbow-workshop --timeout=3600 --version=0.8.2 --values longbow-workshop-helm.yaml

I then configure traefik using the basic steps here docs.traefik.io/user-guide/kubernetes/

Since my jupyterhub is in its own namespace that traefik is not deployed to I have to create a service to expose the public-proxy like this:

apiVersion: v1
kind: Service
metadata:
  name: longbow-workshop
  namespace: default
spec:
  type: ExternalName
  externalName: proxy-public.longbow-workshop.svc.cluster.local
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https

My Traefik ingress rules look like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ccpbiosim
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.frontend.rule.type: PathPrefixStrip
spec:
  rules:
  - host: hubs.ccpbiosim.ac.uk
    http:
      paths:
      - path: /longbow-workshop
        backend:
          serviceName: longbow-workshop
          servicePort: http

If I then try to go to hubs.ccpbiosim.ac.uk/longbow-workshop it will redirect recursively adding /hub to the end of the url until the browser kills it and it looks like /longbow-workshop/hub/hub/hub/hub/hub/hub/hub/hub/hub/hub.

Oddly, if I configure traefik to use subdomains instead of paths then everything works fine, but it is quite time consuming for us to get A records for subdomains and would prefer to use path based routing instead. Anyone seen this before and know what I’m doing wrong?

2

@minrk suggested the following in the chat:

use PathPrefix instead of PathPrefixStrip . JupyterHub wants to know its absolute URL, but PathPrefixStrip is telling the hub that it’s receiving a request for / when it’s actually receiving a request for /hub/ . JupyterHub redirects requests for / to /hub/ so that’s where it’s getting caught up.

I’ll try this now and post back

2 Likes
3

@minrk Your solution works! And makes a lot of sense thinking about it. Thank you for your help!

4

Sorry for resurrecting such an old post, but…

I’m using more or less the same config as you and want different services on the same machine. I’m using k3s and their install of traefik. In my config, I have:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: jupyter-ingress
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.frontend.rule.type: PathPrefix
spec:
  rules:
  - host: HOSTNAME
    http:
      paths:
      - path: /hadoop
        pathType: Prefix
        backend:
          serviceName: hadoop-hadoop-yarn-ui
          servicePort: 8088
      - path: /jh
        pathType: Prefix
        backend:
          serviceName: proxy-public
          servicePort: 80
  tls:
  - hosts:
    - HOSTNAME
    secretName: jupyter-tls

Using curl -Ls -o /dev/null -w %{url_effective} http://HOSTNAME/jh I see that I am being redirected to http://HOSTNAME/hub/jh

I.e. for some reason, the ‘jh’ is being passed through. Equally important, there is no /hub prefix and thus this results in a 404. Similar problem when accessing the hadoop cluster. I’ve been digging through the traefik documentation but it’s not clarifying things.

5

Hi @dirkcgrunwald,

I’ve since moved to traefik 2.x as I wanted auto-renewing wildcard domain certs from LE, so my current config is a bit different (so I won’t post it). However your traefik ingress looks ok and the 404 would imply traefik is actually routing. Are you deploying jhub with helm? If so do you have the lines

hub:
  baseUrl: /longbow-workshop/

In your configuration yaml for jhub?

1 Like
6

The “baseURL” was the trick - I was unaware of that config. Thanks!

1 Like