Letsencrypt cert on self-hosted K3S single-node using ztjh helm stops working after a few hours?

JupyterHub Zero to JupyterHub on Kubernetes
1

I have a desktop workstation running K3S and the ztjh helm chart but am struggling to enable https via the letsencrypt configuration.

My config.yaml looks like this:

ingress:
  enabled: true
proxy:
  service:
    loadBalancerIP: "<my-IP-address>"
  https:
    enabled: true
    hosts:
      - <my-domain>
    letsencrypt:
      contactEmail: <my-email>

and I’ve verified the DNS is correct (my host name resolves to my specified IP). I launch with helm following the zthj guide succesfully, but my proxy-public service remains stuck with EXTERNAL-IP pending.

kubectl --namespace=testjupyter get service proxy-public
NAME           TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
proxy-public   LoadBalancer   10.43.2.137   <pending>     443:30868/TCP   6m26s

I understand that the default helm config is using a LoadBalancer type as shown above, and given that I have a single node here that’s probably not doing much for me, but I also understand that K3S includes a built-in ‘klipper’ load balancer suitable for a single node setup

If the ServiceLB Pod runs on a node that has an external IP configured, the node’s external IP is populated into the Service’s status.loadBalancer.ingress address list. Otherwise, the node’s internal IP is used.

I don’t see how to specify the external IP as described, notably I don’t seem to be allowed to use the field externalIP in the helm config. But I can patch the service, after which it shows my external IP instead of Pending.

kubectl patch svc SERVICENAME -p '{"spec": {"type": "LoadBalancer", "externalIPs":["<my-server-ip>"]}}'

From what I can tell, this works, but feels clunky to patch the service like this. What’s the correct way to do this?
EDIT This works for a while, and I can log in to the jupyterhub, but after an hour or so, my browser tells me that the site is insecure because the SSL cert is self-signed (TRAEFIK DEFAULT CERT)

2

I’ve never used the k3s built-in load-balancer. For debugging purposes it’s going to be easier to investigate with without Z2JH, for example by creating a standalone service manually: Create an External Load Balancer | Kubernetes

Alternatively since this is a standalone node you could try using a node-port instead? Service | Kubernetes
Or perhaps use K3s’ default ingress? Networking | K3s

3

Thanks @manics! Right, I am starting to think the problem is that k3s is trying to also set up https with a default (self-signed Traefik certificate Advanced Options / Configuration | K3s ), and these two are somehow competing.

I tried changing the service type in the config to a NodePort, and I could get HTTP, but no HTTPS / SSL. Likewise with K3S’s default ingress, it doesn’t seem to be set up to use letsencrypt+with automatic updating, and I’d really rather not mess with manually handling my HTTPS certs.

Actually my ideal would be to use a NodePort and allow my existing Caddy server to take care of reverse proxy and external networking. But whenever I point Caddy at the internal Nodeport, it complains that it can’t register the HTTPS cert, though HTTP works this way. Perhaps this is somehow the same conflict with K3S traefik setup? Maybe I need to start my K38s with --disable=traefik.

I know there’s lots of possibly valid configurations here depending on the use-case, but is there any you’ve used or recommend for self-hosted system? Or your mostly just going with the load balancers supplied by the commercial cloud providers?

4

I’ve never used a load balancer that’s not on a standalone cluster since it’s inevitably going to be a compromise between having a single IP, or having to deal with external networking to ensure the server can have multiple IPs. I usually use an ingress controller instead, either in host network mode so it can listen on port 80 and/or 443, or listening on a nodeport and using an external proxy.

If you’ve already got Caddy that should work, and would also manage the Let’s Encrypt certificate. As you suggest it’s possible the Traefik ingress included in k3s may also conflict, but with some investigation I expect you can probably get it to work instead of Caddy.

5

Thanks @manics, I think you nailed this. Yes, it does look like the traefik ingress controller from K3s was conflicting. disabled that and everything is working fine.

It probably does make more sense to switch over to a NodePort + my existing Caddyserver. The traefik conflict was maybe causing that to fail before as well. I guess I just bind the internal IP address that the nodeport chooses? (Sorry still struggling with K8s networking model – in docker I can anticipate that any service just appears and <container_name>:port on the docker network group and just put that in my Caddyfile). Anyway I’ll poke around.

And promise a little blog post to help others once I get this working. (I was very pleased to figure out GPU+timesharing for K3s, really slick).

Thanks for all the help!

6

A Kubernetes NodePort is a bit like Docker port-forwarding- the container port is bound to a host port: Service | Kubernetes
In Z2JH you can choose the NodePort instead of letting K8s choose a random one, be aware it must still be in the allowed K8s range:
Configuration Reference — Zero to JupyterHub with Kubernetes documentation

You can disable traefik when you setup k3s: Networking | K3s

1 Like
7

Nice, this is all working perfectly for me following the directions you’ve given. I’m able to host with the built-in https in jupyter or bind to my external caddyserver (both options require disabling traefik in k3s, just as you note).

I may try k8s cert-manager to replace my caddyserver once I get comfortable deploying all my other services via k8s, but already this is working so well for my team. what an amazing project and amazing community here.

1 Like