Enable HTTPS, a Nov-2021 attempt

Binder BinderHub
1

I have followed the zero-to-binderhub guide to install jupyterhub/binderhub in GKE, but when I tried to enable https, the documentation is pretty outdated, and my final setup is not working. I document here all my steps, any help is deeply appreciated.

Just as a background, I had a working environment installed from a chart from Jun-2020. My primary reason to upgrade was the expired letsencrypt DST Root CA X3. I ended up uninstalling everything first and starting from scratch. I have 1.19.14-gke.1900 (kubectl client 1.22.3), and helm 3.5.0.

Basic http
I followed the guide, and the http installation went fine, I only had to add --create-namespace to helm install, i.e.

helm install binderhub-dev jupyterhub/binderhub --version=0.2.0-n852.h7c39292 --namespace=binder-dev --create-namespace -f secret.yaml -f config.yaml

with the standard .yaml’s as per instructions. I get

dmoroni@cloudshell:~/binderhub2$ helm list -n=binder-dev
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS     CHART                            APP VERSION
binderhub-dev   binder-dev      2               2021-11-18 21:59:12.513460362 +0000 UTC deployed   binderhub-0.2.0-n852.h7c39292

dmoroni@cloudshell:~/binderhub2 (edbr-196622)$ kubectl get svc -n=binder-dev
NAME           TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE
binder         LoadBalancer   10.48.6.1      B1.B2.B3.B4      80:32144/TCP   2m3s
hub            ClusterIP      10.48.5.244    <none>           8081/TCP       2m3s
proxy-api      ClusterIP      10.48.13.133   <none>           8001/TCP       2m3s
proxy-public   LoadBalancer   10.48.2.93     PP1.PP2.PP3.PP4  80:32130/TCP   2m3s

If I navigate to B1.B2.B3.B4, the standard binderhub webpage is there, and I can build and launch ok.

Setup IP
I reserved an external, static, regional, IPv4 address in GCE, let’s call it E1.E2.E3.E4. It has to be regional for the ingress-nginx to work later on.

Install cert-manager

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml

I created binderhub-issuer.yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-staging
  namespace: binder-dev
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: <my_email>
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: binder-dev-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: nginx

and applied

kubectl apply -f binderhub-issuer.yaml

So far so good

dmoroni@cloudshell:~/binderhub2$ kubectl get issuer --all-namespaces
NAMESPACE    NAME                  READY   AGE
binder-dev   letsencrypt-staging   True    27s

Install ingress-nginx
I added the official repo, because there I can get the latest version

dmoroni@cloudshell:~/binderhub2$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
"ingress-nginx" has been added to your repositories
dmoroni@cloudshell:~/binderhub2$ helm repo update
Hang tight while we grab the latest from your chart repositories...
[..]

Then created ingress-nginx.yaml

controller:
  service:
    loadBalancerIP: E1.E2.E3.E4

and installed

helm install binderhub-proxy ingress-nginx/ingress-nginx --namespace=binder-dev -f ingress-nginx.yaml

After a minute or so

dmoroni@cloudshell:~/binderhub2$ helm list -n=binder-dev
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS     CHART                            APP VERSION
binderhub-dev   binder-dev      2               2021-11-18 21:59:12.513460362 +0000 UTC deployed   binderhub-0.2.0-n852.h7c39292
binderhub-proxy binder-dev      1               2021-11-20 21:45:46.309560351 +0000 UTC deployed   ingress-nginx-4.0.9              1.0.5
dmoroni@cloudshell:~/binderhub2$ kubectl get svc -n=binder-dev
NAME                                                 TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
binder                                               LoadBalancer   10.48.6.1      B1.B2.B3.B4      80:32144/TCP                 47h
binderhub-proxy-ingress-nginx-controller             LoadBalancer   10.48.7.3      E1.E2.E3.E4      80:31227/TCP,443:32018/TCP   70s
binderhub-proxy-ingress-nginx-controller-admission   ClusterIP      10.48.8.35     <none>           443/TCP                      70s
hub                                                  ClusterIP      10.48.5.244    <none>           8081/TCP                     47h
proxy-api                                            ClusterIP      10.48.13.133   <none>           8001/TCP                     47h
proxy-public                                         LoadBalancer   10.48.2.93     PP1.PP2.PP3.PP4  80:32130/TCP                 47h

Adjust binderhub
I modified my config.yaml to:

config:
  BinderHub:
    use_registry: true
    image_prefix: "my_organization/binder-dev-"
    hub_url: https://<jupyterhub-URL>
service:
  type: NodePort

jupyterhub:
  proxy:
    service:
      type: NodePort
  ingress:
    enabled: true
    hosts:
      - <jupyterhub-URL>
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
      cert-manager.io/issuer: letsencrypt-staging
    tls:
       - secretName: <jupyterhub-URL-with-dashes-instead-of-dots>-tls
         hosts:
          - <jupyterhub-URL>

ingress:
  enabled: true
  hosts:
     - <binderhub-URL>
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    cert-manager.io/issuer: letsencrypt-staging
  tls:
    - secretName: <binderhub-URL-with-dashes-instead-of-dots>-tls
      hosts:
        - <binderhub-URL>

and to be clear, <binderhub-URL> points to E1.E2.E3.E4, and that’s where I expect the https version of the binderhub webpage to be. Instead <jupyterhub-URL> points to a different IP and that’s where I expect my kernels to be spawned.

The secret.yaml is unchanged. Then:

dmoroni@cloudshell:~/binderhub2$ helm upgrade binderhub-dev jupyterhub/binderhub --version=0.2.0-n852.h7c39292 --namespace=binder-dev -f secret.yaml -f config.yaml
Release "binderhub-dev" has been upgraded. Happy Helming!
[..]

Check:

dmoroni@cloudshell:~/binderhub2$ kubectl get svc -n=binder-dev
NAME                                                 TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
binder                                               NodePort       10.48.6.1      <none>          80:32144/TCP                 2d1h
binderhub-proxy-ingress-nginx-controller             LoadBalancer   10.48.7.3      E1.E2.E3.E4     80:31227/TCP,443:32018/TCP   105m
binderhub-proxy-ingress-nginx-controller-admission   ClusterIP      10.48.8.35     <none>          443/TCP                      105m
cm-acme-http-solver-4z94c                            NodePort       10.48.6.198    <none>          8089:32231/TCP               21m
hub                                                  ClusterIP      10.48.5.244    <none>          8081/TCP                     2d1h
proxy-api                                            ClusterIP      10.48.13.133   <none>          8001/TCP                     2d1h
proxy-public                                         NodePort       10.48.2.93     <none>          80:32130/TCP                 2d1h
dmoroni@cloudshell:~/binderhub2$ kubectl get certificates --all-namespaces
NAMESPACE    NAME               READY   SECRET             AGE
binder-dev   <binderhub>-tls    True    <binderhub>-tls    22m
binder-dev   <jupyterhub>-tls   False   <jupyterhub>-tls   22m
dmoroni@cloudshell:~/binderhub2$ kubectl describe certificate <jupyterhub>-tls -n=binder-dev
Name:         <jupyterhub>-tls
Namespace:    binder-dev
Labels:       app=jupyterhub
              app.kubernetes.io/managed-by=Helm
              chart=jupyterhub-1.1.2
              component=ingress
              heritage=Helm
              release=binderhub-dev
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  [..]
Spec:
  Dns Names:
    <jupyterhub-URL>
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       Issuer
    Name:       letsencrypt-staging
  Secret Name:  <jupyterhub>-tls
  Usages:
    digital signature
    key encipherment
Status:
  Conditions:
    Last Transition Time:        2021-11-20T23:09:04Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2021-11-20T23:09:04Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  <jupyterhub>-tls-q89qd
Events:                          <none>

Results
The webpage is reachable but with a NET::ERR_CERT_AUTHORITY_INVALID, while is not reachable.

What is going on?

2

You’re right that the docs are out of date, it’s on the todo-list to update them.

There’ was an older discussion here which may provide some help:

Howver since this uses an Ingress your HTTPS setup is effectively independent of JupyterHub/BinderHub. This means you should be able to follow any other guide to getting HTTPS working with nginx-ingress, and once that’s working configure JupyterHub/BinderHub behind it.

3

Thank you, I had a read at that discussion and now … I’m even more confused. I also had the idea that there were two IP’s in play: one for the binderhub instance, the webpage where to specify repos and launch; one for the jupyterhub instance, actually spawning the kernels.

In fact under plain http setup I get two external IP’s for the services binder and proxy-public respectively. I point my browser to http://binder-IP and I get kernels from http://proxy-public-IP.

Then I’ve bought two domains: my.example.binder.com, my.example.jupyter.com. And I’ve pointed them to binder-IP and proxy-public-IP. I want to point my browser to my.example.binder.com and get kernels from my.example.jupyter.com. In the old versions this was achievable by specifying jupyterhub.proxy.http in config.yaml, but now that setting is gone and I see the service proxy-http is also gone. Is that not possible at all? If in config.yaml I change hub_url: http://proxy-public-IP to http://my.example.jupyter.com it just doesn’t work.

4

This is surprising. Your configuration sets both services to be of type NodePort so you shouldn’t have an external IP. I’d expect only your nginx-ingress controller would have a LoadBalancer IP. Did something change?

5

Hi, sorry, I was referring to the initial setup, plain http, basically the zero-to-binderhub guide install takes me to this situation

dmoroni@cloudshell:~/binderhub2 (edbr-196622)$ kubectl get svc -n=binder-dev
NAME           TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE
binder         LoadBalancer   10.48.6.1      B1.B2.B3.B4      80:32144/TCP   2m3s
hub            ClusterIP      10.48.5.244    <none>           8081/TCP       2m3s
proxy-api      ClusterIP      10.48.13.133   <none>           8001/TCP       2m3s
proxy-public   LoadBalancer   10.48.2.93     PP1.PP2.PP3.PP4  80:32130/TCP   2m3s

and I see two IP address. So I pointed my two domains to them, but I can’t spawn kernels from my.example.jupyter.com, only the IP address.

Meantime I’ve managed to setup https for my binder instance, but not for the jupyter one. I gave up on cert-manager and ingress-nginx, and leveraged the GKE native managedcertificates and static-ip. Here is how I did it:

dmoroni@cloudshell:~/binderhub3$ more clip-cert.yaml
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: my-binder-cert
  namespace: binder-dev
spec:
  domains:
    - my.example.binder.com
dmoroni@cloudshell:~/binderhub3$ kubectl apply -f clip-cert.yaml

dmoroni@cloudshell:~/binderhub3$ more clip-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-nodeport-service
  namespace: binder-dev
  labels:
    app: binder
spec:
  selector:
    app: binder
  type: NodePort
  ports:
    - protocol: TCP
      port: 8328
      targetPort: 8585
dmoroni@cloudshell:~/binderhub3$ kubectl apply -f clip-service.yaml

dmoroni@cloudshell:~/binderhub3$ more clip-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ssl-ingress
  namespace: binder-dev
  annotations:
    kubernetes.io/ingress.global-static-ip-name: my-example-binder-com
    networking.gke.io/managed-certificates: my-binder-cert
  labels:
    app: binder
spec:
  backend:
    serviceName: my-nodeport-service
    servicePort: 8328
  rules:
  - host: my.example.binder.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: my-nodeport-service
          servicePort: 8328
dmoroni@cloudshell:~/binderhub3$ kubectl apply -f clip-ingress.yaml

The last one creating the ingress is using a deprecated api, but it still works for me on gke-1.19. The domain my.example.binder.com is pointing to E1.E2.E3.E4 which I created in GCE as a static IPv4 address, named my-example-binder-com.

At this point I wait for the certificate to be provisioned, i.e. until kubectl describe managedcertificate my-binder-cert shows Status: Active. And now I can point to https://my.example.binder.com and I get the correct webpage, which spawns kernels under http://PP1.PP2.PP3.PP4.

This hybrid is the best I could get, but I’d like to spawn kernels under https, and use my.example.jupyter.com. Here is where I get confused, should I create another static-ip and repeat the above? Then how do I connect the two https-enabled domains in binderhub?

6

I presume you’ve got an ingress controller installed on your cluster? If so you don’t need any LoadBalancers in BinderHub/JupyterHub, you can configure your ingress for both using the Helm chart by specifying the domain(s).

You can then point your DNS to your ingress controller, and configure all certificates in that controller.

7

Thank you, could you elaborate a bit more please? This is my situation:

dmoroni@cloudshell:~$ kubectl get ingress -n=binder-dev
NAME             CLASS    HOSTS                   ADDRESS       PORTS   AGE
my-ssl-ingress   <none>   my.example.binder.com   E1.E2.E3.E4   80      7d21h
dmoroni@cloudshell:~$ kubectl get svc -n=binder-dev
NAME                    TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
binder                  LoadBalancer   10.48.1.240    B1.B2.B3.B4     80:32634/TCP                 8d
my-nodeport-service     NodePort       10.48.11.13    <none>          8328:31093/TCP               7d21h
hub                     ClusterIP      10.48.10.200   <none>          8081/TCP                     8d
proxy-api               ClusterIP      10.48.13.28    <none>          8001/TCP                     8d
proxy-http              ClusterIP      10.48.8.27     <none>          8000/TCP                     39h
proxy-public            LoadBalancer   10.48.11.62    PP1.PP2.PP3.PP4 443:31490/TCP,80:31732/TCP   8d
dmoroni@cloudshell:~$ kubectl get managedcertificates -n=binder-dev
NAME             AGE     STATUS
my-binder-cert   7d21h   Active

My domain my.example.binder.com is pointing to E1.E2.E3.E4 and produces a working webpage, while my.example.jupyter.com is pointing to PP1.PP2.PP3.PP4 but it’s not reachable.

How should my helm chart for binderhub/jupyterhub look?

8

The thread is a bit old, but I thought it would be useful to post the solution I finally went for. It was so straightforward that I missed it completely in the first place.

  • Install jupyter/binderhub under plain http, like I showed at the very beginning of my first post here
  • Reserve two static IPs, point my domains two each one
  • Create two GKE ManagedCertificate’s for them, wait and be sure they get provisioned
  • Create an ingress for binderhub and point it to the ‘binder’ service, port 80
  • Create another ingress for jupyterhub and point it the ‘proxy-public’ service, port 80

That’s it! Now the https traffic is nice and certified and gets redirected internally in the cluster to http traffic, which is nicely setup. The ingress also contains the cors annotations.

I was so confused while trying to use one external IP, when there are two involved. Also I don’t know why I was trying to use NodePort services while LoadBalancer was working perfectly fine.

I then had some problems with websockets dropping out, but that’s another story.