# Enable HTTPS, a Nov-2021 attempt

**URL:** <https://discourse.jupyter.org/t/enable-https-a-nov-2021-attempt/11800>\
**Category:** BinderHub\
**Tags:** help-wanted\
**Created:** [November 22, 2021, 2:48pm UTC](https://discourse.jupyter.org/t/enable-https-a-nov-2021-attempt/11800 "2021-11-22T14:48:08Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![danielemoroni](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/danielemoroni/32/5729_2.png) [@danielemoroni](https://discourse.jupyter.org/u/danielemoroni)\
**Post date:** [November 22, 2021, 2:48pm UTC](https://discourse.jupyter.org/t/enable-https-a-nov-2021-attempt/11800/1 "2021-11-22T14:48:09Z")

</div>

I have followed the zero-to-binderhub [guide](https://binderhub.readthedocs.io/en/latest/zero-to-binderhub/index.html) to install jupyterhub/binderhub in GKE, but when I tried to [enable https](https://binderhub.readthedocs.io/en/latest/https.html), the documentation is pretty outdated, and my final setup is not working. I document here all my steps, any help is deeply appreciated.

Just as a background, I had a working environment installed from a chart from Jun-2020. My primary reason to upgrade was the expired letsencrypt DST Root CA X3. I ended up uninstalling everything first and starting from scratch. I have 1.19.14-gke.1900 (kubectl client 1.22.3), and helm 3.5.0.

**Basic http**  
I followed the guide, and the http installation went fine, I only had to add `--create-namespace` to helm install, i.e.

```auto
helm install binderhub-dev jupyterhub/binderhub --version=0.2.0-n852.h7c39292 --namespace=binder-dev --create-namespace -f secret.yaml -f config.yaml

```

with the standard .yaml’s as per instructions. I get

```auto
dmoroni@cloudshell:~/binderhub2$ helm list -n=binder-dev
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
binderhub-dev binder-dev 2 2021-11-18 21:59:12.513460362 +0000 UTC deployed binderhub-0.2.0-n852.h7c39292

dmoroni@cloudshell:~/binderhub2 (edbr-196622)$ kubectl get svc -n=binder-dev
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
binder LoadBalancer 10.48.6.1 B1.B2.B3.B4 80:32144/TCP 2m3s
hub ClusterIP 10.48.5.244 <none> 8081/TCP 2m3s
proxy-api ClusterIP 10.48.13.133 <none> 8001/TCP 2m3s
proxy-public LoadBalancer 10.48.2.93 PP1.PP2.PP3.PP4 80:32130/TCP 2m3s

```

If I navigate to B1.B2.B3.B4, the standard binderhub webpage is there, and I can build and launch ok.

**Setup IP**  
I reserved an external, static, regional, IPv4 address in GCE, let’s call it E1.E2.E3.E4. It has to be regional for the ingress-nginx to work later on.

**Install cert-manager**

```auto
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml

```

I created `binderhub-issuer.yaml`

```auto
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-staging
  namespace: binder-dev
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: <my_email>
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: binder-dev-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: nginx

```

and applied

```auto
kubectl apply -f binderhub-issuer.yaml

```

So far so good

```auto
dmoroni@cloudshell:~/binderhub2$ kubectl get issuer --all-namespaces
NAMESPACE NAME READY AGE
binder-dev letsencrypt-staging True 27s

```

**Install ingress-nginx**  
I added the official repo, because there I can get the latest version

```auto
dmoroni@cloudshell:~/binderhub2$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
"ingress-nginx" has been added to your repositories
dmoroni@cloudshell:~/binderhub2$ helm repo update
Hang tight while we grab the latest from your chart repositories...
[..]

```

Then created `ingress-nginx.yaml`

```auto
controller:
  service:
    loadBalancerIP: E1.E2.E3.E4

```

and installed

```auto
helm install binderhub-proxy ingress-nginx/ingress-nginx --namespace=binder-dev -f ingress-nginx.yaml

```

After a minute or so

```auto
dmoroni@cloudshell:~/binderhub2$ helm list -n=binder-dev
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
binderhub-dev binder-dev 2 2021-11-18 21:59:12.513460362 +0000 UTC deployed binderhub-0.2.0-n852.h7c39292
binderhub-proxy binder-dev 1 2021-11-20 21:45:46.309560351 +0000 UTC deployed ingress-nginx-4.0.9 1.0.5
dmoroni@cloudshell:~/binderhub2$ kubectl get svc -n=binder-dev
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
binder LoadBalancer 10.48.6.1 B1.B2.B3.B4 80:32144/TCP 47h
binderhub-proxy-ingress-nginx-controller LoadBalancer 10.48.7.3 E1.E2.E3.E4 80:31227/TCP,443:32018/TCP 70s
binderhub-proxy-ingress-nginx-controller-admission ClusterIP 10.48.8.35 <none> 443/TCP 70s
hub ClusterIP 10.48.5.244 <none> 8081/TCP 47h
proxy-api ClusterIP 10.48.13.133 <none> 8001/TCP 47h
proxy-public LoadBalancer 10.48.2.93 PP1.PP2.PP3.PP4 80:32130/TCP 47h

```

**Adjust binderhub**  
I modified my `config.yaml` to:

```auto
config:
  BinderHub:
    use_registry: true
    image_prefix: "my_organization/binder-dev-"
    hub_url: https://<jupyterhub-URL>
service:
  type: NodePort

jupyterhub:
  proxy:
    service:
      type: NodePort
  ingress:
    enabled: true
    hosts:
      - <jupyterhub-URL>
    annotations:
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
      cert-manager.io/issuer: letsencrypt-staging
    tls:
       - secretName: <jupyterhub-URL-with-dashes-instead-of-dots>-tls
         hosts:
          - <jupyterhub-URL>

ingress:
  enabled: true
  hosts:
     - <binderhub-URL>
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    cert-manager.io/issuer: letsencrypt-staging
  tls:
    - secretName: <binderhub-URL-with-dashes-instead-of-dots>-tls
      hosts:
        - <binderhub-URL>

```

and to be clear, `<binderhub-URL>` points to E1.E2.E3.E4, and that’s where I expect the https version of the binderhub webpage to be. Instead `<jupyterhub-URL>` points to a different IP and that’s where I expect my kernels to be spawned.

The `secret.yaml` is unchanged. Then:

```auto
dmoroni@cloudshell:~/binderhub2$ helm upgrade binderhub-dev jupyterhub/binderhub --version=0.2.0-n852.h7c39292 --namespace=binder-dev -f secret.yaml -f config.yaml
Release "binderhub-dev" has been upgraded. Happy Helming!
[..]

```

Check:

```auto
dmoroni@cloudshell:~/binderhub2$ kubectl get svc -n=binder-dev
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
binder NodePort 10.48.6.1 <none> 80:32144/TCP 2d1h
binderhub-proxy-ingress-nginx-controller LoadBalancer 10.48.7.3 E1.E2.E3.E4 80:31227/TCP,443:32018/TCP 105m
binderhub-proxy-ingress-nginx-controller-admission ClusterIP 10.48.8.35 <none> 443/TCP 105m
cm-acme-http-solver-4z94c NodePort 10.48.6.198 <none> 8089:32231/TCP 21m
hub ClusterIP 10.48.5.244 <none> 8081/TCP 2d1h
proxy-api ClusterIP 10.48.13.133 <none> 8001/TCP 2d1h
proxy-public NodePort 10.48.2.93 <none> 80:32130/TCP 2d1h
dmoroni@cloudshell:~/binderhub2$ kubectl get certificates --all-namespaces
NAMESPACE NAME READY SECRET AGE
binder-dev <binderhub>-tls True <binderhub>-tls 22m
binder-dev <jupyterhub>-tls False <jupyterhub>-tls 22m
dmoroni@cloudshell:~/binderhub2$ kubectl describe certificate <jupyterhub>-tls -n=binder-dev
Name: <jupyterhub>-tls
Namespace: binder-dev
Labels: app=jupyterhub
              app.kubernetes.io/managed-by=Helm
              chart=jupyterhub-1.1.2
              component=ingress
              heritage=Helm
              release=binderhub-dev
Annotations: <none>
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
  [..]
Spec:
  Dns Names:
    <jupyterhub-URL>
  Issuer Ref:
    Group: cert-manager.io
    Kind: Issuer
    Name: letsencrypt-staging
  Secret Name: <jupyterhub>-tls
  Usages:
    digital signature
    key encipherment
Status:
  Conditions:
    Last Transition Time: 2021-11-20T23:09:04Z
    Message: Issuing certificate as Secret does not exist
    Observed Generation: 1
    Reason: DoesNotExist
    Status: False
    Type: Ready
    Last Transition Time: 2021-11-20T23:09:04Z
    Message: Issuing certificate as Secret does not exist
    Observed Generation: 1
    Reason: DoesNotExist
    Status: True
    Type: Issuing
  Next Private Key Secret Name: <jupyterhub>-tls-q89qd
Events: <none>

```

**Results**  
The webpage is reachable but with a NET::ERR\_CERT\_AUTHORITY\_INVALID, while is not reachable.

What is going on?

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [November 22, 2021, 10:08pm UTC](https://discourse.jupyter.org/t/enable-https-a-nov-2021-attempt/11800/2 "2021-11-22T22:08:39Z")

</div>

You’re right that the docs are out of date, it’s on the todo-list to update them.

There’ was an older discussion here which may provide some help:

> [@BinderHub HTTPS](https://discourse.jupyter.org/t/binderhub-https/7597):
>
> Hi, I just did a basic BinderHub deployment in GCP, but I really just want to get to the next level. For that I simply need some help to wrap my head around documentation on this. [https://binderhub.readthedocs.io/en/latest/https.html](https://binderhub.readthedocs.io/en/latest/https.html) I got passed the deprecated syntax since I am using helm v3. BUT then it simply derails for me (I have not done a helm update yet because I don’t understand. And I don’t like to deploy something that I don’t understand). We agree that the we have TWO IP’s in pla…

Howver since this uses an Ingress your HTTPS setup is effectively independent of JupyterHub/BinderHub. This means you should be able to follow any other guide to getting HTTPS working with nginx-ingress, and once that’s working configure JupyterHub/BinderHub behind it.

---

<div class="post-metadata">

**Author:** ![danielemoroni](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/danielemoroni/32/5729_2.png) [@danielemoroni](https://discourse.jupyter.org/u/danielemoroni)\
**Post date:** [November 23, 2021, 11:12pm UTC](https://discourse.jupyter.org/t/enable-https-a-nov-2021-attempt/11800/3 "2021-11-23T23:12:09Z")

</div>

Thank you, I had a read at that discussion and now … I’m even more confused. I also had the idea that there were two IP’s in play: one for the binderhub instance, the webpage where to specify repos and launch; one for the jupyterhub instance, actually spawning the kernels.

In fact under plain http setup I get two external IP’s for the services binder and proxy-public respectively. I point my browser to [http://binder-IP](http://binder-IP) and I get kernels from [http://proxy-public-IP](http://proxy-public-IP).

Then I’ve bought two domains: [my.example.binder.com](http://my.example.binder.com), [my.example.jupyter.com](http://my.example.jupyter.com). And I’ve pointed them to binder-IP and proxy-public-IP. I want to point my browser to [my.example.binder.com](http://my.example.binder.com) and get kernels from [my.example.jupyter.com](http://my.example.jupyter.com). In the old versions this was achievable by specifying jupyterhub.proxy.http in config.yaml, but now that setting is gone and I see the service proxy-http is also gone. Is that not possible at all? If in config.yaml I change hub\_url: [http://proxy-public-IP](http://proxy-public-IP) to [http://my.example.jupyter.com](http://my.example.jupyter.com) it just doesn’t work.

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [November 25, 2021, 6:45pm UTC](https://discourse.jupyter.org/t/enable-https-a-nov-2021-attempt/11800/4 "2021-11-25T18:45:20Z")

</div>

> [@danielemoroni](#):
>
> In fact under plain http setup I get two external IP’s for the services binder and proxy-public respectively. I point my browser to [http://binder-IP](http://binder-IP) and I get kernels from [http://proxy-public-IP](http://proxy-public-IP).

This is surprising. Your configuration sets both services to be of type `NodePort` so you shouldn’t have an external IP. I’d expect only your nginx-ingress controller would have a LoadBalancer IP. Did something change?

---

<div class="post-metadata">

**Author:** ![danielemoroni](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/danielemoroni/32/5729_2.png) [@danielemoroni](https://discourse.jupyter.org/u/danielemoroni)\
**Post date:** [November 28, 2021, 3:33pm UTC](https://discourse.jupyter.org/t/enable-https-a-nov-2021-attempt/11800/5 "2021-11-28T15:33:40Z")

</div>

Hi, sorry, I was referring to the initial setup, plain http, basically the zero-to-binderhub [guide](https://binderhub.readthedocs.io/en/latest/zero-to-binderhub/index.html) install takes me to this situation

```auto
dmoroni@cloudshell:~/binderhub2 (edbr-196622)$ kubectl get svc -n=binder-dev
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
binder LoadBalancer 10.48.6.1 B1.B2.B3.B4 80:32144/TCP 2m3s
hub ClusterIP 10.48.5.244 <none> 8081/TCP 2m3s
proxy-api ClusterIP 10.48.13.133 <none> 8001/TCP 2m3s
proxy-public LoadBalancer 10.48.2.93 PP1.PP2.PP3.PP4 80:32130/TCP 2m3s

```

and I see two IP address. So I pointed my two domains to them, but I can’t spawn kernels from [my.example.jupyter.com](http://my.example.jupyter.com), only the IP address.

Meantime I’ve managed to setup https for my binder instance, but not for the jupyter one. I gave up on cert-manager and ingress-nginx, and leveraged the GKE native `managedcertificates` and `static-ip`. Here is how I did it:

```auto
dmoroni@cloudshell:~/binderhub3$ more clip-cert.yaml
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: my-binder-cert
  namespace: binder-dev
spec:
  domains:
    - my.example.binder.com
dmoroni@cloudshell:~/binderhub3$ kubectl apply -f clip-cert.yaml

dmoroni@cloudshell:~/binderhub3$ more clip-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-nodeport-service
  namespace: binder-dev
  labels:
    app: binder
spec:
  selector:
    app: binder
  type: NodePort
  ports:
    - protocol: TCP
      port: 8328
      targetPort: 8585
dmoroni@cloudshell:~/binderhub3$ kubectl apply -f clip-service.yaml

dmoroni@cloudshell:~/binderhub3$ more clip-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ssl-ingress
  namespace: binder-dev
  annotations:
    kubernetes.io/ingress.global-static-ip-name: my-example-binder-com
    networking.gke.io/managed-certificates: my-binder-cert
  labels:
    app: binder
spec:
  backend:
    serviceName: my-nodeport-service
    servicePort: 8328
  rules:
  - host: my.example.binder.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: my-nodeport-service
          servicePort: 8328
dmoroni@cloudshell:~/binderhub3$ kubectl apply -f clip-ingress.yaml

```

The last one creating the ingress is using a deprecated api, but it still works for me on gke-1.19. The domain [my.example.binder.com](http://my.example.binder.com) is pointing to E1.E2.E3.E4 which I created in GCE as a static IPv4 address, named my-example-binder-com.

At this point I wait for the certificate to be provisioned, i.e. until `kubectl describe managedcertificate my-binder-cert` shows `Status: Active`. And now I can point to [https://my.example.binder.com](https://my.example.binder.com) and I get the correct webpage, which spawns kernels under [http://PP1.PP2.PP3.PP4](http://PP1.PP2.PP3.PP4).

This hybrid is the best I could get, but I’d like to spawn kernels under https, and use [my.example.jupyter.com](http://my.example.jupyter.com). Here is where I get confused, should I create another static-ip and repeat the above? Then how do I connect the two https-enabled domains in binderhub?

---

<div class="post-metadata">

**Author:** ![manics](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/manics/32/85_2.png) [@manics](https://discourse.jupyter.org/u/manics)\
**Post date:** [November 30, 2021, 11:17pm UTC](https://discourse.jupyter.org/t/enable-https-a-nov-2021-attempt/11800/6 "2021-11-30T23:17:23Z")

</div>

I presume you’ve got an ingress controller installed on your cluster? If so you don’t need any LoadBalancers in BinderHub/JupyterHub, you can configure your ingress for both using the Helm chart by specifying the domain(s).

You can then point your DNS to your ingress controller, and configure all certificates in that controller.

---

<div class="post-metadata">

**Author:** ![danielemoroni](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/danielemoroni/32/5729_2.png) [@danielemoroni](https://discourse.jupyter.org/u/danielemoroni)\
**Post date:** [December 2, 2021, 1:34pm UTC](https://discourse.jupyter.org/t/enable-https-a-nov-2021-attempt/11800/7 "2021-12-02T13:34:04Z")

</div>

Thank you, could you elaborate a bit more please? This is my situation:

```auto
dmoroni@cloudshell:~$ kubectl get ingress -n=binder-dev
NAME CLASS HOSTS ADDRESS PORTS AGE
my-ssl-ingress <none> my.example.binder.com E1.E2.E3.E4 80 7d21h
dmoroni@cloudshell:~$ kubectl get svc -n=binder-dev
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
binder LoadBalancer 10.48.1.240 B1.B2.B3.B4 80:32634/TCP 8d
my-nodeport-service NodePort 10.48.11.13 <none> 8328:31093/TCP 7d21h
hub ClusterIP 10.48.10.200 <none> 8081/TCP 8d
proxy-api ClusterIP 10.48.13.28 <none> 8001/TCP 8d
proxy-http ClusterIP 10.48.8.27 <none> 8000/TCP 39h
proxy-public LoadBalancer 10.48.11.62 PP1.PP2.PP3.PP4 443:31490/TCP,80:31732/TCP 8d
dmoroni@cloudshell:~$ kubectl get managedcertificates -n=binder-dev
NAME AGE STATUS
my-binder-cert 7d21h Active

```

My domain [my.example.binder.com](http://my.example.binder.com) is pointing to E1.E2.E3.E4 and produces a working webpage, while [my.example.jupyter.com](http://my.example.jupyter.com) is pointing to PP1.PP2.PP3.PP4 but it’s not reachable.

How should my helm chart for binderhub/jupyterhub look?

---

<div class="post-metadata">

**Author:** ![danielemoroni](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/danielemoroni/32/5729_2.png) [@danielemoroni](https://discourse.jupyter.org/u/danielemoroni)\
**Post date:** [February 17, 2022, 11:03am UTC](https://discourse.jupyter.org/t/enable-https-a-nov-2021-attempt/11800/8 "2022-02-17T11:03:35Z")

</div>

The thread is a bit old, but I thought it would be useful to post the solution I finally went for. It was so straightforward that I missed it completely in the first place.

- Install jupyter/binderhub under plain http, like I showed at the very beginning of my first post here
- Reserve two static IPs, point my domains two each one
- Create two GKE ManagedCertificate’s for them, wait and be sure they get provisioned
- Create an ingress for binderhub and point it to the ‘binder’ service, port 80
- Create another ingress for jupyterhub and point it the ‘proxy-public’ service, port 80

That’s it! Now the https traffic is nice and certified and gets redirected internally in the cluster to http traffic, which is nicely setup. The ingress also contains the cors annotations.

I was so confused while trying to use one external IP, when there are two involved. Also I don’t know why I was trying to use NodePort services while LoadBalancer was working perfectly fine.

I then had some problems with websockets dropping out, but that’s another [story](https://discourse.jupyter.org/t/websocket-closed-unexpectedly/12781/12).