Trouble getting HTTPS / letsencrypt working with 0.9.0-beta.4

#1

I’ve been using JH 0.8 (included as a requirements.yaml for my own service) on google cloud, tried playing with 0.9.0-beta.4, and have not been able to get https working after an upgrade. It looks like autohttps isn’t properly serving the acme challenge on port 80 is 0.9.0-betas?

When I visit https://improc.ceresimaging.net (which maps correctly to 35.203.130.226) I get an SSL protocol error (basically https server had an internal error). If I visit http://improc.ceresimaging.net, I get redirected to 443/https.

All the logs look ok, except the autohttps pod is failing to complete the letsencript http challenge, timing out trying to access .well-know/acme-challenge on port 80:

Screen Shot 2020-03-10 at 3.35.00 PM
Screen Shot 2020-03-10 at 3.35.00 PM1968×250 65 KB

Running wget on my computer produces the same results:

➜  ~ wget http://improc.ceresimaging.net/.well-known/acme-challenge/018QQqoEpMphNo8_7J61TOcmQ7oGhZ7WOAl3VMfJuJc
--2020-03-10 15:37:04--  http://improc.ceresimaging.net/.well-known/acme-challenge/018QQqoEpMphNo8_7J61TOcmQ7oGhZ7WOAl3VMfJuJc
Resolving improc.ceresimaging.net (improc.ceresimaging.net)... 35.203.130.226
Connecting to improc.ceresimaging.net (improc.ceresimaging.net)|35.203.130.226|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-03-10 15:38:06 ERROR 404: Not Found.

The services look like they’re up and healthy, as do the pods.

Relevant part of values.yaml is as follows:

  proxy:
    https:
      hosts:
        - improc.ceresimaging.net
      letsencrypt:
        contactEmail: seth@ceresimaging.net
    secretToken: "SECRETS DELETED"
    service:
      loadBalancerIP: 35.203.130.226

In case its helpful, here’s the full values.yaml with secrets elided:

jupyterhub:
  proxy:
    https:
      hosts:
        - improc.ceresimaging.net
      letsencrypt:
        contactEmail: seth@ceresimaging.net
    secretToken: "SECRETS DELETED"
    service:
      loadBalancerIP: 35.203.130.226
	  
  singleuser:
    defaultUrl: "/lab"
    image:
      name: gcr.io/ceres-imaging-science/improc-notebook
      tag: latest

    extraEnv:
      JUPYTER_ENABLE_LAB: "yes"
      GRANT_SUDO: "yes"

    storage:
      homeMountPath: /home/{username}
      extraVolumes:
        - name: ceres-flights
          persistentVolumeClaim:
            claimName: ceres-flights
      extraVolumeMounts:
        - name: ceres-flights
          mountPath: /home/{username}/flights

    cmd: "start-singleuser.sh"

    # start as root, we drop privs once NB_USER is set by CustomGoogleOAuthenticator below
    uid: 0
  hub:
    image:
      name: gcr.io/ceres-imaging-science/improc-hub
      tag: latest
    imagePullSecret:
      registry: gcr.io
      username: _json_key
      password: |-
        {
          "type": "service_account",
		  # SECRETS DELETED
        }
    extraConfig:
      logo: |
        c.JupyterHub.logo_file = '/usr/local/share/jupyterhub/static/images/ceres-logo.svg'
      useCeresOAuthenticator: |
        c.JupyterHub.authenticator_class = CeresOAuthenticator
  prePuller:
    hook:
      enabled: false

  auth:
    admin:
      users:
        - SECRETS DELETED
    type: google
    google:
	  # SECRETS DELETED

    state:
      enabled: true
      cryptoKey: SECRETS DELETED

debug:
  enabled: true
#2

Which pod should be responding to the acme challenge, and what’s the path of loadbalancer/service/route that the request should be taking from public-proxy to that pod?

I notice the kube-lego pod(s) and service are no longer present, I’m guessing that autohttps is taking over this roll?