Jupyter on EKS with TLS Listener won't respond

JupyterHub Zero to JupyterHub on Kubernetes
1

Hello!
I am attempting to run jupyterhub on EKS. I have an internal VPC and load balancer that can accept external traffic on port 443. I have used ambassador to expose kubernetes services with this same VPC.
To try to get jupyterhub exposed, I create a TLS listener for a Network Load Balancer and added a cert for the custom domain I wish to use.
Currently, I do not get a response from the jupyterhub service when I put the endpoint into my browser.

Expected behaviour
I expect to be able to access jupyterhub via my custom domain

Actual behaviour
Currently, the browser never loads the page, and I see this output from curl:

(base) HQSML-1712547:amp-jhub ntorba605$ curl -v custom-host

  • Trying ip:443…
  • TCP_NODELAY set
  • Connected to host (ip) port 443 (#0)
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: path/to/cacert.pem
    CApath: none
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • TLSv1.2 (IN), TLS handshake, Server finished (14):
  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.2 (OUT), TLS handshake, Finished (20):
  • TLSv1.2 (IN), TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
  • ALPN, server accepted to use http/1.1
  • Server certificate:
  • subject: C=US; postalCode=19103; ST=Pennsylvania; L=Philadelphia; street=1 Comcast Center; O=Comcast Corporation; OU=NSO; CN=custom-host
  • start date: Oct 19 00:00:00 2020 GMT
  • expire date: Oct 19 23:59:59 2021 GMT
  • subjectAltName: host “host” matched cert’s “host”
  • issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO RSA Organization Validation Secure Server CA
  • SSL certificate verify ok.

GET / HTTP/1.1
Host: host
User-Agent: curl/7.65.3
Accept: /

  • TLSv1.2 (IN), TLS alert, close notify (256):
  • Empty reply from server
  • Closing connection 0
  • TLSv1.2 (OUT), TLS alert, close notify (256):
    curl: (52) Empty reply from server
    This looks like the the request was received, but I can’t figure out why it looks like it never actually gets sent to the service.

How to reproduce
I’m using an internal vpc and internal network load balancer on AWS EKS.
I’m using the following annoations on my load balancer service to get this working

annotations:
    meta.helm.sh/release-name: jhub
    meta.helm.sh/release-namespace: jhub
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-internal: 10.0.0.0/8
    service.beta.kubernetes.io/aws-load-balancer-type: nlb

I created a TLS listener with a cert for the custom domain I wish to use.
I also enabled proxy protocol v2.

I have the custom domain setup to point to the external-ip that is generated for the load balancer