Running Jupyterhub on kubernetes with traefik and etcd

JupyterHub Zero to JupyterHub on Kubernetes
1

Hey Folks,
So I just started playing around with jupyterhub with traefik backed by etcd
I have an etcd cluster (v3.3.8) running and then I deploy traefik (v1.7.5) using the deployments here: GitHub - maulikjs/traefik-poc

my config looks like

   defaultentrypoints = ["http"]
   debug = true
   logLevel = "INFO"
   [api]
   dashboard = true
   entrypoint = "auth_api"
   [wss]
   protocol = "http"
   [entryPoints.http]
   address = "127.0.0.1:8000"
   [entryPoints.auth_api]
   address = "127.0.0.1:8099"
   [entryPoints.auth_api.auth.basic]
   users = [ "abc:$apr1$eS/j3kum$q/X2khsIEG/bBGsteP.x./",]
   [etcd]
   endpoint = "example-etcd-cluster-client:2379"
   prefix = "/traefik/"
   useapiv3 = true
   watch = true

I keep getting an error

E0623 07:02:04.516704       1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: v1.ServiceList.Items: []v1.Service: v1.Service.ObjectMeta: v1.ObjectMeta.readObjectFieldAsBytes: expect : after object field, but found p, error found in #10 byte of ...|:{},"k:{\"port\":80,|..., bigger context ...|{},"f:app":{}}},"f:spec":{"f:ports":{".":{},"k:{\"port\":80,\"protocol\":\"TCP\"}":{".":{},"f:name":|...

What might be going on?
this is what my rbac looks like

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
    - extensions
    resources:
    - ingresses/status
    verbs:
    - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: proxy-poc

Any pointers here are helpful. This is my what jupyterhub config looks like

  jupyterhub_config.py: |
      from jupyterhub_traefik_proxy import TraefikEtcdProxy
      # mark the proxy as externally managed
      c.TraefikEtcdProxy.should_start = False
      # traefik api endpoint login password
      c.TraefikEtcdProxy.traefik_api_password = "123"
      # traefik api endpoint login username
      c.TraefikEtcdProxy.traefik_api_username = "abc"
      # etcd url where it accepts client requests
      c.TraefikEtcdProxy.kv_url = "example-etcd-cluster-client:2379"
      # configure JupyterHub to use TraefikEtcdProxy
      c.JupyterHub.proxy_class = TraefikEtcdProxy