Proxy error 599 - when i try to make an offline installation

hello,

i just spent my night on this probleme and i didn’t find a solution.

I deployed a jupyterhub (zero to jupyterhub), in my personnal computer with 4 vms: one master, 2 workers and one docker repository (for the jupyterhub images). In this installation, only the master had internet access. For that i modified the value yaml (in the helm archive), so that the images point to my personnal repository.

With this procedure, the deployment was a success, jupyterhub was workin, i was happy;

Unfortunately, i tried to reproduce this procedure, in my production (no machine has internet access), and it didn’t work,

first time, i’ve had a problem with the hook-image-awaiter :

“client.go:505: [debug] Add/Modify event for hook-image-awaiter: ADDED
client.go:544: [debug] hook-image-awaiter: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:505: [debug] Add/Modify event for hook-image-awaiter: MODIFIED
client.go:544: [debug] hook-image-awaiter: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:505: [debug] Add/Modify event for hook-image-awaiter: MODIFIED
client.go:544: [debug] hook-image-awaiter: Jobs active: 1, jobs failed: 1, jobs succeeded: 0”

i read in this issue

i can disable the prepuller, and after that i got a new error :

" 2020-12-03 10:06:05.248 JupyterHub proxy:785] api_request to the proxy failed with status code 599, retrying…
[E 2020-12-03 10:06:05.249 JupyterHub app:2844]
Traceback (most recent call last):
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py”, line 2842, in launch_instance_async
await self.start()
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py”, line 2615, in start
await self.proxy.get_all_routes()
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py”, line 832, in get_all_routes
resp = await self.api_request(’’, client=client)
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py”, line 796, in api_request
result = await exponential_backoff(
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py”, line 179, in exponential_backoff
raise TimeoutError(fail_message)
TimeoutError: Repeated api_request to proxy path “” failed"

If anyone has any idea how to fix thos problem, i would be grateful to him.

Are you using a local container registry? Can you show us your Z2JH config (redacted as necessary)?

1 Like

Hello Manics,

thank you for your answer. Yes that’s what i do, I installed z2jh from my private repository, and it worked as long as the master node had internet access.

the config is minimal :

“[root] # cat config.yaml
proxy:
secretToken: “********”
singleuser:
storage:
dynamic:
storageClass: local-storage”

and the values.yaml modified to point to my private registry :

# custom can contain anything you want to pass to the hub pod, as all passed
# Helm template values will be made available there.
custom: {}

# imagePullSecret is configuration to create a k8s Secret that Helm chart's pods
# can get credentials from to pull their images.
imagePullSecret:
  create: false
  automaticReferenceInjection: true
  registry: ''
  username: ''
  email: ''
  password: ''
# imagePullSecrets is configuration to reference the k8s Secret resources the
# Helm chart's pods can get credentials from to pull their images.
imagePullSecrets: []


# hub relates to the hub pod, responsible for running JupyterHub, its configured
# Authenticator class KubeSpawner, and its configured Proxy class
# ConfigurableHTTPProxy. KubeSpawner creates the user pods, and
# ConfigurableHTTPProxy speaks with the actual ConfigurableHTTPProxy server in
# the proxy pod.
hub:
  service:
    type: ClusterIP
    annotations: {}
    ports:
      nodePort:
    loadBalancerIP:
  baseUrl: /
  cookieSecret:
  publicURL:
  initContainers: []
  fsGid: 1000
  nodeSelector: {}
  tolerations: []
  concurrentSpawnLimit: 64
  consecutiveFailureLimit: 5
  activeServerLimit:
  deploymentStrategy:
    ## type: Recreate
    ## - sqlite-pvc backed hubs require the Recreate deployment strategy as a
    ##   typical PVC storage can only be bound to one pod at the time.
    ## - JupyterHub isn't designed to support being run in parallell. More work
    ##   needs to be done in JupyterHub itself for a fully highly available (HA)
    ##   deployment of JupyterHub on k8s is to be possible.
    type: Recreate
  db:
    type: sqlite-pvc
    upgrade:
    pvc:
      annotations: {}
      selector: {}
      accessModes:
        - ReadWriteOnce
      storage: 1Gi
      subPath:
      storageClassName:
    url:
    password:
  labels: {}
  annotations: {}
  command: []
  args: []
  extraConfig: {}
  extraConfigMap: {}
  extraEnv: {}
  extraContainers: []
  extraVolumes: []
  extraVolumeMounts: []
  image:
    name: myipregistry:5000/k8s-hub
    tag: '0.10.4'
    pullPolicy: ''
    pullSecrets: []
  resources:
    requests:
      cpu: 200m
      memory: 512Mi
  containerSecurityContext:
    runAsUser: 1000
    runAsGroup: 1000
    allowPrivilegeEscalation: false
  services: {}
  pdb:
    enabled: true
    minAvailable: 1
  networkPolicy:
    enabled: true
    ingress: []
    ## egress for JupyterHub already includes Kubernetes internal DNS and
    ## access to the proxy, but can be restricted further, but ensure to allow
    ## access to the Kubernetes API server that couldn't be pinned ahead of
    ## time.
    ##
    egress:
      - to:
          - ipBlock:
              cidr: 0.0.0.0/0
    interNamespaceAccessLabels: ignore
    allowedIngressPorts: []
  allowNamedServers: false
  namedServerLimitPerUser:
  authenticatePrometheus:
  redirectToServer:
  shutdownOnLogout:
  templatePaths: []
  templateVars: {}
  livenessProbe:
    enabled: false
    initialDelaySeconds: 180
    periodSeconds: 10
    failureThreshold: 3
    timeoutSeconds: 1
  readinessProbe:
    enabled: true
    initialDelaySeconds: 0
    periodSeconds: 2
    failureThreshold: 3
    timeoutSeconds: 1
  existingSecret:

rbac:
  enabled: true


# proxy relates to the proxy pod, the proxy-public service, and the autohttps
# pod and proxy-http service.
proxy:
  secretToken: ''
  annotations: {}
  deploymentStrategy:
    ## type: Recreate
    ## - JupyterHub's interaction with the CHP proxy becomes a lot more robust
    ##   with this configuration. To understand this, consider that JupyterHub
    ##   during startup will interact a lot with the k8s service to reach a
    ##   ready proxy pod. If the hub pod during a helm upgrade is restarting
    ##   directly while the proxy pod is making a rolling upgrade, the hub pod
    ##   could end up running a sequence of interactions with the old proxy pod
    ##   and finishing up the sequence of interactions with the new proxy pod.
    ##   As CHP proxy pods carry individual state this is very error prone. One
    ##   outcome when not using Recreate as a strategy has been that user pods
    ##   have been deleted by the hub pod because it considered them unreachable
    ##   as it only configured the old proxy pod but not the new before trying
    ##   to reach them.
    type: Recreate
    ## rollingUpdate:
    ## - WARNING:
    ##   This is required to be set explicitly blank! Without it being
    ##   explicitly blank, k8s will let eventual old values under rollingUpdate
    ##   remain and then the Deployment becomes invalid and a helm upgrade would
    ##   fail with an error like this:
    ##
    ##     UPGRADE FAILED
    ##     Error: Deployment.apps "proxy" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy `type` is 'Recreate'
    ##     Error: UPGRADE FAILED: Deployment.apps "proxy" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy `type` is 'Recreate'
    rollingUpdate:
  # service relates to the proxy-public service
  service:
    type: LoadBalancer
    labels: {}
    annotations: {}
    nodePorts:
      http:
      https:
    extraPorts: []
    loadBalancerIP:
    loadBalancerSourceRanges: []
  # chp relates to the proxy pod, which is responsible for routing traffic based
  # on dynamic configuration sent from JupyterHub to CHP's REST API.
  chp:
    containerSecurityContext:
      runAsUser: 65534  # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
    image:
      name: myipregistry:5000/configurable-http-proxy
      tag: 4.2.2
      pullPolicy: ''
      pullSecrets: []
    extraCommandLineFlags: []
    livenessProbe:
      enabled: true
      initialDelaySeconds: 60
      periodSeconds: 10
    readinessProbe:
      enabled: true
      initialDelaySeconds: 0
      periodSeconds: 2
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
    extraEnv: {}
    nodeSelector: {}
    tolerations: []
    networkPolicy:
      enabled: true
      ingress: []
      egress:
        - to:
            - ipBlock:
                cidr: 0.0.0.0/0
      interNamespaceAccessLabels: ignore
      allowedIngressPorts: [http, https]
  # traefik relates to the autohttps pod, which is responsible for TLS
  # termination when proxy.https.type=letsencrypt.
  traefik:
    containerSecurityContext:
      runAsUser: 65534  # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
    image:
      name: traefik
      # tag for Traefik should be thoroughly tested before bumping, because it
      # has been prone to have intermittency issues in the past. See for example
      tag: v2.3.2 
      pullPolicy: ''
      pullSecrets: []
    hsts:
      includeSubdomains: false
      preload: false
      maxAge: 15724800 # About 6 months
    resources: {}
    extraEnv: {}
    extraVolumes: []
    extraVolumeMounts: []
    extraStaticConfig: {}
    extraDynamicConfig: {}
    nodeSelector: {}
    tolerations: []
    extraPorts: []
    networkPolicy:
      enabled: true
      ingress: []
      egress:
        - to:
            - ipBlock:
                cidr: 0.0.0.0/0
      interNamespaceAccessLabels: ignore
      allowedIngressPorts: [http, https]
  secretSync:
    containerSecurityContext:
      runAsUser: 65534  # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
    image:
      name: myipregistry:5000/k8s-secret-sync
      tag: '0.10.4'
      pullPolicy: ''
      pullSecrets: []
    resources: {}
  labels: {}
  pdb:
    enabled: true
    minAvailable: 1
  https:
    enabled: false
    type: letsencrypt
    #type: letsencrypt, manual, offload, secret
    letsencrypt:
      contactEmail: ''
      acmeServer: https://acme-v02.api.letsencrypt.org/directory
    manual:
      key:
      cert:
    secret:
      name: ''
      key: tls.key
      crt: tls.crt
    hosts: []


# auth relates to the configuration of JupyterHub's Authenticator class.
auth:
  type: dummy
  whitelist:
    users:
  admin:
    access: true
    users:
  dummy:
    password:
  ldap:
    dn:
      search: {}
      user: {}
    user: {}
  state:
    enabled: false
    cryptoKey:


# singleuser relates to the configuration of KubeSpawner which runs in the hub
# pod, and its spawning of user pods such as jupyter-myusername.
singleuser:
  podNameTemplate:
  extraTolerations: []
  nodeSelector: {}
  extraNodeAffinity:
    required: []
    preferred: []
  extraPodAffinity:
    required: []
    preferred: []
  extraPodAntiAffinity:
    required: []
    preferred: []
  networkTools:
    image:
      name: myipregistry:5000/k8s-network-tools
      tag: '0.10.4'
      pullPolicy: ''
      pullSecrets: []
  cloudMetadata:
    # block set to true will append a privileged initContainer using the
    # iptables to block the sensitive metadata server at the provided ip.
    blockWithIptables: true
    ip: 169.254.169.254
  networkPolicy:
    enabled: true
    ingress: []
    egress:
      # Required egress to communicate with the hub and DNS servers will be
      # augmented to these egress rules.
      #
      # This default rule explicitly allows all outbound traffic from singleuser
      # pods, except to a typical IP used to return metadata that can be used by
      # someone with malicious intent.
      - to:
          - ipBlock:
              cidr: 0.0.0.0/0
              except:
                - 169.254.169.254/32
    interNamespaceAccessLabels: ignore
    allowedIngressPorts: []
  events: true
  extraAnnotations: {}
  extraLabels:
    hub.jupyter.org/network-access-hub: 'true'
  extraEnv: {}
  lifecycleHooks: {}
  initContainers: []
  extraContainers: []
  uid: 1000
  fsGid: 2206
  serviceAccountName:
  storage:
    type: dynamic
    extraLabels: {}
    extraVolumes: []
    extraVolumeMounts: []
    static:
      pvcName:
      subPath: '{username}'
    capacity: 10Gi
    homeMountPath: /app/cfvr
    dynamic:
      storageClass:
      pvcNameTemplate: claim-{username}{servername}
      volumeNameTemplate: volume-{username}{servername}
      storageAccessModes: [ReadWriteOnce]
  image:
    name: myipregistry:5000/jupyterhub/k8s-singleuser-sample
    tag: '0.10.6'
    pullPolicy: ''
    pullSecrets: []
  startTimeout: 300
  cpu:
    limit:
    guarantee:
  memory:
    limit:
    guarantee: 1G
  extraResource:
    limits: {}
    guarantees: {}
  cmd: jupyterhub-singleuser
  defaultUrl:
  extraPodConfig: {}
  profileList: []


# scheduling relates to the user-scheduler pods and user-placeholder pods.
scheduling:
  userScheduler:
    enabled: true
    replicas: 2
    logLevel: 4
    plugins:
      score:
        disabled:
          - name: SelectorSpread
          - name: TaintToleration
          - name: PodTopologySpread
          - name: NodeResourcesBalancedAllocation
          - name: NodeResourcesLeastAllocated
          # Disable plugins to be allowed to enable them again with a different
          # weight and avoid an error.
          - name: NodePreferAvoidPods
          - name: NodeAffinity
          - name: InterPodAffinity
          - name: ImageLocality
        enabled:
          - name: NodePreferAvoidPods
            weight: 161051
          - name: NodeAffinity
            weight: 14631
          - name: InterPodAffinity
            weight: 1331
          - name: NodeResourcesMostAllocated
            weight: 121
          - name: ImageLocality
            weight: 11
    containerSecurityContext:
      runAsUser: 65534  # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
    image:
      name: myipregistry:5000/kube-scheduler
      tag: v1.19.2
      pullPolicy: ''
      pullSecrets: []
    nodeSelector: {}
    tolerations: []
    pdb:
      enabled: true
      minAvailable: 1
    resources:
      requests:
        cpu: 50m
        memory: 256Mi
  podPriority:
    enabled: false
    globalDefault: false
    defaultPriority: 0
    userPlaceholderPriority: -10
  userPlaceholder:
    enabled: true
    replicas: 0
    containerSecurityContext:
      runAsUser: 65534  # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
  corePods:
    nodeAffinity:
      matchNodePurpose: prefer
  userPods:
    nodeAffinity:
      matchNodePurpose: prefer


# prePuller relates to the hook|continuous-image-puller DaemonsSets
prePuller:
  annotations: {}
  resources:
    requests:
      cpu: 0
      memory: 0
  containerSecurityContext:
    runAsUser: 65534  # nobody user
    runAsGroup: 65534 # nobody group
    allowPrivilegeEscalation: false
  extraTolerations: []
  # hook relates to the hook-image-awaiter Job and hook-image-puller DaemonSet
  hook:
    enabled: True
    # image and the configuration below relates to the hook-image-awaiter Job
    image:
      name: myipregistry:5000/k8s-image-awaiter
      tag: '0.10.4'
      pullPolicy: ''
      pullSecrets: []
    containerSecurityContext:
      runAsUser: 65534  # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
    podSchedulingWaitDuration: 10
    nodeSelector: {}
    tolerations: []
    resources:
      requests:
        cpu: 0
        memory: 0
  continuous:
    enabled: true
  pullProfileListImages: true
  extraImages: {}
  pause:
    containerSecurityContext:
      runAsUser: 65534  # nobody user
      runAsGroup: 65534 # nobody group
      allowPrivilegeEscalation: false
    image:
      name: myipregistry:5000/pause
      tag: '3.2' 
      pullPolicy: ''
      pullSecrets: []

ingress:
  enabled: false
  annotations: {}
  hosts: []
  pathSuffix: ''
  tls: []


cull:
  enabled: true
  users: false
  removeNamedServers: false
  timeout: 3600
  every: 600
  concurrency: 10
  maxAge: 0


debug:
  enabled: false

[root] # kubectl get pods -n jhub
NAME READY STATUS RESTARTS AGE
continuous-image-puller-8lgmc 1/1 Running 0 3m3s
continuous-image-puller-fb72z 1/1 Running 0 3m3s
hub-855785dc9f-w4hjp 0/1 Running 0 3m2s
proxy-58864974c5-sgbk6 1/1 Running 0 3m2s
user-scheduler-74fdd4b58f-mddzb 1/1 Running 0 3m2s
user-scheduler-74fdd4b58f-nws6q 1/1 Running 0 3m2s

[root] # kubectl logs hub-855785dc9f-w4hjp -n jhub
No config at /etc/jupyterhub/config/values.yaml
Loading /etc/jupyterhub/secret/values.yaml
[I 2020-12-03 20:29:36.010 JupyterHub app:2332] Running JupyterHub version 1.2.1
[I 2020-12-03 20:29:36.010 JupyterHub app:2362] Using Authenticator: dummyauthenticator.dummyauthenticator.DummyAuthenticator
[I 2020-12-03 20:29:36.010 JupyterHub app:2362] Using Spawner: kubespawner.spawner.KubeSpawner-0.14.1
[I 2020-12-03 20:29:36.010 JupyterHub app:2362] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.2.1
[I 2020-12-03 20:29:36.011 JupyterHub app:1412] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[W 2020-12-03 20:29:36.035 JupyterHub app:1687] No admin users, admin interface will be unavailable.
[W 2020-12-03 20:29:36.035 JupyterHub app:1688] Add any administrative users to c.Authenticator.admin_users in config.
[I 2020-12-03 20:29:36.035 JupyterHub app:1717] Not using allowed_users. Any authenticated user will be allowed.
[I 2020-12-03 20:29:36.074 JupyterHub app:2399] Initialized 0 spawners in 0.001 seconds
[I 2020-12-03 20:29:36.075 JupyterHub app:2611] Not starting proxy
[W 2020-12-03 20:29:56.110 JupyterHub proxy:785] api_request to the proxy failed with status code 599, retrying…
[W 2020-12-03 20:30:16.259 JupyterHub proxy:785] api_request to the proxy failed with status code 599, retrying…
[E 2020-12-03 20:30:16.259 JupyterHub app:2844]
Traceback (most recent call last):
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py”, line 2842, in launch_instance_async
await self.start()
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py”, line 2615, in start
await self.proxy.get_all_routes()
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py”, line 832, in get_all_routes
resp = await self.api_request(’’, client=client)
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py”, line 796, in api_request
result = await exponential_backoff(
File “/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py”, line 179, in exponential_backoff
raise TimeoutError(fail_message)
TimeoutError: Repeated api_request to proxy path “” failed.

Thank you again

I’m encountering a similar issue, where JupyterHub kept complaining

api_request to the proxy failed with status code 503, retrying

and eventually exited. It turns out that api requests were sent over a proxy because the http_proxy and https_proxy environmental variables were set. This issue can be resolved by setting NO_PROXY (only the uppercased version works!) to “localhost,127.0.0.1”. Perhaps it’s not pretty related to OP’s problem, but this is the first post I came across with Google, so I hope this solution might help more users in the same circumstance.

Hello I am having the same problems described here. I did not fully understand your solution. Could you re-explained me what you mean by setting NO_PROXY please?
Thanks

Hello,
Trying to do the same thing as you here, did you make it work at the end ?
Thanks

This solution only applies if you were using proxies for JupyterHub. If your JupyterHub is also deployed on Kubernetes like mine, you could run kubectl edit deployment jupyterhub and add an env entry filled as follows:

env:
- name: NO_PROXY
  value: "localhost,127.0.0.1"

Or if you were starting JupyterHub via shell you can try something like

NO_PROXY="localhost,127.0.0.1" jupyterhub