This is a z2jh 0.9.0 deployment on Google Cloud.
One user continually gets a dialog saying Server Not Running
Your server at /user/xxxxxxxxxxxxxx/ is not running. Would you like to restart it? Restart/Dismiss
In the hub pod container logs are a bunch of entries like this:
[W 2020-05-27 13:26:49.448 JupyterHub proxy:355] Updating route for /user/xxxxxxxxxxx/ (http://10.4.14.44:8888 → Server(url=http://10.4.14.43:8888/user/xxxxxxxxxxxxxx/, bind_url=http://10.4.14.43:8888/user/xxxxxxxxxxxx/))
[I 2020-05-27 13:26:49.449 JupyterHub proxy:262] Adding user xxxxxxxxxxxx to proxy /user/xxxxxxxxxxxxxx/ => http://10.4.14.43:8888
But the user’s pod is on 10.4.14.44, not 10.4.14.43:
kubectl get pod -o wide --namespace jhub NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ... jupyter-xxxxxxxxxxxxxxxx 1/1 Running 0 8h 10.4.14.44 gke-jhub-big-user-pool-xxxxxxxxxx <none> <none>
Indeed, there are previous messages in the hub logs showing the route updated to 10.4.14.44 and the user added at that correct IP address, but they are followed a second later by messages of the form above, pointing to the wrong IP address.
Any ideas on how to fix this? Grateful for any suggestions.