Hello,
We are using Jupyterlab 1.0.0 / ConfigProxy and we are seeing an issue occasionally where a user will be unable to get a notebook created and instead get a 503 error.
The logs are nothing exciting only showing that there is a Proxy error: read ECONNRESET
Please see below:
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub 19:20:00.404 - error: [ConfigProxy] Proxy error: Error: read ECONNRESET
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub at TCP.onStreamRead (internal/stream_base_commons.js:167:27)
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub 19:20:00.404 - error: [ConfigProxy] 503 GET /[meff/jupyterhub/user/some.user@someorg.com](mailto:meff/jupyterhub/user/some.user@someorg.com)/
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub 19:20:00.405 - error: [ConfigProxy] Proxy error: Error: read ECONNRESET
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub at TCP.onStreamRead (internal/stream_base_commons.js:167:27)
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub 19:20:00.405 - error: [ConfigProxy] 503 GET /[meff/jupyterhub/user/some.user@someorg.com](mailto:meff/jupyterhub/user/some.user@someorg.com)/
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub 19:20:00.405 - error: [ConfigProxy] Proxy error: Error: read ECONNRESET
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub at TCP.onStreamRead (internal/stream_base_commons.js:167:27)
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub 19:20:00.405 - error: [ConfigProxy] 503 GET /[meff/jupyterhub/user/some.user@someorg.com](mailto:meff/jupyterhub/user/some.user@someorg.com)/
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub [D 2020-08-26 19:20:00.406 JupyterHub pages:522] No template for 503
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub [I 2020-08-26 19:20:00.407 JupyterHub log:174] 200 GET /[meff/jupyterhub/hub/error/503?url=%2Fmeff%2Fjupyterhub%2Fuser%2Fsome.user@someorg.com](mailto:meff/jupyterhub/hub/error/503?url=%2Fmeff%2Fjupyterhub%2Fuser%2Fsome.user@someorg.com)%2F (@10.253.37.166) 1.43ms
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub [D 2020-08-26 19:20:00.408 JupyterHub pages:522] No template for 503
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub [I 2020-08-26 19:20:00.409 JupyterHub log:174] 200 GET /[meff/jupyterhub/hub/error/503?url=%2Fmeff%2Fjupyterhub%2Fuser%2Fsome.user@someorg.com](mailto:meff/jupyterhub/hub/error/503?url=%2Fmeff%2Fjupyterhub%2Fuser%2Fsome.user@someorg.com)%2F (@10.253.37.166) 1.36ms
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub [D 2020-08-26 19:20:00.409 JupyterHub pages:522] No template for 503
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub [I 2020-08-26 19:20:00.410 JupyterHub log:174] 200 GET /[meff/jupyterhub/hub/error/503?url=%2Fmeff%2Fjupyterhub%2Fuser%2Fsome.user@someorg.com](mailto:meff/jupyterhub/hub/error/503?url=%2Fmeff%2Fjupyterhub%2Fuser%2Fsome.user@someorg.com)%2F (@10.253.37.166) 1.81ms
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub 19:20:06.329 - error: [ConfigProxy] Proxy error: Error: read ECONNRESET
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub at TCP.onStreamRead (internal/stream_base_commons.js:167:27)
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub 19:20:06.329 - error: [ConfigProxy] 503 GET /[meff/jupyterhub/user/some.user@someorg.com](mailto:meff/jupyterhub/user/some.user@someorg.com)/
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub [D 2020-08-26 19:20:02.232 JupyterHub reflector:263] events watcher timeout
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub [D 2020-08-26 19:20:02.232 JupyterHub reflector:202] Connecting events watcher
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub 19:20:02.244 - error: [ConfigProxy] Proxy error: Error: read ECONNRESET
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub at TCP.onStreamRead (internal/stream_base_commons.js:167:27)
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub 19:20:02.244 - error: [ConfigProxy] 503 GET /[meff/jupyterhub/user/some.user@someorg.com](mailto:meff/jupyterhub/user/some.user@someorg.com)/
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub [D 2020-08-26 19:20:02.245 JupyterHub pages:522] No template for 503
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub [I 2020-08-26 19:20:02.246 JupyterHub log:174] 200 GET /[meff/jupyterhub/hub/error/503?url=%2Fmeff%2Fjupyterhub%2Fuser%2Fsome.user@someorg.com](mailto:meff/jupyterhub/hub/error/503?url=%2Fmeff%2Fjupyterhub%2Fuser%2Fsome.user@someorg.com)%2F (@10.253.37.166) 1.17ms
meff-dss-jupyterhub-6696b998f6-sdpmw dss-jupyterhub 19:20:02.253 - debug: [ConfigProxy] PROXY WEB /[meff/jupyterhub/user/some.user@someorg.com](mailto:meff/jupyterhub/user/some.user@someorg.com)/ to <http://10.234.68.250:8888>
It also seems to only affect a single user, as other users are able to work and spawn notebooks during this time.
It almost seems like it’s keeping an old session in memory and even though this session doesn’t exist anymore, it’s “stuck”.
What is storing the debug: [ConfigProxy] PROXY WEB /[meff/jupyterhub/user/some.user@someorg.com](mailto:meff/jupyterhub/user/some.user@someorg.com)/ to <http://10.234.68.250:8888> line and is it possible for us to clear this?
Restarting the Jupyterhub pod always resolves this issue, but it crops up every week or so so i’d like to understand more on why this is happening and what we can do to resolve it.
Please let me know what other information I can provide!