Jupyterhub keep crashing with error server unavailable or unreachable Jupyterhub version 2.3.1

remoteEntry.7dd115740677e3b366fe.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/notebook (required ^3.6.3)
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.7dd115740677e3b366fe.js:1 Unsatisfied version 2.1.1 from @jupyterlab/application-top of shared singleton module @lumino/coreutils (required ^1.11.0)
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.7dd115740677e3b366fe.js:1 Unsatisfied version 2.1.1 from @jupyterlab/application-top of shared singleton module @lumino/signaling (required ^1.10.0)
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.7dd115740677e3b366fe.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/rendermime (required ^3.6.3)
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.7dd115740677e3b366fe.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/codeeditor (required ^3.6.3)
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.7dd115740677e3b366fe.js:1 Unsatisfied version 2.0.0 from @jupyterlab/application-top of shared singleton module @lumino/messaging (required ^1.10.0)
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.7dd115740677e3b366fe.js:1 Unsatisfied version 4.1.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/apputils (required ^3.6.3)
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.7dd115740677e3b366fe.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/settingregistry (required ^3.6.3)
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.7dd115740677e3b366fe.js:1 No satisfying version (^3.6.3) of shared module @jupyterlab/nbformat found in shared scope default.
Available versions: 4.0.2 from @jupyterlab/application-top
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.7dd115740677e3b366fe.js:1 Unsatisfied version 2.1.1 from @jupyterlab/application-top of shared singleton module @lumino/disposable (required ^1.10.0)
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.7dd115740677e3b366fe.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/codemirror (required ^3.6.3)
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.7dd115740677e3b366fe.js:1 Unsatisfied version 6.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/coreutils (required ^5.6.3)
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.7dd115740677e3b366fe.js:1 Unsatisfied version 7.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/services (required ^6.6.3)
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.7dd115740677e3b366fe.js:1 Unsatisfied version 2.1.1 from @jupyterlab/application-top of shared singleton module @lumino/widgets (required ^1.37.2)
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.7dd115740677e3b366fe.js:1 Unsatisfied version 2.0.0 from @jupyterlab/application-top of shared singleton module @lumino/algorithm (required ^1.9.0)
c @ remoteEntry.7dd115740677e3b366fe.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 18.2.0 from @jupyterlab/application-top of shared singleton module react (required ^17.0.1)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 2.1.1 from @jupyterlab/application-top of shared singleton module @lumino/coreutils (required ^1.11.0)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 2.1.1 from @jupyterlab/application-top of shared singleton module @lumino/signaling (required ^1.10.0)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 18.2.0 from @jupyterlab/application-top of shared singleton module react-dom (required ^17.0.1)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 2.0.0 from @jupyterlab/application-top of shared singleton module @lumino/algorithm (required ^1.9.0)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/translation (required ^3.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 7.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/services (required ^6.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/statusbar (required ^3.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 No satisfying version (^3.6.0-alpha.5) of shared module @jupyterlab/nbformat found in shared scope default.
Available versions: 4.0.2 from @jupyterlab/application-top
p @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/codeeditor (required ^3.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 4.1.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/apputils (required ^3.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 2.0.0 from @jupyterlab/application-top of shared singleton module @lumino/messaging (required ^1.10.0)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 6.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/coreutils (required ^5.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 2.1.1 from @jupyterlab/application-top of shared singleton module @lumino/widgets (required ^1.33.0)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/docmanager (required ^3.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/mainmenu (required ^3.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 2.0.0 from @jupyterlab/application-top of shared singleton module @lumino/properties (required ^1.8.0)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/codemirror (required ^3.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 2.1.1 from @jupyterlab/application-top of shared singleton module @lumino/commands (required ^1.19.0)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 No satisfying version (^3.6.0-alpha.5) of shared module @jupyterlab/docregistry found in shared scope default.
Available versions: 4.0.2 from @jupyterlab/application-top
p @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 2.1.1 from @jupyterlab/application-top of shared singleton module @lumino/dragdrop (required ^1.13.0)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/application (required ^3.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/ui-components (required ^3.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/statedb (required ^3.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/rendermime (required ^3.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 No satisfying version (^3.6.0-alpha.5) of shared module @jupyterlab/outputarea found in shared scope default.
Available versions: 4.0.2 from @jupyterlab/application-top
p @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/filebrowser (required ^3.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 2.0.0 from @jupyterlab/application-top of shared singleton module @lumino/domutils (required ^1.8.0)
f @ remoteEntry.4318d00d399dc15e3535.js:1
remoteEntry.4318d00d399dc15e3535.js:1 Unsatisfied version 4.0.2 from @jupyterlab/application-top of shared singleton module @jupyterlab/settingregistry (required ^3.6.0-alpha.5)
f @ remoteEntry.4318d00d399dc15e3535.js:1
812.8b0e8939a4ace07ad5c8.js?v=8b0e8939a4ace07ad5c8:1 Failed to create module: package: nbdime-jupyterlab; module: ./extension
F @ 812.8b0e8939a4ace07ad5c8.js?v=8b0e8939a4ace07ad5c8:1
812.8b0e8939a4ace07ad5c8.js?v=8b0e8939a4ace07ad5c8:1 Failed to create module: package: @jupyterlab/git; module: ./extension
F @ 812.8b0e8939a4ace07ad5c8.js?v=8b0e8939a4ace07ad5c8:1
812.8b0e8939a4ace07ad5c8.js?v=8b0e8939a4ace07ad5c8:1 Error: Shared module codemirror doesn’t exist in shared scope default
at i (remoteEntry.7dd115740677e3b366fe.js:1:4660)
at remoteEntry.7dd115740677e3b366fe.js:1:5702
at remoteEntry.7dd115740677e3b366fe.js:1:5610
at 8681 (remoteEntry.7dd115740677e3b366fe.js:1:6407)
at remoteEntry.7dd115740677e3b366fe.js:1:6898
at Array.forEach ()
at E.f.consumes (remoteEntry.7dd115740677e3b366fe.js:1:6728)
at remoteEntry.7dd115740677e3b366fe.js:1:1020
at Array.reduce ()
at E.e (remoteEntry.7dd115740677e3b366fe.js:1:998)
(anonymous) @ 812.8b0e8939a4ace07ad5c8.js?v=8b0e8939a4ace07ad5c8:1
812.8b0e8939a4ace07ad5c8.js?v=8b0e8939a4ace07ad5c8:1 Error: Shared module codemirror doesn’t exist in shared scope default
at o (remoteEntry.4318d00d399dc15e3535.js:1:6507)
at remoteEntry.4318d00d399dc15e3535.js:1:7591
at remoteEntry.4318d00d399dc15e3535.js:1:7499
at 8681 (remoteEntry.4318d00d399dc15e3535.js:1:9978)
at remoteEntry.4318d00d399dc15e3535.js:1:10797
at Array.forEach ()
at k.f.consumes (remoteEntry.4318d00d399dc15e3535.js:1:10627)
at remoteEntry.4318d00d399dc15e3535.js:1:1174
at Array.reduce ()
at k.e (remoteEntry.4318d00d399dc15e3535.js:1:1152)
(anonymous) @ 812.8b0e8939a4ace07ad5c8.js?v=8b0e8939a4ace07ad5c8:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Language pack ‘en_US’ not installed!
fetch @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
gmwcraig.png:1 Failed to load resource: the server responded with a status of 404 ()
github_username.png:1 Failed to load resource: the server responded with a status of 404 ()
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Terminal websocket closed: 1006
_onWSClose @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Connection lost, reconnecting in 0 seconds.
_reconnect @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Terminal websocket closed: 1006
_onWSClose @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Connection lost, reconnecting in 0 seconds.
_reconnect @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Terminal websocket closed: 1006
_onWSClose @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Connection lost, reconnecting in 0 seconds.
_reconnect @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/terminals/websocket/1’ failed:
_createSocket @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Terminal websocket closed: undefined
_onWSClose @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Connection lost, reconnecting in 0 seconds.
_reconnect @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/terminals/websocket/1’ failed:
_createSocket @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Terminal websocket closed: undefined
_onWSClose @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Connection lost, reconnecting in 0 seconds.
_reconnect @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/terminals/websocket/1’ failed:
_createSocket @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Terminal websocket closed: undefined
_onWSClose @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Connection lost, reconnecting in 0 seconds.
_reconnect @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/terminals/websocket/1’ failed:
_createSocket @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Terminal websocket closed: undefined
_onWSClose @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Connection lost, reconnecting in 3 seconds.
_reconnect @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/terminals/websocket/1’ failed:
_createSocket @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Terminal websocket closed: undefined
_onWSClose @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Connection lost, reconnecting in 13 seconds.
_reconnect @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/terminals/websocket/1’ failed:
_createSocket @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Terminal websocket closed: undefined
_onWSClose @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Connection lost, reconnecting in 60 seconds.
_reconnect @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/terminals/websocket/1’ failed:
_createSocket @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 Terminal websocket closed: undefined
_onWSClose @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
/hub/user/punjabi%2C%20gaurav/api/kernelspecs?1689689802853:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
/hub/user/punjabi%2C%20gaurav/api/contents/pso/pov-jupyter-notebooks?content=1&1689689803022:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
/hub/user/punjabi%2C%20gaurav/api/terminals?1689689803021:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
/user/punjabi%2C%20gaurav/lab/api/workspaces/default?1689689803544:1 Failed to load resource: the server responded with a status of 405 (Method Not Allowed)
/hub/user/punjabi%2C%20gaurav/api/me?1689689803020:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
/hub/user/punjabi%2C%20gaurav/api/sessions?1689689803024:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
/hub/user/punjabi%2C%20gaurav/api/kernels?1689689803024:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
index.css:1 Failed to load resource: the server responded with a status of 503 (Service Unavailable)
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_subscribe @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
factory @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_execute @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
o @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_subscribe @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
factory @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_execute @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
o @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_subscribe @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
factory @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_execute @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
o @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_subscribe @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
factory @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_execute @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
o @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_subscribe @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
factory @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_execute @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
o @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_subscribe @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
factory @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_execute @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
o @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_subscribe @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
factory @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_execute @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
o @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_subscribe @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
factory @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_execute @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
o @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_subscribe @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
factory @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_execute @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
o @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_subscribe @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
factory @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_execute @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
o @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_subscribe @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
factory @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_execute @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
o @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_subscribe @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
factory @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_execute @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
o @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1 WebSocket connection to ‘wss://jhub.eap.tceh.net/user/punjabi%2C%20gaurav/api/events/subscribe?token=JLCnIMEt96yLDG1mFz62xok21TVvyb’ failed:
(anonymous) @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_subscribe @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
factory @ jlab_core.f68a597bc4700114bad4.js?v=f68a597bc4700114bad4:1
_execute @ jlab_core.f68a597bc470011
Screenshot 2023-07-19 181813
14bad4.js?v=f68a597bc4700114bad4:1

You need to provide bit more context. What is the environment where single user servers are spawned? At a first glance, it seems like you are mixing packages from JupyterLab 3 and 4. They tend to not work together. Make sure you pin the JupyterLab version to <4.0.0 and retry.

1 Like

@mahendrapaipuri It is deployed on kubernetes running as service on cluster-IP we are using single user with image jupyter/minimal-notebook:hub-2.3.1 deployed in kubernetes pod.

Also another question how do I update conda if it comes very old version with the image?
as I am getting this error message
"
[Errno 13] Permission denied: ‘/opt/conda/lib/python3.10/site-packages/GitPython-3.1.31.dist-info/AUTHORS’ → ‘/opt/conda/lib/python3.10/site-packages/GitPython-3.1.31.dist-info/AUTHORS.c~’ () "

Are you installing any packages/extensions inside this image? The image you mentioned comes shipped with JupyterLab 3.* and it should work out-of-the-box if you are not changing anything in the image.

It seems like a permission issue. Are you updating conda as root user? I dont have a lot of experience using jupyter images.

@mahendrapaipuri yes we are installing awscli,saml2aws yes jupyterlab version is 3.4.6 with the image,

I didn’t get your point “it should work out-of-the-box if you are not changing anything in the image.”?

How do I get the latest conda while provisioning a single user environment do I need to update my base image?

The image should have JupyterLab 3.4.6 (as you said). But from the console log that you posted, there seems to be some packages coming from JupyterLab 4. So, I assume there are some packages that are being installed that are messing up JupyterLab version. For example, check remoteEntry.7dd115740677e3b366fe.js where is this js coming from?

Could you share your Hub and single user config?

IFROM jupyter/minimal-notebook:hub-2.3.1

USER root

COPY --chown=$NB_UID:$NB_GID docker-images/jhub/assets/vistra-poststart.sh /tmp/vistra-poststart.sh
COPY --chown=$NB_UID:$NB_GID docker-images/jhub/assets/saml2aws.profile /tmp/saml2aws.profile
COPY docker-images/jhub/assets/vistra-image.sh /personal_image.sh

RUN apt update && apt upgrade -y &&
bash /personal_image.sh &&
rm /personal_image.sh

USER $NB_UID

GPU image:

FROM nvidia/cuda:11.5.2-cudnn8-devel-ubuntu20.04

To update this image one can update the nvidia/cuda base image

AND

go through this notebook and pull in changes from github manually

it needs changes from base notebook and minimal python

START base notebook

https://github.com/jupyter/docker-stacks/tree/master/base-notebook

LAST UPDATE WAS COMMIT AUG 23 2022 e050d8c5678257e866c7e43aba9d568db84baa87

ARG NB_USER=“jovyan”
ARG NB_UID=“1000”
ARG NB_GID=“100”

Fix: DL4006 · hadolint/hadolint Wiki · GitHub

Fix: SC3014 · koalaman/shellcheck Wiki · GitHub

SHELL [“/bin/bash”, “-o”, “pipefail”, “-c”]

USER root

Install all OS dependencies for notebook server that starts but lacks all

features (e.g., download as all possible file formats)

ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update --yes &&
# - apt-get upgrade is run to patch known vulnerabilities in apt-get packages as
# the ubuntu base image is rebuilt too seldom sometimes (less than once a month)
apt-get upgrade --yes &&
apt-get install --yes --no-install-recommends
# - bzip2 is necessary to extract the micromamba executable.
bzip2
ca-certificates
fonts-liberation
locales
# - pandoc is used to convert notebooks to html files
# it’s not present in arm64 ubuntu image, so we install it here
pandoc
# - run-one - a wrapper script that runs no more
# than one unique instance of some command with a unique set of arguments,
# we use run-one-constantly to support RESTARTABLE option
run-one
sudo
# - tini is installed as a helpful container entrypoint that reaps zombie
# processes and such of the actual executable we want to start, see
# GitHub - krallin/tini: A tiny but valid `init` for containers for details.
tini
wget &&
apt-get clean && rm -rf /var/lib/apt/lists/* &&
echo “en_US.UTF-8 UTF-8” > /etc/locale.gen &&
locale-gen

Configure environment

ENV CONDA_DIR=/opt/conda
SHELL=/bin/bash
NB_USER=“${NB_USER}”
NB_UID=${NB_UID}
NB_GID=${NB_GID}
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
ENV PATH=“${CONDA_DIR}/bin:${PATH}”
HOME=“/home/${NB_USER}”

Copy a script that we will use to correct permissions after running certain commands

COPY docker-images/jhub/assets/fix-permissions /usr/local/bin/fix-permissions
RUN chmod a+rx /usr/local/bin/fix-permissions

Enable prompt color in the skeleton .bashrc before creating the default NB_USER

hadolint ignore=SC2016

RUN sed -i ‘s/^#force_color_prompt=yes/force_color_prompt=yes/’ /etc/skel/.bashrc && \

Add call to conda init script see bash - Python - Activate conda env through shell script - Stack Overflow

echo ‘eval “$(command conda shell.bash hook 2> /dev/null)”’ >> /etc/skel/.bashrc

Create NB_USER with name jovyan user with UID=1000 and in the ‘users’ group

and make sure these dirs are writable by the users group.

RUN echo “auth requisite pam_deny.so” >> /etc/pam.d/su &&
sed -i.bak -e ‘s/^%admin/#%admin/’ /etc/sudoers &&
sed -i.bak -e ‘s/^%sudo/#%sudo/’ /etc/sudoers &&
useradd -l -m -s /bin/bash -N -u “${NB_UID}” “${NB_USER}” &&
mkdir -p “${CONDA_DIR}” &&
chown “${NB_USER}:${NB_GID}” “${CONDA_DIR}” &&
chmod g+w /etc/passwd &&
fix-permissions “${HOME}” &&
fix-permissions “${CONDA_DIR}”

USER ${NB_UID}

Pin python version here, or set it to “default”

ARG PYTHON_VERSION=3.10

Setup work directory for backward-compatibility

RUN mkdir “/home/${NB_USER}/work” &&
fix-permissions “/home/${NB_USER}”

Download and install Micromamba, and initialize Conda prefix.

https://github.com/mamba-org/mamba#micromamba

Similar projects using Micromamba:

- Micromamba-Docker: https://github.com/mamba-org/micromamba-docker

- repo2docker: https://github.com/jupyterhub/repo2docker

Install Python, Mamba, Jupyter Notebook, Lab, and Hub

Generate a notebook server config

Cleanup temporary files and remove Micromamba

Correct permissions

Do all this in a single RUN command to avoid duplicating all of the

files across image layers when the permissions change

COPY --chown=“${NB_UID}:${NB_GID}” docker-images/jhub/assets/initial-condarc “${CONDA_DIR}/.condarc”
WORKDIR /tmp
RUN set -x &&
arch=$(uname -m) &&
if [ “${arch}” = “x86_64” ]; then
# Should be simpler, see https://github.com/mamba-org/mamba/issues/1437
arch=“64”;
fi &&
wget -qO /tmp/micromamba.tar.bz2
https://micromamba.snakepit.net/api/micromamba/linux-${arch}/latest” &&
tar -xvjf /tmp/micromamba.tar.bz2 --strip-components=1 bin/micromamba &&
rm /tmp/micromamba.tar.bz2 &&
PYTHON_SPECIFIER=“python=${PYTHON_VERSION}” &&
if [[ “${PYTHON_VERSION}” == “default” ]]; then PYTHON_SPECIFIER=“python”; fi &&
# Install the packages
./micromamba install
–root-prefix=“${CONDA_DIR}”
–prefix=“${CONDA_DIR}”
–yes
“${PYTHON_SPECIFIER}”
‘mamba’
‘notebook’
‘jupyterhub=2.3.1’
‘jupyterlab’ &&
rm micromamba &&
# Pin major.minor version of python
mamba list python | grep '^python ’ | tr -s ’ ’ | cut -d ’ ’ -f 1,2 >> “${CONDA_DIR}/conda-meta/pinned” &&
jupyter notebook --generate-config &&
mamba clean --all -f -y &&
npm cache clean --force &&
jupyter lab clean &&
rm -rf “/home/${NB_USER}/.cache/yarn” &&
fix-permissions “${CONDA_DIR}” &&
fix-permissions “/home/${NB_USER}”

RUN echo “jupyterhub version: $(jupyterhub --version)”
EXPOSE 8888

Configure container startup

ENTRYPOINT [“tini”, “-g”, “–”]
CMD [“start-notebook.sh”]

Copy local files as late as possible to avoid cache busting

COPY docker-images/jhub/assets/start.sh docker-images/jhub/assets/start-notebook.sh docker-images/jhub/assets/start-singleuser.sh /usr/local/bin/

Currently need to have both jupyter_notebook_config and jupyter_server_config to support classic and lab

COPY docker-images/jhub/assets/jupyter_server_config.py /etc/jupyter/

Fix permissions on /etc/jupyter as root

USER root

KEEP FOR VISTRA

RUN chmod a+rx /usr/local/bin/start.sh
/usr/local/bin/start-notebook.sh
/usr/local/bin/start-singleuser.sh
/etc/jupyter/jupyter_server_config.py

Legacy for Jupyter Notebook Server, see: #1205

RUN sed -re “s/c.ServerApp/c.NotebookApp/g”
/etc/jupyter/jupyter_server_config.py > /etc/jupyter/jupyter_notebook_config.py &&
fix-permissions /etc/jupyter/

HEALTHCHECK documentation: Dockerfile reference | Docker Documentation

This healtcheck works well for lab, notebook, nbclassic, server and retro jupyter commands

HEALTHCHECK Instructions? · Issue #915 · jupyter/docker-stacks · GitHub

HEALTHCHECK --interval=15s --timeout=3s --start-period=5s --retries=3
CMD wget -O- --no-verbose --tries=1 --no-check-certificate
http${GEN_CERT:+s}://localhost:8888${JUPYTERHUB_SERVICE_PREFIX:-/}api || exit 1

Switch back to jovyan to avoid accidental container runs as root

USER ${NB_UID}

WORKDIR “${HOME}”

END base notebook

START minimal python

https://github.com/jupyter/docker-stacks/tree/master/minimal-notebook

Fix: DL4006 · hadolint/hadolint Wiki · GitHub

Fix: SC3014 · koalaman/shellcheck Wiki · GitHub

SHELL [“/bin/bash”, “-o”, “pipefail”, “-c”]

USER root

Install all OS dependencies for fully functional notebook server

RUN apt-get update --yes &&
apt-get install --yes --no-install-recommends
# Common useful utilities
git
nano-tiny
tzdata
unzip
vim-tiny
# git-over-ssh
openssh-client
# less is needed to run help in R
# see: The `help` command doesn't work in notebooks with the R interpreter · Issue #1588 · jupyter/docker-stacks · GitHub
less
# nbconvert dependencies
# Installation — nbconvert 7.7.2 documentation
texlive-xetex
texlive-fonts-recommended
texlive-plain-generic &&
apt-get clean && rm -rf /var/lib/apt/lists/*

Create alternative for nano → nano-tiny

RUN update-alternatives --install /usr/bin/nano nano /bin/nano-tiny 10

Switch back to jovyan to avoid accidental container runs as root

USER ${NB_UID}

Add R mimetype option to specify how the plot returns from R to the browser

COPY --chown=${NB_UID}:${NB_GID} docker-images/jhub/assets/Rprofile.site /opt/conda/lib/R/etc/

END minimal python

START Vistra code

USER root

COPY --chown=$NB_UID:$NB_GID docker-images/jhub/assets/vistra-poststart.sh /tmp/vistra-poststart.sh
COPY --chown=$NB_UID:$NB_GID docker-images/jhub/assets/saml2aws.profile /tmp/saml2aws.profile
COPY docker-images/jhub/assets/v-image.sh /vistra_image.sh

RUN apt update && apt upgrade -y &&
bash /v_image.sh &&
rm /v_image.sh

USER $NB_UID

@mahendrapaipuri I have tried updating the jlab version to 3.4.6 but still it is the same

I could not follow all your config. Anyways, I see you are installing jupyterlab in the mamba without pinning its version. Obviously, latest jupyterlab will be installed in that environment and probably that is where the problem is coming from?

Could you share JupyterHub and single user server logs?

@mahendrapaipuri Okay so I have pinned the jupyterlab to 3.4.6 but still the issue is same

[I 2023-07-25 05:47:36.040 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/kernels?1690264055954 (mehra, gaurang@::ffff:10.170.35.107) 0.80ms
[D 2023-07-25 05:47:36.756 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:47:36.756 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/terminals?1690264056675 (mehr a, gaurang@::ffff:10.170.35.107) 0.65ms
[D 2023-07-25 05:47:43.261 SingleUserLabApp.jupyterlab_git git:158] Code: 0
Output: ## No commits yet on master?? .bash_history?? .bashrc?? .cache/conda/notices/notices.cache?? .cache/jedi/CP ython-38-33/29eef7fec4dc8afb41b76ae425aa0d6225daff32de9e10abf7a1ce54fa5907f1-01a5770cee9ccb57e0b8c27398a26c0625cbb3d2bd e504b421ec2a2a71eda746.pkl?? .cache/jedi/CPython-38-33/29eef7fec4dc8afb41b76ae425aa0d6225daff32de9e10abf7a1ce54fa5907f1 -0d603725587dbbca0f31f7bf3c1d8478aa0c53254481d6f8f09e2c1135bf9307.pkl?? .cache/jedi/CPython-38-33/29eef7fec4dc8afb41b76 ae425aa0d6225daff32de9e10abf7a…
Error:
[D 2023-07-25 05:47:43.263 SingleUserLabApp.jupyterlab_git git:138] Execute [‘git’, ‘diff’, ‘–numstat’, ‘-z’, ‘–cache d’, ‘4b825dc642cb6eb9a060e54bf8d69288fbee4904’] in /home/jovyan/.
[D 2023-07-25 05:47:43.279 SingleUserLabApp.jupyterlab_git git:158] Code: 0
Output:
Error:
[I 2023-07-25 05:47:43.852 SingleUserLabApp log:189] 200 POST /user/mehra%2C%20gaurang/git/status?1690264032752 (mehra, gaurang@::ffff:10.170.35.107) 31017.89ms
[D 2023-07-25 05:48:31.342 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:47:43.784006Z
[D 2023-07-25 05:49:52.453 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:49:52.463 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )?content=1&1690264192327 (mehra, gaurang@::ffff:10.170.35.107) 10.81ms
[D 2023-07-25 05:49:52.464 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[D 2023-07-25 05:49:52.465 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:49:52.465 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/kernels?1690264192330 (mehra, gaurang@::ffff:10.170.35.107) 1.20ms
[I 2023-07-25 05:49:52.466 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/sessions?1690264192329 (mehra , gaurang@::ffff:10.170.35.107) 1.13ms
[D 2023-07-25 05:49:52.535 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:49:52.536 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/terminals?1690264192331 (mehr a, gaurang@::ffff:10.170.35.107) 0.83ms
[D 2023-07-25 05:49:55.340 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[D 2023-07-25 05:49:55.342 SingleUserLabApp.jupyterlab_git git:138] Execute [‘git’, ‘status’, ‘–porcelain’, ‘-b’, ‘-u’ , ‘-z’] in /home/jovyan/.
[D 2023-07-25 05:49:55.343 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[D 2023-07-25 05:49:58.051 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[D 2023-07-25 05:49:58.052 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/files/Mass_X%2BY%20(Shadow)/Queri es_to_run.csv?_xsrf=2%7C72baaba0%7C8d897fe9f5e773a363b377100b70146f%7C1688738922 (mehra, gaurang@::ffff:10.170.35.107) 2.19ms
[D 2023-07-25 05:50:02.617 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:50:02.618 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/kernels?1690264202541 (mehra, gaurang@::ffff:10.170.35.107) 0.94ms
[D 2023-07-25 05:50:02.619 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:50:02.621 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )?content=1&1690264202540 (mehra, gaurang@::ffff:10.170.35.107) 2.20ms
[D 2023-07-25 05:50:02.633 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:50:02.633 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/sessions?1690264202557 (mehra , gaurang@::ffff:10.170.35.107) 0.70ms
[D 2023-07-25 05:50:02.690 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:50:02.690 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/terminals?1690264202603 (mehr a, gaurang@::ffff:10.170.35.107) 0.51ms
[E 2023-07-25 05:50:15.345 SingleUserLabApp log:181] {
“X-Forwarded-Host”: “jhub.eap.tceh.net”,
“X-Forwarded-Proto”: “http”,
“X-Forwarded-Port”: “80”,
“X-Forwarded-For”: “::ffff:10.170.35.107”,
“Cookie”: “_xsrf=[secret]; jupyterhub-user-mehra%2C%20gaurang=[secret]; logged_out_marketing_header_id=[secret]; jupyterhub-session-id=[secret]”,
“Accept-Language”: “en-US,en;q=0.9,ja-JP;q=0.8,ja;q=0.7,zh-CN;q=0.6,zh;q=0.5”,
“Accept-Encoding”: “gzip, deflate, br”,
“Referer”: “https://jhub.eap.tceh.net/user/mehra%2C%20gaurang/lab/tree/Mass_X%2BY%20(Shadow)/Queries_to_run.csv”,
“Sec-Fetch-Dest”: “empty”,
“Sec-Fetch-Mode”: “cors”,
“Sec-Fetch-Site”: “same-origin”,
“Origin”: “https://jhub.eap.tceh.net”,
“Accept”: “/”,
“Sec-Ch-Ua-Platform”: “"Windows"”,
“User-Agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36”,
“Authorization”: “token [secret]”,
“Sec-Ch-Ua-Mobile”: “?0”,
“X-Xsrftoken”: “2|72baaba0|8d897fe9f5e773a363b377100b70146f|1688738922”,
“Content-Type”: “application/json”,
“Sec-Ch-Ua”: “"Not.A/Brand";v="8", "Chromium";v="114", "Google Chrome";v="114"”,
“Content-Length”: “0”,
“Connection”: “close”,
“Host”: “jhub.eap.tceh.net
}
[E 2023-07-25 05:50:15.345 SingleUserLabApp log:189] 500 POST /user/mehra%2C%20gaurang/git/status?1690264195261 (mehra, gaurang@::ffff:10.170.35.107) 20002.58ms
[D 2023-07-25 05:50:26.327 SingleUserLabApp.jupyterlab_git git:158] Code: 0
Output: ## No commits yet on master?? .bash_history?? .bashrc?? .cache/conda/notices/notices.cache?? .cache/jedi/CP ython-38-33/29eef7fec4dc8afb41b76ae425aa0d6225daff32de9e10abf7a1ce54fa5907f1-01a5770cee9ccb57e0b8c27398a26c0625cbb3d2bd e504b421ec2a2a71eda746.pkl?? .cache/jedi/CPython-38-33/29eef7fec4dc8afb41b76ae425aa0d6225daff32de9e10abf7a1ce54fa5907f1 -0d603725587dbbca0f31f7bf3c1d8478aa0c53254481d6f8f09e2c1135bf9307.pkl?? .cache/jedi/CPython-38-33/29eef7fec4dc8afb41b76 ae425aa0d6225daff32de9e10abf7a…
Error:
[D 2023-07-25 05:50:26.328 SingleUserLabApp.jupyterlab_git git:138] Execute [‘git’, ‘diff’, ‘–numstat’, ‘-z’, ‘–cache d’, ‘4b825dc642cb6eb9a060e54bf8d69288fbee4904’] in /home/jovyan/.
[D 2023-07-25 05:50:26.344 SingleUserLabApp.jupyterlab_git git:158] Code: 0
Output:
Error:
[I 2023-07-25 05:50:26.917 SingleUserLabApp log:189] 200 POST /user/mehra%2C%20gaurang/git/status?1690264195259 (mehra, gaurang@::ffff:10.170.35.107) 31577.28ms
[D 2023-07-25 05:51:36.442 SingleUserLabApp auth:384] HubAuth cache miss: token:7d2ad2f4745645f7b958c50b1d7dc2d5:ca8943 f36b9e3733378951e72258cde4873b7daa04f2b3d9d5d0a77b24b7ec25
[D 2023-07-25 05:51:36.450 SingleUserLabApp auth:395] Received request from Hub user {‘kind’: ‘user’, ‘name’: ‘mehra, g aurang’, ‘admin’: False, ‘groups’: , ‘server’: ‘/user/mehra%2C%20gaurang/’, ‘pending’: None, ‘created’: ‘2023-07-10T1 4:42:39.504545Z’, ‘last_activity’: ‘2023-07-25T05:51:36.448569Z’, ‘servers’: None}
[D 2023-07-25 05:51:36.450 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:36.451 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/kernels?1690264296293 (mehra, gaurang@::ffff:10.170.35.107) 9.40ms
[D 2023-07-25 05:51:36.452 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:36.460 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )?content=1&1690264296298 (mehra, gaurang@::ffff:10.170.35.107) 9.08ms
[D 2023-07-25 05:51:36.471 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:36.472 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/sessions?1690264296305 (mehra , gaurang@::ffff:10.170.35.107) 0.70ms
[D 2023-07-25 05:51:36.477 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:36.477 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/terminals?1690264296307 (mehr a, gaurang@::ffff:10.170.35.107) 0.51ms
[D 2023-07-25 05:51:39.407 SingleUserLabApp kernelmanager:529] activity on 87992975-2172-4b73-8ecb-4e86013ee7c2: status (busy)
[D 2023-07-25 05:51:39.408 SingleUserLabApp kernelmanager:529] activity on 87992975-2172-4b73-8ecb-4e86013ee7c2: status (idle)
[D 2023-07-25 05:51:39.492 SingleUserLabApp kernelmanager:529] activity on 87992975-2172-4b73-8ecb-4e86013ee7c2: status (busy)
[D 2023-07-25 05:51:39.493 SingleUserLabApp kernelmanager:529] activity on 87992975-2172-4b73-8ecb-4e86013ee7c2: status (idle)
[D 2023-07-25 05:51:39.590 SingleUserLabApp kernelmanager:529] activity on d4e3936c-c032-4412-b4a4-303432e7ecbf: status (busy)
[D 2023-07-25 05:51:39.590 SingleUserLabApp kernelmanager:529] activity on d4e3936c-c032-4412-b4a4-303432e7ecbf: status (idle)
[D 2023-07-25 05:51:39.652 SingleUserLabApp kernelmanager:529] activity on d4e3936c-c032-4412-b4a4-303432e7ecbf: status (busy)
[D 2023-07-25 05:51:39.654 SingleUserLabApp kernelmanager:529] activity on d4e3936c-c032-4412-b4a4-303432e7ecbf: status (idle)
[D 2023-07-25 05:51:40.025 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:40.041 SingleUserLabApp log:189] 204 PUT /user/mehra%2C%20gaurang/lab/api/workspaces/default?169026 4299945 (mehra, gaurang@::ffff:10.170.35.107) 16.18ms
[D 2023-07-25 05:51:46.601 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:46.602 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/kernels?1690264306513 (mehra, gaurang@::ffff:10.170.35.107) 0.99ms
[D 2023-07-25 05:51:46.619 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:46.622 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )?content=1&1690264306540 (mehra, gaurang@::ffff:10.170.35.107) 3.29ms
[D 2023-07-25 05:51:46.633 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:46.634 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/sessions?1690264306554 (mehra , gaurang@::ffff:10.170.35.107) 0.78ms
[D 2023-07-25 05:51:46.636 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:46.636 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/terminals?1690264306555 (mehr a, gaurang@::ffff:10.170.35.107) 0.79ms
[D 2023-07-25 05:51:48.079 SingleUserLabApp kernelmanager:529] activity on 87992975-2172-4b73-8ecb-4e86013ee7c2: status (busy)
[D 2023-07-25 05:51:48.079 SingleUserLabApp kernelmanager:536] activity on 87992975-2172-4b73-8ecb-4e86013ee7c2: execut e_input
[D 2023-07-25 05:51:48.080 SingleUserLabApp kernelmanager:529] activity on 87992975-2172-4b73-8ecb-4e86013ee7c2: status (idle)
[D 2023-07-25 05:51:48.332 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:48.335 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )?content=1&1690264308252 (mehra, gaurang@::ffff:10.170.35.107) 3.25ms
[D 2023-07-25 05:51:49.489 SingleUserLabApp kernelmanager:529] activity on 87992975-2172-4b73-8ecb-4e86013ee7c2: status (busy)
[D 2023-07-25 05:51:49.489 SingleUserLabApp kernelmanager:536] activity on 87992975-2172-4b73-8ecb-4e86013ee7c2: execut e_input
[D 2023-07-25 05:51:49.490 SingleUserLabApp kernelmanager:536] activity on 87992975-2172-4b73-8ecb-4e86013ee7c2: execut e_result
[D 2023-07-25 05:51:49.491 SingleUserLabApp kernelmanager:529] activity on 87992975-2172-4b73-8ecb-4e86013ee7c2: status (idle)
[D 2023-07-25 05:51:49.677 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:49.680 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )?content=1&1690264309600 (mehra, gaurang@::ffff:10.170.35.107) 2.85ms
[D 2023-07-25 05:51:53.346 SingleUserLabApp kernelmanager:529] activity on 87992975-2172-4b73-8ecb-4e86013ee7c2: status (busy)
[D 2023-07-25 05:51:53.347 SingleUserLabApp kernelmanager:536] activity on 87992975-2172-4b73-8ecb-4e86013ee7c2: execut e_input
[D 2023-07-25 05:51:53.381 SingleUserLabApp kernelmanager:529] activity on 87992975-2172-4b73-8ecb-4e86013ee7c2: status (idle)
[D 2023-07-25 05:51:53.599 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:53.603 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )?content=1&1690264313483 (mehra, gaurang@::ffff:10.170.35.107) 3.95ms
[D 2023-07-25 05:51:56.300 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[D 2023-07-25 05:51:56.302 SingleUserLabApp.jupyterlab_git git:138] Execute [‘git’, ‘status’, ‘–porcelain’, ‘-b’, ‘-u’ , ‘-z’] in /home/jovyan/.
[D 2023-07-25 05:51:56.303 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[D 2023-07-25 05:51:56.762 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:56.762 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/kernels?1690264316681 (mehra, gaurang@::ffff:10.170.35.107) 0.77ms
[D 2023-07-25 05:51:56.779 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:56.779 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/sessions?1690264316697 (mehra , gaurang@::ffff:10.170.35.107) 0.70ms
[D 2023-07-25 05:51:56.792 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:51:56.793 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/terminals?1690264316714 (mehr a, gaurang@::ffff:10.170.35.107) 0.54ms
[D 2023-07-25 05:52:03.384 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[D 2023-07-25 05:52:03.389 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/files/Mass_X%2BY%20(Shadow)/Queri es_to_run.csv?_xsrf=2%7C72baaba0%7C8d897fe9f5e773a363b377100b70146f%7C1688738922 (mehra, gaurang@::ffff:10.170.35.107) 5.28ms
[D 2023-07-25 05:52:03.745 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:52:03.747 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )?content=1&1690264323669 (mehra, gaurang@::ffff:10.170.35.107) 2.71ms
[D 2023-07-25 05:52:07.148 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:52:07.149 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/kernels?1690264327058 (mehra, gaurang@::ffff:10.170.35.107) 0.91ms
[D 2023-07-25 05:52:07.158 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:52:07.158 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/sessions?1690264327059 (mehra , gaurang@::ffff:10.170.35.107) 1.04ms
[D 2023-07-25 05:52:07.159 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:52:07.159 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/terminals?1690264327060 (mehr a, gaurang@::ffff:10.170.35.107) 0.53ms
[E 2023-07-25 05:52:16.307 SingleUserLabApp log:181] {
“X-Forwarded-Host”: “jhub.eap.tceh.net”,
“X-Forwarded-Proto”: “http”,
“X-Forwarded-Port”: “80”,
“X-Forwarded-For”: “::ffff:10.170.35.107”,
“Cookie”: “_xsrf=[secret]; jupyterhub-user-mehra%2C%20gaurang=[secret]; logged_out_marketing_header_id=[secret]; jupyterhub-session-id=[secret]”,
“Accept-Language”: “en-US,en;q=0.9,ja-JP;q=0.8,ja;q=0.7,zh-CN;q=0.6,zh;q=0.5”,
“Accept-Encoding”: “gzip, deflate, br”,
“Referer”: “https://jhub.eap.tceh.net/user/mehra%2C%20gaurang/lab/tree/Mass_X%2BY%20(Shadow)/Untitled.ipynb”,
“Sec-Fetch-Dest”: “empty”,
“Sec-Fetch-Mode”: “cors”,
“Sec-Fetch-Site”: “same-origin”,
“Origin”: “https://jhub.eap.tceh.net”,
“Accept”: “/”,
“Sec-Ch-Ua-Platform”: “"Windows"”,
“User-Agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36”,
“Authorization”: “token [secret]”,
“Sec-Ch-Ua-Mobile”: “?0”,
“X-Xsrftoken”: “2|72baaba0|8d897fe9f5e773a363b377100b70146f|1688738922”,
“Content-Type”: “application/json”,
“Sec-Ch-Ua”: “"Not.A/Brand";v="8", "Chromium";v="114", "Google Chrome";v="114"”,
“Content-Length”: “0”,
“Connection”: “close”,
“Host”: “jhub.eap.tceh.net
}
[E 2023-07-25 05:52:16.307 SingleUserLabApp log:189] 500 POST /user/mehra%2C%20gaurang/git/status?1690264316217 (mehra, gaurang@::ffff:10.170.35.107) 20003.98ms
[D 2023-07-25 05:52:17.667 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:52:17.668 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/kernels?1690264337587 (mehra, gaurang@::ffff:10.170.35.107) 0.94ms
[D 2023-07-25 05:52:17.707 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:52:17.708 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/sessions?1690264337621 (mehra , gaurang@::ffff:10.170.35.107) 0.79ms
[D 2023-07-25 05:52:17.717 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:52:17.717 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/terminals?1690264337635 (mehr a, gaurang@::ffff:10.170.35.107) 0.52ms
[D 2023-07-25 05:52:26.995 SingleUserLabApp.jupyterlab_git git:158] Code: 0
Output: ## No commits yet on master?? .bash_history?? .bashrc?? .cache/conda/notices/notices.cache?? .cache/jedi/CP ython-38-33/29eef7fec4dc8afb41b76ae425aa0d6225daff32de9e10abf7a1ce54fa5907f1-01a5770cee9ccb57e0b8c27398a26c0625cbb3d2bd e504b421ec2a2a71eda746.pkl?? .cache/jedi/CPython-38-33/29eef7fec4dc8afb41b76ae425aa0d6225daff32de9e10abf7a1ce54fa5907f1 -0d603725587dbbca0f31f7bf3c1d8478aa0c53254481d6f8f09e2c1135bf9307.pkl?? .cache/jedi/CPython-38-33/29eef7fec4dc8afb41b76 ae425aa0d6225daff32de9e10abf7a…
Error:
[D 2023-07-25 05:52:26.997 SingleUserLabApp.jupyterlab_git git:138] Execute [‘git’, ‘diff’, ‘–numstat’, ‘-z’, ‘–cache d’, ‘4b825dc642cb6eb9a060e54bf8d69288fbee4904’] in /home/jovyan/.
[D 2023-07-25 05:52:27.014 SingleUserLabApp.jupyterlab_git git:158] Code: 0
Output:
Error:
[I 2023-07-25 05:52:27.586 SingleUserLabApp log:189] 200 POST /user/mehra%2C%20gaurang/git/status?1690264316216 (mehra, gaurang@::ffff:10.170.35.107) 31285.76ms
[D 2023-07-25 05:53:15.868 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:53:15.871 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )/Untitled.ipynb?content=0&1690264395788 (mehra, gaurang@::ffff:10.170.35.107) 3.40ms
[D 2023-07-25 05:53:16.016 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:53:16.017 SingleUserLabApp handlers:184] Saving file at /Mass_X+Y (Shadow)/Untitled.ipynb
[D 2023-07-25 05:53:16.017 SingleUserLabApp filemanager:428] Saving /home/jovyan/Mass_X+Y (Shadow)/Untitled.ipynb
[I 2023-07-25 05:53:16.175 SingleUserLabApp log:189] 200 PUT /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )/Untitled.ipynb?1690264395935 (mehra, gaurang@::ffff:10.170.35.107) 158.69ms
[D 2023-07-25 05:53:16.351 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[D 2023-07-25 05:53:16.351 SingleUserLabApp.jupyterlab_git git:138] Execute [‘git’, ‘status’, ‘–porcelain’, ‘-b’, ‘-u’ , ‘-z’] in /home/jovyan/.
[D 2023-07-25 05:53:16.352 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 05:53:16.354 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )/Untitled.ipynb/checkpoints?1690264396278 (mehra, gaurang@::ffff:10.170.35.107) 1.78ms
[D 2023-07-25 05:53:26.763 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:16.353739Z
[D 2023-07-25 05:53:47.291 SingleUserLabApp.jupyterlab_git git:158] Code: 0
Output: ## No commits yet on master?? .bash_history?? .bashrc?? .cache/conda/notices/notices.cache?? .cache/jedi/CP ython-38-33/29eef7fec4dc8afb41b76ae425aa0d6225daff32de9e10abf7a1ce54fa5907f1-01a5770cee9ccb57e0b8c27398a26c0625cbb3d2bd e504b421ec2a2a71eda746.pkl?? .cache/jedi/CPython-38-33/29eef7fec4dc8afb41b76ae425aa0d6225daff32de9e10abf7a1ce54fa5907f1 -0d603725587dbbca0f31f7bf3c1d8478aa0c53254481d6f8f09e2c1135bf9307.pkl?? .cache/jedi/CPython-38-33/29eef7fec4dc8afb41b76 ae425aa0d6225daff32de9e10abf7a…
Error:
[D 2023-07-25 05:53:47.293 SingleUserLabApp.jupyterlab_git git:138] Execute [‘git’, ‘diff’, ‘–numstat’, ‘-z’, ‘–cache d’, ‘4b825dc642cb6eb9a060e54bf8d69288fbee4904’] in /home/jovyan/.
[D 2023-07-25 05:53:47.310 SingleUserLabApp.jupyterlab_git git:158] Code: 0
Output:
Error:
[I 2023-07-25 05:53:47.884 SingleUserLabApp log:189] 200 POST /user/mehra%2C%20gaurang/git/status?1690264396272 (mehra, gaurang@::ffff:10.170.35.107) 31533.21ms
[D 2023-07-25 05:58:33.915 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 06:03:04.045 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 06:07:38.714 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 06:12:13.636 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 06:17:12.233 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 06:21:57.623 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 06:27:08.208 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 06:31:41.773 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 06:37:01.742 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 06:42:17.665 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 06:46:48.935 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 06:52:05.939 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 06:57:28.285 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 07:02:01.715 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 07:06:58.387 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 07:12:27.055 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 07:17:32.144 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T05:53:47.815803Z
[D 2023-07-25 07:22:25.611 SingleUserLabApp auth:384] HubAuth cache miss: token:7d2ad2f4745645f7b958c50b1d7dc2d5:ca8943 f36b9e3733378951e72258cde4873b7daa04f2b3d9d5d0a77b24b7ec25
[D 2023-07-25 07:22:25.620 SingleUserLabApp auth:395] Received request from Hub user {‘kind’: ‘user’, ‘name’: ‘mehra, g aurang’, ‘admin’: False, ‘groups’: , ‘server’: ‘/user/mehra%2C%20gaurang/’, ‘pending’: None, ‘created’: ‘2023-07-10T1 4:42:39.504545Z’, ‘last_activity’: ‘2023-07-25T07:22:25.617868Z’, ‘servers’: None}
[D 2023-07-25 07:22:25.620 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:25.636 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )?content=1&1690269745422 (mehra, gaurang@::ffff:10.170.36.191) 25.47ms
[D 2023-07-25 07:22:25.637 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:25.637 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/kernels?1690269745500 (mehra, gaurang@::ffff:10.170.37.202) 0.71ms
[D 2023-07-25 07:22:25.643 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:25.644 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/sessions?1690269745502 (mehra , gaurang@::ffff:10.170.36.181) 0.70ms
[D 2023-07-25 07:22:25.653 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:25.653 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/terminals?1690269745503 (mehr a, gaurang@::ffff:10.170.36.191) 0.88ms
[D 2023-07-25 07:22:34.819 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[D 2023-07-25 07:22:34.832 SingleUserLabApp kernelspec:220] Found kernel gaurang_fnl in /home/jovyan/.local/share/jupyt er/kernels
[D 2023-07-25 07:22:34.832 SingleUserLabApp kernelspec:220] Found kernel gaurang in /home/jovyan/.local/share/jupyter/k ernels
[D 2023-07-25 07:22:34.832 SingleUserLabApp kernelspec:220] Found kernel python3 in /opt/conda/share/jupyter/kernels
[I 2023-07-25 07:22:34.839 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/kernelspecs?1690269754734 (me hra, gaurang@::ffff:10.170.37.202) 20.54ms
[D 2023-07-25 07:22:35.885 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:35.885 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/kernels?1690269755804 (mehra, gaurang@::ffff:10.170.36.191) 0.88ms
[D 2023-07-25 07:22:35.906 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:35.909 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )?content=1&1690269755803 (mehra, gaurang@::ffff:10.170.36.181) 3.37ms
[D 2023-07-25 07:22:35.956 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:35.957 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/sessions?1690269755805 (mehra , gaurang@::ffff:10.170.37.202) 0.72ms
[D 2023-07-25 07:22:35.957 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:35.958 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/terminals?1690269755805 (mehr a, gaurang@::ffff:10.170.36.181) 0.55ms
[D 2023-07-25 07:22:39.950 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[D 2023-07-25 07:22:39.950 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[D 2023-07-25 07:22:39.952 SingleUserLabApp.jupyterlab_git git:138] Execute [‘git’, ‘status’, ‘–porcelain’, ‘-b’, ‘-u’ , ‘-z’] in /home/jovyan/.
[D 2023-07-25 07:22:46.198 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:46.198 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/terminals?1690269766087 (mehr a, gaurang@::ffff:10.170.37.202) 0.86ms
[D 2023-07-25 07:22:46.199 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:46.202 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )?content=1&1690269766081 (mehra, gaurang@::ffff:10.170.36.191) 3.15ms
[D 2023-07-25 07:22:46.222 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:46.223 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/kernels?1690269766079 (mehra, gaurang@::ffff:10.170.36.181) 0.67ms
[D 2023-07-25 07:22:46.223 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:46.224 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/sessions?1690269766083 (mehra , gaurang@::ffff:10.170.36.181) 0.73ms
[D 2023-07-25 07:22:51.616 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[D 2023-07-25 07:22:51.626 SingleUserLabApp filecheckpoints:72] Renaming checkpoint /home/jovyan/Mass_X+Y (Shadow)/.ipy nb_checkpoints/Untitled-checkpoint.ipynb → /home/jovyan/Mass_X+Y (Shadow)/.ipynb_checkpoints/Script_query-checkpoint.i pynb
[I 2023-07-25 07:22:51.635 SingleUserLabApp log:189] 200 PATCH /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shad ow)/Untitled.ipynb?1690269771534 (mehra, gaurang@::ffff:10.170.36.191) 19.32ms
[D 2023-07-25 07:22:51.805 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:51.806 SingleUserLabApp log:189] 200 PATCH /user/mehra%2C%20gaurang/api/sessions/0c96a0ef-89c4-4e2d -afe5-925a610da4e7?1690269771696 (mehra, gaurang@::ffff:10.170.36.181) 0.82ms
[D 2023-07-25 07:22:51.806 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:51.807 SingleUserLabApp log:189] 200 PATCH /user/mehra%2C%20gaurang/api/sessions/0c96a0ef-89c4-4e2d -afe5-925a610da4e7?1690269771694 (mehra, gaurang@::ffff:10.170.37.202) 0.72ms
[D 2023-07-25 07:22:51.823 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:51.824 SingleUserLabApp log:189] 200 PATCH /user/mehra%2C%20gaurang/api/sessions/0c96a0ef-89c4-4e2d -afe5-925a610da4e7?1690269771706 (mehra, gaurang@::ffff:10.170.36.191) 0.70ms
[D 2023-07-25 07:22:51.840 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:51.843 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )/Script_query.ipynb/checkpoints?1690269771709 (mehra, gaurang@::ffff:10.170.37.202) 3.70ms
[D 2023-07-25 07:22:51.944 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[D 2023-07-25 07:22:51.969 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:51.974 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/contents/Mass_X%2BY%20(Shadow )?content=1&1690269771716 (mehra, gaurang@::ffff:10.170.36.191) 5.18ms
[D 2023-07-25 07:22:51.979 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:51.980 SingleUserLabApp log:189] 200 GET /user/mehra%2C%20gaurang/api/sessions?1690269771893 (mehra , gaurang@::ffff:10.170.37.202) 0.65ms
[D 2023-07-25 07:22:52.312 SingleUserLabApp auth:1023] Allowing Hub user mehra, gaurang
[I 2023-07-25 07:22:52.336 SingleUserLabApp log:189] 204 PUT /user/mehra%2C%20gaurang/lab/api/workspaces/default?169026 9772220 (mehra, gaurang@::ffff:10.170.36.181) 24.55ms
[D 2023-07-25 07:22:53.669 SingleUserLabApp mixins:559] Notifying Hub of activity 2023-07-25T07:22:52.335920Z
[D 2023-07-25 07:22:54.379 SingleUserLabApp handlers:706] Websocket closed 87992975-2172-4b73-8ecb-4e86013ee7c2:c7ba995 3-bf95-4af4-95a5-bca88b8bc0e4

Also who do you sync the versions of jupyterhub, jupyterlab servers and other packages? I am using 1.2.0 helmchart of jhub

Selected Jupyter core packages…
IPython : 8.5.0
ipykernel : 6.15.2
ipywidgets : not installed
jupyter_client : 7.3.5
jupyter_core : 4.11.1
jupyter_server : 1.18.1
jupyterlab : 3.4.6
nbclient : 0.6.8
nbconvert : 7.0.0
nbformat : 5.4.0
notebook : 6.4.12
qtconsole : not installed
traitlets : 5.3.

I dont have experience with JupyterHub deployment with Kubernetes, so I dont know how to keep up with version sync between different components.

From your logs, I dont see any “apparent” errors. Seems like you have issues with web socket connections. Here are some things to look/test:

  • Test in a different browser and see if you can reproduce the issue
  • Ensure that you are not behind any firewalls that are blocking web socket connections. Seems like your JupyterLab server can communicate normally with Hub using usual API calls. The problem lies only with websockets
  • Try upgrading jupyter_server version to latest in the single user environment. Your jupyter_server version is bit old.
  • If you can, upgrade to JupyterHub 3.x as well.

sorry for the revive of the thread, but @gpunjabi did you get a fix for this?
I’m in the same situation now…

for those experiencing this issue and using traefik as the cluster ingress, there was a refresh issue in version v2.11.1. Updating to the latest traefik version solved the issue for me.