Failed to connect jupyter enterprise gateway from jupyterhub

JupyterHub
, ,
1

failed to connect jupyter enterprise gateway 3.2.1 from jupyterhub 3.2.1 on k8s EKS cluster with below error

[E 2024-01-10 05:53:04.198 ServerApp] Bad message (TypeError('not all arguments converted during string formatting')): {'name': 'ServerApp', 'msg': 'Exception while trying to launch kernel via Gateway URL http://enterprise-gateway.enterprise-gateway.svc.cluster.local:8888 , [Errno 110] Connection timed out', 'args': (TimeoutError(110, 'Connection timed out'),), 'levelname': 'ERROR', 'levelno': 40, 'pathname': '/usr/local/lib/python3.11/site-packages/jupyter_server/gateway/gateway_client.py', 'filename': 'gateway_client.py', 'module': 'gateway_client', 'exc_info': None, 'exc_text': None, 'stack_info': None, 'lineno': 812, 'funcName': 'gateway_request', 'created': 1704865984.1986964, 'msecs': 198.0, 'relativeCreated': 451740.5037879944, 'thread': 140521199601472, 'threadName': 'MainThread', 'processName': 'MainProcess', 'process': 7}
2024-01-10T05:53:04.199786278Z [E 2024-01-10 05:53:04.198 SingleUserLabApp web:1871] Uncaught exception GET /user/jovyan/api/kernels?1704865797316 (127.0.0.1)
2024-01-10T05:53:04.199806236Z     HTTPServerRequest(protocol='https', host='jupyterhub.xyz.com', method='GET', uri='/user/jovyan/api/kernels?1704865797316', version='HTTP/1.1', remote_ip='127.0.0.1')
2

The immediate issue appears to be a bug in the gateway’s log formatting (fixed here), but that’s not the real problem, it’s a problem in how the real problem is logged.

The timeout error connecting to the gateway client could be due to a network policy. You may need to explicitly grant single-user server pods access to the gateway service. Specifically, singleuser pods do not have access to other kubernetes resources by default. The relevant configuration is in singleuser.networkPolicy.

3

Hi @minrk as per your comment i have been tested with below egress rule for singleuser pods but getting same timeout error.

egress:
- to:
- ipBlock:
cidr: 10.1.0.0/16
ports:
- protocol: TCP
port: 5978

egress:
- to:
- namespaceSelector:
matchLabels:
name: enterprise-gateway
- podSelector:
matchLabels:
app: enterprise-gateway
ports:
- protocol: TCP
port: 8888

4

@minrk you are right i did networkPolicy false and timeout error gone away.

networkPolicy:
enabled: false

Now error which i am getting is unauthorized 401, i have passed below env vars in jupyterhub but still getting unathorised 401 below error

tornado.web.HTTPError: HTTP 401: Unauthorized (Error from Gateway: [Unauthorized] Unauthorized. Ensure gateway url is valid and the Gateway instance is running.)
2024-01-11T11:49:53.223395636Z [W 2024-01-11 11:49:53.223 ServerApp] 401 GET /user/jovyan/api/kernels?1704973793047 (jovyan@127.0.0.1) 11.65ms

i have passed below env var
extraEnv:
EDITOR: “vim”
JUPYTER_GATEWAY_URL: “http://enterprise-gateway.enterprise-gateway.svc.cluster.local:8888”
LOG_LEVEL: “DEBUG”
JUPYTER_GATEWAY_CONNECT_TIMEOUT: “1200”
JUPYTER_GATEWAY_REQUEST_TIMEOUT: “1200”
JUPYTER_GATEWAY_HTTP_USER: “jovyan”
KERNEL_USERNAME: “jovyan”

do i need to set password? but where @JEG ? or Jupyterhub login pass do i need to specify ? but what is the env var for that?

5

This is going to be up to how your gateway is configured, which isn’t part of JupyterHub.

From the gateway docs, you can set either $JUPYTER_GATEWAY_AUTH_TOKEN to auth with a token or $JUPYTER_GATEWAY_HTTP_PWD to auth with a password. What values to use will depend on what you’ve provided to your enterprise gateway.

6

@minrk i am able to connect jupyterhub to JEG

ss2
ss23337×1877 334 KB

after selecting notebook from given images new pod has been started but at the jupyterhub its showing kernel unknown status.

@ New JEG kernel pods there is no any error
@ Jupyterhub Jovyan pod also dont see any error

how to go forward ? i am unable to execute anything on notebook.

ss3
ss33012×886 53.4 KB

found some logs in JEG

[D 2024-01-15 12:43:58.002 EnterpriseGatewayApp] Connecting to: tcp://10.1.22.46:50171

Mon, Jan 15 2024 4:43:58 pm[D 2024-01-15 12:43:58.003 EnterpriseGatewayApp] Nudge: attempt 1 on kernel 0f029e04-7079-4e7d-bd85-0eeb73ecd494

Mon, Jan 15 2024 4:43:58 pm[D 2024-01-15 12:43:58.051 EnterpriseGatewayApp] Nudge: IOPub received: 0f029e04-7079-4e7d-bd85-0eeb73ecd494

Mon, Jan 15 2024 4:43:58 pm[D 2024-01-15 12:43:58.051 EnterpriseGatewayApp] Nudge: resolving iopub future: 0f029e04-7079-4e7d-bd85-0eeb73ecd494

Mon, Jan 15 2024 4:43:58 pm[D 2024-01-15 12:43:58.052 EnterpriseGatewayApp] activity on 0f029e04-7079-4e7d-bd85-0eeb73ecd494: status (busy)

Mon, Jan 15 2024 4:43:58 pm[D 2024-01-15 12:43:58.052 EnterpriseGatewayApp] Nudge: shell info reply received: 0f029e04-7079-4e7d-bd85-0eeb73ecd494 2024-01-15T12:43:58.053052576Z [D 2024-01-15 12:43:58.052 EnterpriseGatewayApp] Nudge: resolving shell future: 0f029e04-7079-4e7d-bd85-0eeb73ecd494

Mon, Jan 15 2024 4:43:58 pm[D 2024-01-15 12:43:58.053 EnterpriseGatewayApp] activity on 0f029e04-7079-4e7d-bd85-0eeb73ecd494: status (idle)

Mon, Jan 15 2024 4:43:58 pm[E 240115 12:43:58 web:1798] Uncaught exception GET /api/kernels/0f029e04-7079-4e7d-bd85-0eeb73ecd494/channels (10.1.41.111)

Mon, Jan 15 2024 4:43:58 pm HTTPServerRequest(protocol='http', host='enterprise-gateway.enterprise-gateway.svc.cluster.local:8888', method='GET', uri='/api/kernels/0f029e04-7079-4e7d-bd85-0eeb73ecd494/channels', version='HTTP/1.1', remote_ip='10.1.41.111')

Mon, Jan 15 2024 4:43:58 pm Traceback (most recent call last): 2024-01-15T12:43:58.174586767Z File "/opt/conda/lib/python3.10/site-packages/tornado/websocket.py", line 635, in _run_callback

Mon, Jan 15 2024 4:43:58 pm result = callback(*args, **kwargs) 2024-01-15T12:43:58.174595470Z File "/opt/conda/lib/python3.10/site-packages/jupyter_server/services/kernels/handlers.py", line 521, in on_message 2024-01-15T12:43:58.174600787Z msg = json.loads(ws_msg) 2024-01-15T12:43:58.174604913Z File "/opt/conda/lib/python3.10/json/__init__.py", line 346, in loads 2024-01-15T12:43:58.174608541Z return _default_decoder.decode(s) 2024-01-15T12:43:58.174612132Z File "/opt/conda/lib/python3.10/json/decoder.py", line 337, in decode

Mon, Jan 15 2024 4:43:58 pm obj, end = self.raw_decode(s, idx=_w(s, 0).end())

Mon, Jan 15 2024 4:43:58 pm File "/opt/conda/lib/python3.10/json/decoder.py", line 355, in raw_decode

Mon, Jan 15 2024 4:43:58 pm raise JSONDecodeError("Expecting value", s, err.value) from None 2024-01-15T12:43:58.174627523Z json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 2024-01-15T12:43:58.174641600Z [D 2024-01-15 12:43:58.174 EnterpriseGatewayApp] Websocket closed 0f029e04-7079-4e7d-bd85-0eeb73ecd494:d29115a3-ba9e4b0988ab3fda09d2765d 2024-01-15T12:43:58.174739406Z [I 2024-01-15 12:43:58.174 EnterpriseGatewayApp] Starting buffering for 0f029e04-7079-4e7d-bd85-0eeb73ecd494:d29115a3-ba9e4b0988ab3fda09d2765d 2024-01-15T12:43:58.174816020Z [D 2024-01-15 12:43:58.174 EnterpriseGatewayApp] Clearing buffer for 0f029e04-7079-4e7d-bd85-0eeb73ecd494
7

I’m nmot sure, this is outside my experience for enterprise gateway. It looks like an unexpected non-JSON message on the websocket. You might double check that the jupyter-server version in both the single-user server pod and the enterprise gateway pods is the same.

8

JEG version ([3.2.1 helm chart] (https://github.com/jupyter-server/enterprise_gateway/releases/download/v3.2.1/jupyter_enterprise_gateway_helm-3.2.1.tar.gz)

IPython : 8.0.1
ipykernel : 6.7.0
ipywidgets : not installed
jupyter_client : 6.1.12
jupyter_core : 4.9.1
jupyter_server : 1.13.4
jupyterlab : 3.2.8
nbclient : 0.5.10
nbconvert : 6.4.0
nbformat : 5.1.3
notebook : 6.4.7
qtconsole : not installed
traitlets : 5.9.0

JupyterHub 4.0.2 (3.2.1 all docker images)
IPython : 8.18.1
ipykernel : 6.26.0
ipywidgets : not installed
jupyter_client : 8.6.0
jupyter_core : 5.5.0
jupyter_server : 2.11.2
jupyterlab : 4.0.9
nbclient : 0.9.0
nbconvert : 7.11.0
nbformat : 5.9.2
notebook : not installed
qtconsole : not installed
traitlets : 5.13.0

9

that looks like it - jupyter-server major versions are different. Try using jupyter-server 2 in both places.

10

@minrk

i did below upgrade on JEG docker image but no luck, getting below new error

  • jupyter_client>=8.6
  • jupyter_core>=5.5.0
  • jupyter_server>=2.10.1
  • paramiko>=2.11
  • pexpect>=4.8.0
  • pyzmq>=25.1.1

2024-01-22T12:56:05.493852908Z [I 2024-01-22 12:56:05.493 EnterpriseGatewayApp] Jupyter Enterprise Gateway 3.2.2 is available at http://0.0.0.0:8888
2024-01-22T12:56:06.754804256Z /opt/conda/lib/python3.10/site-packages/jupyter_server/base/handlers.py:607: UserWarning: The Tornado web application does not have an ‘identity_provider’ defined in its settings. In future releases of jupyter_server, this will be a required key for all subclasses of JupyterHandler. For an example, see the jupyter_server source code for how to add an identity provider to the tornado settings: jupyter_server/jupyter_server/serverapp.py at 2a88e2a2c033578b7d72b440c296acbbe80fb398 · jupyter-server/jupyter_server · GitHub
2024-01-22T12:56:06.754829420Z if type(self.identity_provider) is IdentityProvider and mod_obj.name != name:
2024-01-22T12:56:06.756014026Z [E 240122 12:56:06 web:1798] Uncaught exception GET /api/kernels/693f9c8a-02b7-46d7-899c-c7b2166283c0/channels (10.1.32.47)
2024-01-22T12:56:06.756019838Z HTTPServerRequest(protocol=‘http’, host=‘enterprise-gateway.enterprise-gateway.svc.cluster.local:8888’, method=‘GET’, uri=‘/api/kernels/693f9c8a-02b7-46d7-899c-c7b2166283c0/channels’, version=‘HTTP/1.1’, remote_ip=‘10.1.32.47’)
2024-01-22T12:56:06.756022596Z Traceback (most recent call last):
2024-01-22T12:56:06.756024937Z File “/opt/conda/lib/python3.10/site-packages/tornado/web.py”, line 1692, in _execute
2024-01-22T12:56:06.756027526Z result = await result
2024-01-22T12:56:06.756029859Z File “/
opt/conda/lib/python3.10/site-packages/jupyter_server/base/handlers.py”, line 622, in prepare
2024-01-22T12:56:06.756032074Z _user = await _user
2024-01-22T12:56:06.756034346Z File “/opt/conda/lib/python3.10/site-packages/jupyter_server/auth/identity.py”, line 235, in _get_user
2024-01-22T12:56:06.756050446Z _cookie_user = self.get_user_cookie(handler)
2024-01-22T12:56:06.756052824Z File “/opt/conda/lib/python3.10/site-packages/jupyter_server/auth/identity.py”, line 395, in get_user_cookie
_user_cookie = handler.get_secure_cookie(
2024-01-22T12:56:06.756056838Z File “/opt/conda/lib/python3.10/site-packages/tornado/web.py”, line 773, in get_secure_cookie
2024-01-22T12:56:06.756058950Z self.require_setting(“cookie_secret”, “secure cookies”)
2024-01-22T12:56:06.756060956Z File “/opt/conda/lib/python3.10/site-packages/tornado/web.py”, line 1592, in require_setting
2024-01-22T12:56:06.756063393Z raise Exception(
2024-01-22T12:56:06.756065521Z Exception: You must define the ‘cookie_secret’ setting in your application to use secure cookies
2024-01-22T12:56:06.756501970Z [E 240122 12:56:06 web:2271] 500 GET /api/kernels/693f9c8a-02b7-46d7-899c-c7b2166283c0/channels (10.1.32.47) 2.42ms
2024-01-22T12:56:07.003324034Z [E 240122 12:56:07 web:1798] Uncaught exception GET /api/kernels/693f9c8a-02b7-46d7-899c-c7b2166283c0/channels (10.1.32.47)
HTTPServerRequest(protocol=‘http’, host=‘enterprise-gateway.enterprise-gateway.svc.cluster.local:8888’, method=‘GET’, uri=‘/api/kernels/693f9c8a-02b7-46d7-899c-c7b2166283c0/channels’, version=‘HTTP/1.1’, remote_ip=‘10.1.32.47’)
2024-01-22T12:56:07.003352492Z Traceback (most recent call last):

11

i changed jupyter_server : 1.13.4 at jupyterhub and issue has been resolved, thanks for your support @minrk