# Failed to connect jupyter enterprise gateway from jupyterhub

**URL:** <https://discourse.jupyter.org/t/failed-to-connect-jupyter-enterprise-gateway-from-jupyterhub/23319>\
**Category:** JupyterHub\
**Tags:** jupyterlab, jupyterhub, notebook\
**Created:** [January 9, 2024, 9:45am UTC](https://discourse.jupyter.org/t/failed-to-connect-jupyter-enterprise-gateway-from-jupyterhub/23319 "2024-01-09T09:45:21Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aniket\_Gole](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/aniket_gole/32/11918_2.png) [@Aniket\_Gole](https://discourse.jupyter.org/u/Aniket_Gole)\
**Post date:** [January 9, 2024, 9:45am UTC](https://discourse.jupyter.org/t/failed-to-connect-jupyter-enterprise-gateway-from-jupyterhub/23319/1 "2024-01-09T09:45:21Z")

</div>

failed to connect jupyter enterprise gateway 3.2.1 from jupyterhub 3.2.1 on k8s EKS cluster with below error

```txt
[E 2024-01-10 05:53:04.198 ServerApp] Bad message (TypeError('not all arguments converted during string formatting')): {'name': 'ServerApp', 'msg': 'Exception while trying to launch kernel via Gateway URL http://enterprise-gateway.enterprise-gateway.svc.cluster.local:8888 , [Errno 110] Connection timed out', 'args': (TimeoutError(110, 'Connection timed out'),), 'levelname': 'ERROR', 'levelno': 40, 'pathname': '/usr/local/lib/python3.11/site-packages/jupyter_server/gateway/gateway_client.py', 'filename': 'gateway_client.py', 'module': 'gateway_client', 'exc_info': None, 'exc_text': None, 'stack_info': None, 'lineno': 812, 'funcName': 'gateway_request', 'created': 1704865984.1986964, 'msecs': 198.0, 'relativeCreated': 451740.5037879944, 'thread': 140521199601472, 'threadName': 'MainThread', 'processName': 'MainProcess', 'process': 7}
2024-01-10T05:53:04.199786278Z [E 2024-01-10 05:53:04.198 SingleUserLabApp web:1871] Uncaught exception GET /user/jovyan/api/kernels?1704865797316 (127.0.0.1)
2024-01-10T05:53:04.199806236Z HTTPServerRequest(protocol='https', host='jupyterhub.xyz.com', method='GET', uri='/user/jovyan/api/kernels?1704865797316', version='HTTP/1.1', remote_ip='127.0.0.1')

```

---

<div class="post-metadata">

**Author:** ![minrk](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/minrk/32/13_2.png) [@minrk](https://discourse.jupyter.org/u/minrk)\
**Post date:** [January 11, 2024, 8:42am UTC](https://discourse.jupyter.org/t/failed-to-connect-jupyter-enterprise-gateway-from-jupyterhub/23319/2 "2024-01-11T08:42:00Z")

</div>

The immediate issue appears to be a bug in the gateway’s log formatting ([fixed here](https://github.com/jupyter-server/jupyter_server/pull/1385)), but that’s not the real problem, it’s a problem in how the _real_ problem is logged.

The timeout error connecting to the gateway client could be due to a network policy. You may need to explicitly grant single-user server pods access to the gateway service. Specifically, singleuser pods do not have access to other kubernetes resources by default. The relevant configuration is in [`singleuser.networkPolicy`](https://z2jh.jupyter.org/en/stable/resources/reference.html#singleuser-networkpolicy-egress).

---

<div class="post-metadata">

**Author:** ![Aniket\_Gole](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/aniket_gole/32/11918_2.png) [@Aniket\_Gole](https://discourse.jupyter.org/u/Aniket_Gole)\
**Post date:** [January 11, 2024, 10:54am UTC](https://discourse.jupyter.org/t/failed-to-connect-jupyter-enterprise-gateway-from-jupyterhub/23319/3 "2024-01-11T10:54:43Z")

</div>

Hi @minrk as per your comment i have been tested with below egress rule for singleuser pods but getting same timeout error.

1. 
* * *

egress:  
- to:  
- ipBlock:  
cidr: 10.1.0.0/16  
ports:  
- protocol: TCP  
port: 5978

1. 
* * *

egress:  
- to:  
- namespaceSelector:  
matchLabels:  
name: enterprise-gateway  
- podSelector:  
matchLabels:  
app: enterprise-gateway  
ports:  
- protocol: TCP  
port: 8888

---

<div class="post-metadata">

**Author:** ![Aniket\_Gole](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/aniket_gole/32/11918_2.png) [@Aniket\_Gole](https://discourse.jupyter.org/u/Aniket_Gole)\
**Post date:** [January 11, 2024, 12:09pm UTC](https://discourse.jupyter.org/t/failed-to-connect-jupyter-enterprise-gateway-from-jupyterhub/23319/4 "2024-01-11T12:09:21Z")

</div>

@minrk you are right i did networkPolicy false and timeout error gone away.

networkPolicy:  
enabled: false

Now error which i am getting is unauthorized 401, i have passed below env vars in jupyterhub but still getting unathorised 401 below error

tornado.web.HTTPError: HTTP 401: Unauthorized (Error from Gateway: [Unauthorized] Unauthorized. Ensure gateway url is valid and the Gateway instance is running.)  
2024-01-11T11:49:53.223395636Z [W 2024-01-11 11:49:53.223 ServerApp] 401 GET /user/jovyan/api/kernels?1704973793047 ([jovyan@127.0.0.1](mailto:jovyan@127.0.0.1)) 11.65ms

i have passed below env var  
extraEnv:  
EDITOR: “vim”  
JUPYTER\_GATEWAY\_URL: “[http://enterprise-gateway.enterprise-gateway.svc.cluster.local:8888](http://enterprise-gateway.enterprise-gateway.svc.cluster.local:8888)”  
LOG\_LEVEL: “DEBUG”  
JUPYTER\_GATEWAY\_CONNECT\_TIMEOUT: “1200”  
JUPYTER\_GATEWAY\_REQUEST\_TIMEOUT: “1200”  
JUPYTER\_GATEWAY\_HTTP\_USER: “jovyan”  
KERNEL\_USERNAME: “jovyan”

do i need to set password? but where @JEG ? or Jupyterhub login pass do i need to specify ? but what is the env var for that?

---

<div class="post-metadata">

**Author:** ![minrk](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/minrk/32/13_2.png) [@minrk](https://discourse.jupyter.org/u/minrk)\
**Post date:** [January 12, 2024, 2:06pm UTC](https://discourse.jupyter.org/t/failed-to-connect-jupyter-enterprise-gateway-from-jupyterhub/23319/5 "2024-01-12T14:06:27Z")

</div>

This is going to be up to how your gateway is configured, which isn’t part of JupyterHub.

From [the gateway docs](https://jupyter-enterprise-gateway.readthedocs.io/en/latest/users/client-config.html), you can set either $JUPYTER\_GATEWAY\_AUTH\_TOKEN to auth with a token or $JUPYTER\_GATEWAY\_HTTP\_PWD to auth with a password. What values to use will depend on what you’ve provided to your enterprise gateway.

---

<div class="post-metadata">

**Author:** ![Aniket\_Gole](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/aniket_gole/32/11918_2.png) [@Aniket\_Gole](https://discourse.jupyter.org/u/Aniket_Gole)\
**Post date:** [January 15, 2024, 12:06pm UTC](https://discourse.jupyter.org/t/failed-to-connect-jupyter-enterprise-gateway-from-jupyterhub/23319/6 "2024-01-15T12:06:05Z")

</div>

@minrk i am able to connect jupyterhub to JEG

 ![ss2](https://canada1.discourse-cdn.com/flex031/uploads/jupyter/original/2X/6/6f30f39e727d95583d7f1dbe28c5233f2744a0f9.png)

after selecting notebook from given images new pod has been started but at the jupyterhub its showing kernel unknown status.

@ New JEG kernel pods there is no any error  
@ Jupyterhub Jovyan pod also dont see any error

how to go forward ? i am unable to execute anything on notebook.

 ![ss3](https://canada1.discourse-cdn.com/flex031/uploads/jupyter/original/2X/6/6dc98dedff2b9291f29ac7b9486f59435cf90dc5.png)

found some logs in JEG

```auto
[D 2024-01-15 12:43:58.002 EnterpriseGatewayApp] Connecting to: tcp://10.1.22.46:50171

Mon, Jan 15 2024 4:43:58 pm[D 2024-01-15 12:43:58.003 EnterpriseGatewayApp] Nudge: attempt 1 on kernel 0f029e04-7079-4e7d-bd85-0eeb73ecd494

Mon, Jan 15 2024 4:43:58 pm[D 2024-01-15 12:43:58.051 EnterpriseGatewayApp] Nudge: IOPub received: 0f029e04-7079-4e7d-bd85-0eeb73ecd494

Mon, Jan 15 2024 4:43:58 pm[D 2024-01-15 12:43:58.051 EnterpriseGatewayApp] Nudge: resolving iopub future: 0f029e04-7079-4e7d-bd85-0eeb73ecd494

Mon, Jan 15 2024 4:43:58 pm[D 2024-01-15 12:43:58.052 EnterpriseGatewayApp] activity on 0f029e04-7079-4e7d-bd85-0eeb73ecd494: status (busy)

Mon, Jan 15 2024 4:43:58 pm[D 2024-01-15 12:43:58.052 EnterpriseGatewayApp] Nudge: shell info reply received: 0f029e04-7079-4e7d-bd85-0eeb73ecd494 2024-01-15T12:43:58.053052576Z [D 2024-01-15 12:43:58.052 EnterpriseGatewayApp] Nudge: resolving shell future: 0f029e04-7079-4e7d-bd85-0eeb73ecd494

Mon, Jan 15 2024 4:43:58 pm[D 2024-01-15 12:43:58.053 EnterpriseGatewayApp] activity on 0f029e04-7079-4e7d-bd85-0eeb73ecd494: status (idle)

Mon, Jan 15 2024 4:43:58 pm[E 240115 12:43:58 web:1798] Uncaught exception GET /api/kernels/0f029e04-7079-4e7d-bd85-0eeb73ecd494/channels (10.1.41.111)

Mon, Jan 15 2024 4:43:58 pm HTTPServerRequest(protocol='http', host='enterprise-gateway.enterprise-gateway.svc.cluster.local:8888', method='GET', uri='/api/kernels/0f029e04-7079-4e7d-bd85-0eeb73ecd494/channels', version='HTTP/1.1', remote_ip='10.1.41.111')

Mon, Jan 15 2024 4:43:58 pm Traceback (most recent call last): 2024-01-15T12:43:58.174586767Z File "/opt/conda/lib/python3.10/site-packages/tornado/websocket.py", line 635, in _run_callback

Mon, Jan 15 2024 4:43:58 pm result = callback(*args, **kwargs) 2024-01-15T12:43:58.174595470Z File "/opt/conda/lib/python3.10/site-packages/jupyter_server/services/kernels/handlers.py", line 521, in on_message 2024-01-15T12:43:58.174600787Z msg = json.loads(ws_msg) 2024-01-15T12:43:58.174604913Z File "/opt/conda/lib/python3.10/json/ __init__.py", line 346, in loads 2024-01-15T12:43:58.174608541Z return _default_decoder.decode(s) 2024-01-15T12:43:58.174612132Z File "/opt/conda/lib/python3.10/json/decoder.py", line 337, in decode

Mon, Jan 15 2024 4:43:58 pm obj, end = self.raw_decode(s, idx=_w(s, 0).end())

Mon, Jan 15 2024 4:43:58 pm File "/opt/conda/lib/python3.10/json/decoder.py", line 355, in raw_decode

Mon, Jan 15 2024 4:43:58 pm raise JSONDecodeError("Expecting value", s, err.value) from None 2024-01-15T12:43:58.174627523Z json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 2024-01-15T12:43:58.174641600Z [D 2024-01-15 12:43:58.174 EnterpriseGatewayApp] Websocket closed 0f029e04-7079-4e7d-bd85-0eeb73ecd494:d29115a3-ba9e4b0988ab3fda09d2765d 2024-01-15T12:43:58.174739406Z [I 2024-01-15 12:43:58.174 EnterpriseGatewayApp] Starting buffering for 0f029e04-7079-4e7d-bd85-0eeb73ecd494:d29115a3-ba9e4b0988ab3fda09d2765d 2024-01-15T12:43:58.174816020Z [D 2024-01-15 12:43:58.174 EnterpriseGatewayApp] Clearing buffer for 0f029e04-7079-4e7d-bd85-0eeb73ecd494

```

---

<div class="post-metadata">

**Author:** ![minrk](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/minrk/32/13_2.png) [@minrk](https://discourse.jupyter.org/u/minrk)\
**Post date:** [January 16, 2024, 2:38pm UTC](https://discourse.jupyter.org/t/failed-to-connect-jupyter-enterprise-gateway-from-jupyterhub/23319/7 "2024-01-16T14:38:01Z")

</div>

I’m nmot sure, this is outside my experience for enterprise gateway. It looks like an unexpected non-JSON message on the websocket. You might double check that the jupyter-server version in both the single-user server pod and the enterprise gateway pods is the same.

---

<div class="post-metadata">

**Author:** ![Aniket\_Gole](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/aniket_gole/32/11918_2.png) [@Aniket\_Gole](https://discourse.jupyter.org/u/Aniket_Gole)\
**Post date:** [January 18, 2024, 5:58am UTC](https://discourse.jupyter.org/t/failed-to-connect-jupyter-enterprise-gateway-from-jupyterhub/23319/8 "2024-01-18T05:58:44Z")

</div>

# JEG version ([3.2.1 helm chart] ([https://github.com/jupyter-server/enterprise\_gateway/releases/download/v3.2.1/jupyter\_enterprise\_gateway\_helm-3.2.1.tar.gz](https://github.com/jupyter-server/enterprise_gateway/releases/download/v3.2.1/jupyter_enterprise_gateway_helm-3.2.1.tar.gz))

IPython : 8.0.1  
ipykernel : 6.7.0  
ipywidgets : not installed  
jupyter\_client : 6.1.12  
jupyter\_core : 4.9.1  
jupyter\_server : 1.13.4  
jupyterlab : 3.2.8  
nbclient : 0.5.10  
nbconvert : 6.4.0  
nbformat : 5.1.3  
notebook : 6.4.7  
qtconsole : not installed  
traitlets : 5.9.0

#JupyterHub 4.0.2 (3.2.1 all docker images)  
IPython : 8.18.1  
ipykernel : 6.26.0  
ipywidgets : not installed  
jupyter\_client : 8.6.0  
jupyter\_core : 5.5.0  
jupyter\_server : 2.11.2  
jupyterlab : 4.0.9  
nbclient : 0.9.0  
nbconvert : 7.11.0  
nbformat : 5.9.2  
notebook : not installed  
qtconsole : not installed  
traitlets : 5.13.0

---

<div class="post-metadata">

**Author:** ![minrk](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/minrk/32/13_2.png) [@minrk](https://discourse.jupyter.org/u/minrk)\
**Post date:** [January 19, 2024, 10:49am UTC](https://discourse.jupyter.org/t/failed-to-connect-jupyter-enterprise-gateway-from-jupyterhub/23319/9 "2024-01-19T10:49:18Z")

</div>

that looks like it - jupyter-server major versions are different. Try using jupyter-server 2 in both places.

---

<div class="post-metadata">

**Author:** ![Aniket\_Gole](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/aniket_gole/32/11918_2.png) [@Aniket\_Gole](https://discourse.jupyter.org/u/Aniket_Gole)\
**Post date:** [January 22, 2024, 1:10pm UTC](https://discourse.jupyter.org/t/failed-to-connect-jupyter-enterprise-gateway-from-jupyterhub/23319/10 "2024-01-22T13:10:21Z")

</div>

@minrk

i did below upgrade on JEG docker image but no luck, getting below new error

- jupyter\_client\>=8.6
- jupyter\_core\>=5.5.0
- jupyter\_server\>=2.10.1
- paramiko\>=2.11
- pexpect\>=4.8.0
- pyzmq\>=25.1.1

2024-01-22T12:56:05.493852908Z [I 2024-01-22 12:56:05.493 EnterpriseGatewayApp] Jupyter Enterprise Gateway 3.2.2 is available at [http://0.0.0.0:8888](http://0.0.0.0:8888)  
2024-01-22T12:56:06.754804256Z /opt/conda/lib/python3.10/site-packages/jupyter\_server/base/handlers.py:607: UserWarning: The Tornado web application does not have an ‘identity\_provider’ defined in its settings. In future releases of jupyter\_server, this will be a required key for all subclasses of `JupyterHandler`. For an example, see the jupyter\_server source code for how to add an identity provider to the tornado settings: [jupyter\_server/jupyter\_server/serverapp.py at 2a88e2a2c033578b7d72b440c296acbbe80fb398 · jupyter-server/jupyter\_server · GitHub](https://github.com/jupyter-server/jupyter_server/blob/2a88e2a2c033578b7d72b440c296acbbe80fb398/jupyter_server/serverapp.py#L242)  
2024-01-22T12:56:06.754829420Z if type(self.identity\_provider) is IdentityProvider and mod\_obj. **name**!= **name** :  
2024-01-22T12:56:06.756014026Z [E 240122 12:56:06 web:1798] Uncaught exception GET /api/kernels/693f9c8a-02b7-46d7-899c-c7b2166283c0/channels (10.1.32.47)  
2024-01-22T12:56:06.756019838Z HTTPServerRequest(protocol=‘http’, host=‘enterprise-gateway.enterprise-gateway.svc.cluster.local:8888’, method=‘GET’, uri=‘/api/kernels/693f9c8a-02b7-46d7-899c-c7b2166283c0/channels’, version=‘HTTP/1.1’, remote\_ip=‘10.1.32.47’)  
2024-01-22T12:56:06.756022596Z Traceback (most recent call last):  
2024-01-22T12:56:06.756024937Z File “/opt/conda/lib/python3.10/site-packages/tornado/web.py”, line 1692, in \_execute  
2024-01-22T12:56:06.756027526Z result = await result  
2024-01-22T12:56:06.756029859Z File “/  
opt/conda/lib/python3.10/site-packages/jupyter\_server/base/handlers.py”, line 622, in prepare  
2024-01-22T12:56:06.756032074Z \_user = await \_user  
2024-01-22T12:56:06.756034346Z File “/opt/conda/lib/python3.10/site-packages/jupyter\_server/auth/identity.py”, line 235, in \_get\_user  
2024-01-22T12:56:06.756050446Z \_cookie\_user = self.get\_user\_cookie(handler)  
2024-01-22T12:56:06.756052824Z File “/opt/conda/lib/python3.10/site-packages/jupyter\_server/auth/identity.py”, line 395, in get\_user\_cookie  
\_user\_cookie = handler.get\_secure\_cookie(  
2024-01-22T12:56:06.756056838Z File “/opt/conda/lib/python3.10/site-packages/tornado/web.py”, line 773, in get\_secure\_cookie  
2024-01-22T12:56:06.756058950Z self.require\_setting(“cookie\_secret”, “secure cookies”)  
2024-01-22T12:56:06.756060956Z File “/opt/conda/lib/python3.10/site-packages/tornado/web.py”, line 1592, in require\_setting  
2024-01-22T12:56:06.756063393Z raise Exception(  
2024-01-22T12:56:06.756065521Z Exception: You must define the ‘cookie\_secret’ setting in your application to use secure cookies  
2024-01-22T12:56:06.756501970Z [E 240122 12:56:06 web:2271] 500 GET /api/kernels/693f9c8a-02b7-46d7-899c-c7b2166283c0/channels (10.1.32.47) 2.42ms  
2024-01-22T12:56:07.003324034Z [E 240122 12:56:07 web:1798] Uncaught exception GET /api/kernels/693f9c8a-02b7-46d7-899c-c7b2166283c0/channels (10.1.32.47)  
HTTPServerRequest(protocol=‘http’, host=‘enterprise-gateway.enterprise-gateway.svc.cluster.local:8888’, method=‘GET’, uri=‘/api/kernels/693f9c8a-02b7-46d7-899c-c7b2166283c0/channels’, version=‘HTTP/1.1’, remote\_ip=‘10.1.32.47’)  
2024-01-22T12:56:07.003352492Z Traceback (most recent call last):

---

<div class="post-metadata">

**Author:** ![Aniket\_Gole](https://yyz1.discourse-cdn.com/flex031/user_avatar/discourse.jupyter.org/aniket_gole/32/11918_2.png) [@Aniket\_Gole](https://discourse.jupyter.org/u/Aniket_Gole)\
**Post date:** [January 26, 2024, 11:28am UTC](https://discourse.jupyter.org/t/failed-to-connect-jupyter-enterprise-gateway-from-jupyterhub/23319/11 "2024-01-26T11:28:10Z")

</div>

i changed jupyter\_server : 1.13.4 at jupyterhub and issue has been resolved, thanks for your support @minrk