Getting error "socket.gaierror: [Errno -3] Temporary failure in name resolution" while upgrading from JH 1.1 to JH 2.3.1

JupyterHub
,
1

Hi All, I am deploying Jupyterhub version 2.3.1 using Helm chart version 1.1.3-n726.he9c8d919 on Kubernetes with hub and proxy running as separate pods in same namespace and single user notebook server pods spawned in separate namespace. Same setup worked fine with 0.9.1 helm chart version. However, while upgrading to latest Jupyterhub 2.3.1, I’m getting name resolution error on Hub API URL while notebook server spawning.

Below error is from singleuser server pod logs:
Error: [I 2022-06-23 08:06:35.396 SingleUserNotebookApp mixins:557] Starting jupyterhub-singleuser server version 1.3.0
[E 2022-06-23 08:06:35.401 SingleUserNotebookApp mixins:430] Failed to connect to my Hub at http://hub:8081/hub/api (attempt 1/5). Is it running?
Traceback (most recent call last):
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser/mixins.py”, line 428, in check_hub_version
resp = await client.fetch(self.hub_api_url)
File “/opt/conda/lib/python3.8/site-packages/tornado/simple_httpclient.py”, line 338, in run
stream = await self.tcp_client.connect(
File “/opt/conda/lib/python3.8/site-packages/tornado/tcpclient.py”, line 265, in connect
addrinfo = await self.resolver.resolve(host, port, af)
File “/opt/conda/lib/python3.8/site-packages/tornado/netutil.py”, line 398, in resolve
result = await IOLoop.current().run_in_executor(
File “/opt/conda/lib/python3.8/concurrent/futures/thread.py”, line 57, in run
result = self.fn(self.args, *self.kwargs)
File “/opt/conda/lib/python3.8/site-packages/tornado/netutil.py”, line 382, in _resolve_addr
addrinfo = socket.getaddrinfo(host, port, family, socket.SOCK_STREAM)
File “/opt/conda/lib/python3.8/socket.py”, line 918, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
Can anyone please help me what to look for here?

2

Hi! Please could you:

Thanks!

3

Debug logs from singleuser pod:

Entered start.sh with args: jupyterhub-singleuser --allow-root --ip=0.0.0.0 --port=8888 --debug --allow-root
Updated the jovyan user:

  • username: jovyan → mpreetka
  • home dir: /home/jovyan → /home/mpreetka
    Update mpreetka’s UID:GID to *:
    Changing working directory to /home/mpreetka/
    Running as mpreetka: jupyterhub-singleuser --allow-root --ip=0.0.0.0 --port=8888 --debug --allow-root
    [I 706 SingleUserLabApp mixins:615] Starting jupyterhub single-user server version 2.3.1
    [I 706 SingleUserLabApp mixins:629] Extending jupyterlab.labhubapp.SingleUserLabApp from jupyterlab 3.4.3
    [I 706 SingleUserLabApp mixins:629] Extending jupyter_server.serverapp.ServerApp from jupyter_server 1.18.0
    [D 790 SingleUserLabApp application:183] Searching [‘/home/mpreetka’, ‘/home/mpreetka/.jupyter’, ‘/home/mpreetka/.local/etc/jupyter’, ‘/opt/conda/etc/jupyter’, ‘/usr/local/etc/jupyter’, ‘/etc/jupyter’] for config files
    [D 790 SingleUserLabApp application:837] Looking for jupyter_config in /etc/jupyter
    [D 790 SingleUserLabApp application:837] Looking for jupyter_config in /usr/local/etc/jupyter
    [D 790 SingleUserLabApp application:837] Looking for jupyter_config in /opt/conda/etc/jupyter
    [D 790 SingleUserLabApp application:837] Looking for jupyter_config in /home/mpreetka/.local/etc/jupyter
    [D 790 SingleUserLabApp application:837] Looking for jupyter_config in /home/mpreetka/.jupyter
    [D 790 SingleUserLabApp application:837] Looking for jupyter_config in /home/mpreetka
    [D 791 SingleUserLabApp application:837] Looking for jupyter_server_config in /etc/jupyter
    [D 792 SingleUserLabApp application:858] Loaded config file: /etc/jupyter/jupyter_server_config.py
    [D 792 SingleUserLabApp application:837] Looking for jupyter_server_config in /usr/local/etc/jupyter
    [D 792 SingleUserLabApp application:837] Looking for jupyter_server_config in /opt/conda/etc/jupyter
    [D 792 SingleUserLabApp application:837] Looking for jupyter_server_config in /home/mpreetka/.local/etc/jupyter
    [D 792 SingleUserLabApp application:837] Looking for jupyter_server_config in /home/mpreetka/.jupyter
    [D 792 SingleUserLabApp application:837] Looking for jupyter_server_config in /home/mpreetka
    [W 793 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
    [D 795 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_server_config:
    /etc/jupyter/jupyter_server_config.json
    [D 795 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_server_config:
    /usr/local/etc/jupyter/jupyter_server_config.json
    [D 796 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_server_config:
    /opt/conda/etc/jupyter/jupyter_server_config.d/jupyterlab.json
    /opt/conda/etc/jupyter/jupyter_server_config.d/nbclassic.json
    /opt/conda/etc/jupyter/jupyter_server_config.d/notebook_shim.json
    /opt/conda/etc/jupyter/jupyter_server_config.json
    [D 796 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_server_config:
    /home/mpreetka/.local/etc/jupyter/jupyter_server_config.json
    [D 796 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_server_config:
    /home/mpreetka/.jupyter/jupyter_server_config.json
    [D 797 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_server_config:
    /home/mpreetka/jupyter_server_config.json
    [D 810 LabApp] Config changed: {‘NotebookApp’: {}, ‘ServerApp’: {‘ip’: ‘0.0.0.0’, ‘port’: 8888, ‘open_browser’: False, ‘jpserver_extensions’: <LazyConfigValue {‘update’: {‘jupyterlab’: True, ‘nbclassic’: True, ‘notebook_shim’: True}}>}, ‘FileContentsManager’: {‘delete_to_trash’: False}, ‘SingleUserLabApp’: {‘ip’: ‘0.0.0.0’, ‘port’: 8888, ‘allow_root’: True, ‘log_level’: ‘DEBUG’}, ‘ExtensionApp’: {‘log_level’: ‘DEBUG’}}
    [W 811 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
    [I 812 SingleUserLabApp manager:340] jupyterlab | extension was successfully linked.
    [W 815 NotebookApp] ‘ip’ has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
    [W 815 NotebookApp] ‘port’ has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
    [W 815 NotebookApp] ‘port’ has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
    [D 821 NotebookApp] Config changed: {‘NotebookApp’: {‘open_browser’: False}, ‘ServerApp’: {‘ip’: ‘0.0.0.0’, ‘port’: 8888, ‘open_browser’: False, ‘jpserver_extensions’: <LazyConfigValue value={‘jupyterlab’: True, ‘nbclassic’: True, ‘notebook_shim’: True}>}, ‘FileContentsManager’: {‘delete_to_trash’: False}, ‘SingleUserLabApp’: {‘ip’: ‘0.0.0.0’, ‘port’: 8888, ‘allow_root’: True, ‘log_level’: ‘DEBUG’}, ‘ExtensionApp’: {‘log_level’: ‘DEBUG’}}
    [W 822 SingleUserLabApp configurable:200] Config option open_browser not recognized by SingleUserLabApp. Did you mean browser?
    [I 823 SingleUserLabApp manager:340] nbclassic | extension was successfully linked.
    [D 093 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
    /home/mpreetka/.jupyter/jupyter_notebook_config.json
    [D 094 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
    /etc/jupyter/jupyter_notebook_config.json
    [D 094 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
    /usr/local/etc/jupyter/jupyter_notebook_config.json
    [D 094 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
    /opt/conda/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
    /opt/conda/etc/jupyter/jupyter_notebook_config.json
    [D 094 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
    /home/mpreetka/.local/etc/jupyter/jupyter_notebook_config.json
    [D 095 SingleUserLabApp config_manager:93] Paths used for configuration of jupyter_notebook_config:
    /home/mpreetka/.jupyter/jupyter_notebook_config.json
    [I 095 SingleUserLabApp manager:340] notebook_shim | extension was successfully linked.
    [I 108 SingleUserLabApp manager:362] notebook_shim | extension was successfully loaded.
    [I 109 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.10/site-packages/jupyterlab
    [I 109 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
    [I 112 SingleUserLabApp manager:362] jupyterlab | extension was successfully loaded.
    [I 167 SingleUserLabApp manager:362] nbclassic | extension was successfully loaded.
    [I 167 SingleUserLabApp mixins:648] Starting jupyterhub-singleuser server version 2.3.1
    [E 173 SingleUserLabApp mixins:490] Failed to connect to my Hub at http://hub:8081/hub/api (attempt 1/5). Is it running?
    Traceback (most recent call last):
    File “/opt/conda/lib/python3.10/site-packages/jupyterhub/singleuser/mixins.py”, line 488, in check_hub_version
    resp = await client.fetch(self.hub_api_url)
    File “/opt/conda/lib/python3.10/site-packages/tornado/simple_httpclient.py”, line 340, in run
    stream = await self.tcp_client.connect(
    File “/opt/conda/lib/python3.10/site-packages/tornado/tcpclient.py”, line 265, in connect
    addrinfo = await self.resolver.resolve(host, port, af)
    File “/opt/conda/lib/python3.10/site-packages/tornado/netutil.py”, line 433, in resolve
    for fam, _, _, _, address in await asyncio.get_running_loop().getaddrinfo(
    File “/opt/conda/lib/python3.10/asyncio/base_events.py”, line 860, in getaddrinfo
    return await self.run_in_executor(
    File “/opt/conda/lib/python3.10/concurrent/futures/thread.py”, line 58, in run
    result = self.fn(self.args, **self.kwargs)
    File “/opt/conda/lib/python3.10/socket.py”, line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    socket.gaierror: [Errno -3] Temporary failure in name resolution
    [I 452 SingleUserLabApp log:189] 302 GET /user/mpreetka/ → /user/mpreetka/lab? (@10.42.2.233) 0.48ms
    [I 133 SingleUserLabApp log:189] 302 GET /user/mpreetka/ → /user/mpreetka/lab? (@10.213.100.40) 0.68ms
    [D 349 SingleUserLabApp auth:584] No user identified
    [D 350 SingleUserLabApp auth:959] Redirecting to login url: /hub/api/oauth2/authorize?client_id=jupyterhub-user-mpreetka&redirect_uri=%2Fuser%2Fmpreetka%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjNhZjRlNjZlZWUyOTRkZTdhNmY5NDJiYzdmODdmMmI1IiwgIm5leHRfdXJsIjogIi91c2VyL21wcmVldGthL2xhYj8ifQ
    [I 351 SingleUserLabApp log:189] 302 GET /user/mpreetka/lab? → /hub/api/oauth2/authorize?client_id=jupyterhub-user-mpreetka&redirect_uri=%2Fuser%2Fmpreetka%2Foauth_callback&response_type=code&state=[secret] (@10.213.100.40) 2.09ms
    [D 800 SingleUserLabApp auth:584] No user identified
    [E 804 SingleUserLabApp auth:413] Error connecting to http://hub:8081/hub/api: HTTPConnectionPool(host=‘hub’, port=8081): Max retries exceeded with url: /hub/api/oauth2/token (Caused by NewConnectionError(‘<urllib3.connection.HTTPConnection object at 0x7ffff257cfa0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’))
    [W 804 SingleUserLabApp web:1796] 500 GET /user/mpreetka/oauth_callback?code=n2qsiA2NSeO8OFbocYB6vd1j2uy540&state=eyJ1dWlkIjogIjNhZjRlNjZlZWUyOTRkZTdhNmY5NDJiYzdmODdmMmI1IiwgIm5leHRfdXJsIjogIi91c2VyL21wcmVldGthL2xhYj8ifQ (10.213.100.40): Failed to connect to Hub API at ‘http://hub:8081/hub/api’. Is the Hub accessible at this URL (from host: jupyter-mpreetka)?
    [D 804 SingleUserLabApp handlers:273] Using contents: services/contents
    [D 804 SingleUserLabApp handlers:273] Using contents: services/contents
    [D 816 SingleUserLabApp handlers:881] Path favicon.ico served from /opt/conda/lib/python3.10/site-packages/jupyter_server/static/favicon.ico
    [D 816 SingleUserLabApp handlers:881] Path style/bootstrap.min.css served from /opt/conda/lib/python3.10/site-packages/jupyter_server/static/style/bootstrap.min.css
    [D 816 SingleUserLabApp handlers:881] Path style/bootstrap-theme.min.css served from /opt/conda/lib/python3.10/site-packages/jupyter_server/static/style/bootstrap-theme.min.css
    [D 816 SingleUserLabApp handlers:881] Path style/index.css served from /opt/conda/lib/python3.10/site-packages/jupyter_server/static/style/index.css
    [E 817 SingleUserLabApp log:181] {
    “Cookie”: “jupyterhub-user-mpreetka=[secret]; jupyterhub-user-mpreetka-oauth-state=[secret]; BadgeType=[secret]; CNCampusCode=[secret]; IDSID=[secret]; isManager=[secret]; ajs_user_id=[secret]; ajs_group_id=[secret]; ajs_anonymous_id=[secret]; _cs_c=[secret]; _fbp=[secret]; ELQSTATUS=[secret]; ELOQUA=[secret]; aam_uuid=[secret]; _ga_1BMXQFQY4K=[secret]; _ga=[secret]; _abck=[secret]; AMCV_AD2A1C8B53308E600A490D4D%40AdobeOrg=[secret]; mbox=[secret]; kndctr_AD2A1C8B53308E600A490D4D_AdobeOrg_identity=[secret]; adcloud=[secret]; _cs_id=[secret]; utag_main=[secret]; _xsrf=[secret]; _gid=[secret]; jupyterhub-session-id=[secret]”,
    “Accept-Language”: “en-US,en;q=0.9”,
    “Accept-Encoding”: “gzip, deflate, br”,
    “Referer”: “https:///hub/spawn-pending/mpreetka”,
    “Sec-Ch-Ua-Platform”: “"Windows"”,
    “Sec-Ch-Ua-Mobile”: “?0”,
    “Sec-Ch-Ua”: “".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"”,
    “Sec-Fetch-Dest”: “document”,
    “Sec-Fetch-Mode”: “navigate”,
    “Sec-Fetch-Site”: “same-origin”,
    “Accept”: "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,    /    ;q=0.8,application/signed-exchange;v=b3;q=0.9",
    “User-Agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36”,
    “Upgrade-Insecure-Requests”: “1”,
    “Cache-Control”: “max-age=0”,
    “X-Scheme”: “https”,
    “X-Forwarded-Scheme”: “https”,
    “X-Forwarded-Proto”: “https,http”,
    “X-Forwarded-Port”: “443,80”,
    “X-Forwarded-Host”: “”,
    “X-Forwarded-For”: “10.213.100.40,::ffff:10.42.0.0”,
    “X-Real-Ip”: “10.213.100.40”,
    “X-Request-Id”: “611151c9a53bbbb32ccc461b1da47161”,
    “Host”: “url”,
    “Connection”: “close”
    }
    [E 817 SingleUserLabApp log:189] 500 GET /user/mpreetka/oauth_callback?code=[secret]&state=[secret] (@10.213.100.40) 17.62ms
    [E 178 SingleUserLabApp mixins:490] Failed to connect to my Hub at http://hub:8081/hub/api (attempt 2/5). Is it running?
    Traceback (most recent call last):
    File “/opt/conda/lib/python3.10/site-packages/jupyterhub/singleuser/mixins.py”, line 488, in check_hub_version
    resp = await client.fetch(self.hub_api_url)
    File “/opt/conda/lib/python3.10/site-packages/tornado/simple_httpclient.py”, line 340, in run
    stream = await self.tcp_client.connect(
    File “/opt/conda/lib/python3.10/site-packages/tornado/tcpclient.py”, line 265, in connect
    addrinfo = await self.resolver.resolve(host, port, af)
    File “/opt/conda/lib/python3.10/site-packages/tornado/netutil.py”, line 433, in resolve
    for fam, _, _, _, address in await asyncio.get_running_loop().getaddrinfo(
    File “/opt/conda/lib/python3.10/asyncio/base_events.py”, line 860, in getaddrinfo
    return await self.run_in_executor(
    File “/opt/conda/lib/python3.10/concurrent/futures/thread.py”, line 58, in run
    result = self.fn(*self.args, **self.kwargs)
    File “/opt/conda/lib/python3.10/socket.py”, line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    socket.gaierror: [Errno -3] Temporary failure in name resolution
    [E 2022-07-06 05:50:41.187 SingleUserLabApp mixins:490] Failed to connect to my Hub at http://hub:8081/hub/api (attempt 3/5). Is it running?
    Traceback (most recent call last):
    File “/opt/conda/lib/python3.10/site-packages/jupyterhub/singleuser/mixins.py”, line 488, in check_hub_version
    resp = await client.fetch(self.hub_api_url)
    File “/opt/conda/lib/python3.10/site-packages/tornado/simple_httpclient.py”, line 340, in run
    stream = await self.tcp_client.connect(
    File “/opt/conda/lib/python3.10/site-packages/tornado/tcpclient.py”, line 265, in connect
    addrinfo = await self.resolver.resolve(host, port, af)
    File “/opt/conda/lib/python3.10/site-packages/tornado/netutil.py”, line 433, in resolve
    for fam, _, _, _, address in await asyncio.get_running_loop().getaddrinfo(
    File “/opt/conda/lib/python3.10/asyncio/base_events.py”, line 860, in getaddrinfo
    return await self.run_in_executor(
    File “/opt/conda/lib/python3.10/concurrent/futures/thread.py”, line 58, in run
    result = self.fn(*self.args, **self.kwargs)
    File “/opt/conda/lib/python3.10/socket.py”, line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    socket.gaierror: [Errno -3] Temporary failure in name resolution
    [E 194 SingleUserLabApp mixins:490] Failed to connect to my Hub at http://hub:8081/hub/api (attempt 4/5). Is it running?
    Traceback (most recent call last):
    File “/opt/conda/lib/python3.10/site-packages/jupyterhub/singleuser/mixins.py”, line 488, in check_hub_version
    resp = await client.fetch(self.hub_api_url)
    File “/opt/conda/lib/python3.10/site-packages/tornado/simple_httpclient.py”, line 340, in run
    stream = await self.tcp_client.connect(
    File “/opt/conda/lib/python3.10/site-packages/tornado/tcpclient.py”, line 265, in connect
    addrinfo = await self.resolver.resolve(host, port, af)
    File “/opt/conda/lib/python3.10/site-packages/tornado/netutil.py”, line 433, in resolve
    for fam, _, _, _, address in await asyncio.get_running_loop().getaddrinfo(
    File “/opt/conda/lib/python3.10/asyncio/base_events.py”, line 860, in getaddrinfo
    return await self.run_in_executor(
    File “/opt/conda/lib/python3.10/concurrent/futures/thread.py”, line 58, in run
    result = self.fn(*self.args, **self.kwargs)
    File “/opt/conda/lib/python3.10/socket.py”, line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    socket.gaierror: [Errno -3] Temporary failure in name resolution
5

Configuration:

hub:
  db:
    type: sqlite-memory
  image:
    name: jupyterhub/k8s-hub
  extraEnv:
    GRANT_SUDO: "yes"
  networkPolicy:
    enabled: false
singleuser:
  networkPolicy:
    enabled: false
  uid: 0
  image:
    pullSecrets:
      - <secret>
  storage:
    homeMountPath: /home/{username}
    extraVolumes:
    extraVolumeMounts:
    dynamic:
      storageClass: <sc>
  profileList:
    - <profile>
debug:
  enabled: true
prePuller:
  hook:
    enabled: false
proxy:
  secretToken: <secret>
ingress:
  enabled: true
  hosts:
   - <url>
  tls:
  - hosts:
    - <url>
    secretName: <secret>
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 200m
    nginx.ingress.kubernetes.io/client-header-buffer-size: 4k
c.KubeSpawner.cmd = ['/usr/local/bin/start.sh', 'jupyterhub-singleuser', '--allow-root']
c.Spawner.auth_state_hook = userdata_hook
c.JupyterHub.authenticator_class = LDAPAuthenticatorInfoUID
c.LDAPAuthenticator.auth_state_attributes = ['gidNumber', 'uidNumber', 'memberOf']
c.LDAPAuthenticator.server_address = <secret>
c.LDAPAuthenticator.server_port = <secret>
c.LDAPAuthenticator.lookup_dn = True
c.LDAPAuthenticator.use_ssl = True
c.LDAPAuthenticator.lookup_dn_search_filter = '({login_attr}={login})'
c.LDAPAuthenticator.lookup_dn_search_user = os.environ["LDAP_USER_NAME"]
c.LDAPAuthenticator.lookup_dn_search_password = os.environ["LDAP_USER_PASSWORD"]
c.LDAPAuthenticator.user_search_base = <secret>
c.LDAPAuthenticator.user_attribute = <secret>'
c.LDAPAuthenticator.lookup_dn_user_dn_attribute = 'secret'
c.LDAPAuthenticator.escape_userdn = False
c.LDAPAuthenticator.bind_dn_template = [
    <>
]
c.LDAPAuthenticator.allowed_groups = [
    <groups>
]

c.Authenticator.enable_auth_state = True
c.Authenticator.use_lookup_dn_username = False
c.Spawner.debug = True
6

@consideRatio Thanks for the edit to yaml configuration. I have also attached logs to the discourse post. Not sure if anything more needs to be edited to make the yaml configuration more readable?

7

Thanks for the response Simon. Added the debug logs and configuration for your reference.

8

I can’t see any obvious problems with your configuration. Could you share your hub debug logs too?

9

Hub logs:

[D 2022-07-20 15:56:39.270 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:363] Validating redirection uri /user/mpreetka/oauth_callback for client jupyterhub-user-mpreetka.
[D 2022-07-20 15:56:39.270 oauthlib.oauth2.rfc6749.grant_types.base base:230] Using provided redirect_uri /user/mpreetka/oauth_callback
[D 2022-07-20 15:56:39.270 JupyterHub provider:486] validate_redirect_uri: client_id=jupyterhub-user-mpreetka, redirect_uri=/user/mpreetka/oauth_callback
[D 2022-07-20 15:56:39.271 oauthlib.oauth2.rfc6749.grant_types.base base:171] Validating access to scopes for client ‘jupyterhub-user-mpreetka’ (<OAuthClient(identifier=‘jupyterhub-user-mpreetka’)>).
[D 2022-07-20 15:56:39.272 JupyterHub provider:570] Allowing request for role(s) for jupyterhub-user-mpreetka:
[D 2022-07-20 15:56:39.272 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:246] Pre resource owner authorization validation ok for <oauthlib.Request SANITIZED>.
[D 2022-07-20 15:56:39.272 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:171] Created authorization code grant {‘code’: ‘’, ‘state’: ‘’} for request <oauthlib.Request SANITIZED>.
[D 2022-07-20 15:56:39.272 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:278] Saving grant {‘code’: ‘’, ‘state’: ‘’} for <oauthlib.Request SANITIZED>.
[D 2022-07-20 15:56:39.272 JupyterHub provider:237] Saving authorization code jupyterhub-user-mpreetka, Hy1…, (), {}
[I 2022-07-20 15:56:39.274 JupyterHub log:189] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-mpreetka&redirect_uri=%2Fuser%2Fmpreetka%2Foauth_callback&response_type=code&state=[secret] → /user/mpreetka/oauth_callback?code=[secret]&state=[secret] (mpreetka@10.213.64.65) 9.12ms
[D 2022-07-20 15:56:39.662 JupyterHub log:189] 200 GET /hub/health (@10.184.112.133) 0.56ms
[D 2022-07-20 15:56:41.662 JupyterHub log:189] 200 GET /hub/health (@10.184.112.133) 0.56ms
[D 2022-07-20 15:56:42.260 JupyterHub reflector:362] events watcher timeout
[D 2022-07-20 15:56:42.260 JupyterHub reflector:281] Connecting events watcher
[D 2022-07-20 15:56:43.662 JupyterHub log:189] 200 GET /hub/health (@10.184.112.133) 0.57ms
[D 2022-07-20 15:56:45.090 JupyterHub reflector:362] pods watcher timeout
[D 2022-07-20 15:56:45.090 JupyterHub reflector:281] Connecting pods watcher
[D 2022-07-20 15:56:45.662 JupyterHub log:189] 200 GET /hub/health (@10.184.112.133) 0.57ms
[D 2022-07-20 15:56:45.830 JupyterHub log:189] 200 GET /hub/health (@10.184.112.133) 0.53ms
[E 2022-07-20 15:56:46.283 JupyterHub service:420] Service jupyterhub-idle-culler exited with status -9
[I 2022-07-20 15:56:46.283 JupyterHub service:377] Starting service ‘jupyterhub-idle-culler’: [‘python3’, ‘-m’, ‘jupyterhub_idle_culler’, ‘–url=http://localhost:8081/hub/api’, ‘–timeout=3600’, ‘–cull-every=600’, ‘–concurrency=10’]
[I 2022-07-20 15:56:46.284 JupyterHub service:132] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=3600 --cull-every=600 --concurrency=10
[D 2022-07-20 15:56:46.290 JupyterHub spawner:1258] Polling subprocess every 30s

10

I just noticed this. In your configuration there’s nothing that would spawn the singleuser pods in a different namespace. Is there something you haven’t mentioned about your setup?