Spawner issue - Docker container

Hey,
I am trying to run jupyterhub on docker with my custom config file, but i am unable to.

In the custom config file i am trying to create a custom authenticator and a docker spawner class.

i am getting

500 : Internal Server Error

Unhandled error starting server aditya_admin

You can try restarting your server from the [home page]

and the logs are 
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.369 JupyterHub app:2885] Running JupyterHub version 4.1.5
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.369 JupyterHub app:2915] Using Authenticator: builtins.JWTAuthenticator
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.369 JupyterHub app:2915] Using Spawner: builtins.MyDockerSpawner
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.369 JupyterHub app:2915] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.1.5
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.382 JupyterHub app:1728] Writing cookie_secret to /srv/jupyterhub/jupyterhub_cookie_secret
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.419 alembic.runtime.migration migration:216] Context impl SQLiteImpl.
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.419 alembic.runtime.migration migration:219] Will assume non-transactional DDL.
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.429 alembic.runtime.migration migration:622] Running stamp_revision  -> 0eee8c825d24
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.546 JupyterHub proxy:557] Generating new CONFIGPROXY_AUTH_TOKEN
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.592 JupyterHub roles:239] Adding role admin for User: aditya_admin
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.602 JupyterHub roles:239] Adding role user for User: aditya_admin
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.642 JupyterHub app:2954] Initialized 0 spawners in 0.004 seconds
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.651 JupyterHub metrics:279] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.652 JupyterHub metrics:279] Found 0 active users in the last ActiveUserPeriods.seven_days
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.654 JupyterHub metrics:279] Found 0 active users in the last ActiveUserPeriods.thirty_days
2024-04-26 17:09:10 [W 2024-04-26 11:39:10.655 JupyterHub proxy:747] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
2024-04-26 17:09:10 [I 2024-04-26 11:39:10.655 JupyterHub proxy:751] Starting proxy @ http://:8000
2024-04-26 17:09:11 [I 2024-04-26 11:39:11.431 JupyterHub app:3204] Hub API listening on http://0.0.0.0:8081/hub/
2024-04-26 17:09:11 [I 2024-04-26 11:39:11.432 JupyterHub app:3206] Private Hub API connect url http://89ada3ed423f:8081/hub/
2024-04-26 17:09:11 [I 2024-04-26 11:39:11.433 JupyterHub proxy:478] Adding route for Hub: / => http://89ada3ed423f:8081
2024-04-26 17:09:11 [I 2024-04-26 11:39:11.436 JupyterHub app:3271] JupyterHub is now running at http://:8000
2024-04-26 17:09:15 [I 2024-04-26 11:39:15.644 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'8E7jgxHnap8oe7a0vO-dSy_YJWnp6zIylbeLMLxAeXw=:c8632421e88b46e29f624fa9438cfc2b' {'path': '/hub/'}
2024-04-26 17:09:15 [I 2024-04-26 11:39:15.644 JupyterHub base:937] User logged in: aditya_admin
2024-04-26 17:09:15 [I 2024-04-26 11:39:15.645 JupyterHub log:192] 302 GET /hub/login?next=&token=[secret] -> /hub/spawn?token=[secret] (aditya_admin@::ffff:172.17.0.1) 20.85ms
2024-04-26 17:09:15 [I 2024-04-26 11:39:15.715 JupyterHub provider:660] Creating oauth client jupyterhub-user-aditya_admin
2024-04-26 17:09:15 [E 2024-04-26 11:39:15.750 JupyterHub user:887] Unhandled error starting aditya_admin's server: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-04-26 17:09:15     Traceback (most recent call last):
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 801, in spawn
2024-04-26 17:09:15         url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1294, in start
2024-04-26 17:09:15         await self.pull_image(image)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1268, in pull_image
2024-04-26 17:09:15         await self.docker('inspect_image', image)
2024-04-26 17:09:15       File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
2024-04-26 17:09:15         result = self.fn(*self.args, **self.kwargs)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 992, in _docker
2024-04-26 17:09:15         m = getattr(self.client, method)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 140, in client
2024-04-26 17:09:15         client = docker.APIClient(**kwargs)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/docker/api/client.py", line 197, in __init__
2024-04-26 17:09:15         self._version = self._retrieve_server_version()
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/docker/api/client.py", line 220, in _retrieve_server_version
2024-04-26 17:09:15         raise DockerException(
2024-04-26 17:09:15     docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-04-26 17:09:15     
2024-04-26 17:09:15 [E 2024-04-26 11:39:15.775 JupyterHub user:897] Failed to cleanup aditya_admin's server that failed to start
2024-04-26 17:09:15     Traceback (most recent call last):
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 895, in spawn
2024-04-26 17:09:15         await self.stop(spawner.name)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 989, in stop
2024-04-26 17:09:15         status = await spawner.poll()
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1006, in poll
2024-04-26 17:09:15         container = await self.get_object()
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1029, in get_object
2024-04-26 17:09:15         obj = await self.docker("inspect_%s" % self.object_type, self.object_name)
2024-04-26 17:09:15       File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
2024-04-26 17:09:15         result = self.fn(*self.args, **self.kwargs)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 992, in _docker
2024-04-26 17:09:15         m = getattr(self.client, method)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 140, in client
2024-04-26 17:09:10 11:39:10.891 [ConfigProxy] info: Proxying http://*:8000 to (no default)
2024-04-26 17:09:10 11:39:10.893 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
2024-04-26 17:09:11 11:39:11.431 [ConfigProxy] info: 200 GET /api/routes 
2024-04-26 17:09:11 11:39:11.433 [ConfigProxy] info: 200 GET /api/routes 
2024-04-26 17:09:11 11:39:11.434 [ConfigProxy] info: Adding route / -> http://89ada3ed423f:8081
2024-04-26 17:09:11 11:39:11.435 [ConfigProxy] info: Route added / -> http://89ada3ed423f:8081
2024-04-26 17:09:11 11:39:11.435 [ConfigProxy] info: 201 POST /api/routes/ 
2024-04-26 17:09:15         client = docker.APIClient(**kwargs)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/docker/api/client.py", line 197, in __init__
2024-04-26 17:09:15         self._version = self._retrieve_server_version()
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/docker/api/client.py", line 220, in _retrieve_server_version
2024-04-26 17:09:15         raise DockerException(
2024-04-26 17:09:15     docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-04-26 17:09:15     
2024-04-26 17:09:15 [E 2024-04-26 11:39:15.776 JupyterHub pages:314] Error starting server aditya_admin: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-04-26 17:09:15     Traceback (most recent call last):
2024-04-26 17:09:15     NoneType: None
2024-04-26 17:09:15     
2024-04-26 17:09:15 [E 2024-04-26 11:39:15.776 JupyterHub pages:201] Failed to spawn single-user server with query arguments
2024-04-26 17:09:15     Traceback (most recent call last):
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/handlers/pages.py", line 197, in _get
2024-04-26 17:09:15         return await self._wrap_spawn_single_user(
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/handlers/pages.py", line 318, in _wrap_spawn_single_user
2024-04-26 17:09:15         raise web.HTTPError(
2024-04-26 17:09:15     tornado.web.HTTPError: HTTP 500: Internal Server Error (Unhandled error starting server aditya_admin)
2024-04-26 17:09:15     
2024-04-26 17:09:15 [I 2024-04-26 11:39:15.805 JupyterHub provider:662] Updating oauth client jupyterhub-user-aditya_admin
2024-04-26 17:09:15 [E 2024-04-26 11:39:15.827 JupyterHub user:887] Unhandled error starting aditya_admin's server: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-04-26 17:09:15     Traceback (most recent call last):
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 801, in spawn
2024-04-26 17:09:15         url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1294, in start
2024-04-26 17:09:15         await self.pull_image(image)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1268, in pull_image
2024-04-26 17:09:15         await self.docker('inspect_image', image)
2024-04-26 17:09:15       File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
2024-04-26 17:09:15         result = self.fn(*self.args, **self.kwargs)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 992, in _docker
2024-04-26 17:09:15         m = getattr(self.client, method)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 140, in client
2024-04-26 17:09:15         client = docker.APIClient(**kwargs)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/docker/api/client.py", line 197, in __init__
2024-04-26 17:09:15         self._version = self._retrieve_server_version()
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/docker/api/client.py", line 220, in _retrieve_server_version
2024-04-26 17:09:15         raise DockerException(
2024-04-26 17:09:15     docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-04-26 17:09:15     
2024-04-26 17:09:15 [E 2024-04-26 11:39:15.840 JupyterHub user:897] Failed to cleanup aditya_admin's server that failed to start
2024-04-26 17:09:15     Traceback (most recent call last):
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 895, in spawn
2024-04-26 17:09:15         await self.stop(spawner.name)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/jupyterhub/user.py", line 989, in stop
2024-04-26 17:09:15         status = await spawner.poll()
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1006, in poll
2024-04-26 17:09:15         container = await self.get_object()
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 1029, in get_object
2024-04-26 17:09:15         obj = await self.docker("inspect_%s" % self.object_type, self.object_name)
2024-04-26 17:09:15       File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
2024-04-26 17:09:15         result = self.fn(*self.args, **self.kwargs)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 992, in _docker
2024-04-26 17:09:15         m = getattr(self.client, method)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/dockerspawner/dockerspawner.py", line 140, in client
2024-04-26 17:09:15         client = docker.APIClient(**kwargs)
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/docker/api/client.py", line 197, in __init__
2024-04-26 17:09:15         self._version = self._retrieve_server_version()
2024-04-26 17:09:15       File "/usr/local/lib/python3.10/dist-packages/docker/api/client.py", line 220, in _retrieve_server_version
2024-04-26 17:09:15         raise DockerException(
2024-04-26 17:09:15     docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-04-26 17:09:15     
2024-04-26 17:09:15 [E 2024-04-26 11:39:15.841 JupyterHub pages:314] Error starting server aditya_admin: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2024-04-26 17:09:15     Traceback (most recent call last):
2024-04-26 17:09:15     NoneType: None
2024-04-26 17:09:15     
2024-04-26 17:09:15 [W 2024-04-26 11:39:15.841 JupyterHub web:1873] 500 GET /hub/spawn?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6ImFkaXR5YV9hZG1pbiIsInJlcG9zaXRvcnlfbmFtZSI6Ikp1cHl0ZXJJbnRlZ3JhdGlvbiJ9.o3MRnpKoBtkI9fk55jyKNpVcFuPsiEYJefbdeiE7zF8 (::ffff:172.17.0.1): Unhandled error starting server aditya_admin
2024-04-26 17:09:15 [I 2024-04-26 11:39:15.842 JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'faf65f2b74764a2ea3dd1d454257a717:c8632421e88b46e29f624fa9438cfc2b' {'path': '/hub/'}
2024-04-26 17:09:15 [E 2024-04-26 11:39:15.867 JupyterHub log:184] {
2024-04-26 17:09:15       "X-Forwarded-Host": "localhost:8000",
2024-04-26 17:09:15       "X-Forwarded-Proto": "http",
2024-04-26 17:09:15       "X-Forwarded-Port": "8000",
2024-04-26 17:09:15       "X-Forwarded-For": "::ffff:172.17.0.1",
2024-04-26 17:09:15       "Cookie": "jupyterhub-hub-login=[secret]; _xsrf=[secret]; jupyterhub-session-id=[secret]",
2024-04-26 17:09:15       "Accept-Language": "en-US,en;q=0.9",
2024-04-26 17:09:15       "Accept-Encoding": "gzip, deflate, br, zstd",
2024-04-26 17:09:15       "Sec-Ch-Ua-Platform": "\"Windows\"",
2024-04-26 17:09:15       "Sec-Ch-Ua-Mobile": "?0",
2024-04-26 17:09:15       "Sec-Ch-Ua": "\"Chromium\";v=\"124\", \"Google Chrome\";v=\"124\", \"Not-A.Brand\";v=\"99\"",
2024-04-26 17:09:15       "Sec-Fetch-Dest": "document",
2024-04-26 17:09:15       "Sec-Fetch-User": "?1",
2024-04-26 17:09:15       "Sec-Fetch-Mode": "navigate",
2024-04-26 17:09:15       "Sec-Fetch-Site": "none",
2024-04-26 17:09:15       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
2024-04-26 17:09:15       "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
2024-04-26 17:09:15       "Upgrade-Insecure-Requests": "1",
2024-04-26 17:09:15       "Connection": "keep-alive",
2024-04-26 17:09:15       "Host": "localhost:8000"
2024-04-26 17:09:15     }
2024-04-26 17:09:15 [E 2024-04-26 11:39:15.867 JupyterHub log:192] 500 GET /hub/spawn?token=[secret] (aditya_admin@::ffff:172.17.0.1) 212.44ms

Does it work without your custom config file?

Can you show us your full configuration, and tell us how you installed JupyterHub, including versions of components?

This seems like docker isn’t setup properly, or you / the hub aren’t properly set as docker users.

If you are running from docker-compose, add the docker socket to your volumes:

hub:
  volumes:
    - "/var/run/docker.sock:/var/run/docker.sock"