Jupyterhub with AzureAD

Richard001 · November 1, 2023, 2:59pm

Hello,

I’m trying to create docker image with azure ad login im able to login with my account after login in webpage i get

500 : Internal Server Error

Unhandled error starting server Username

You can try restarting your server from the [home page](

Log

D 2023-11-01 14:34:46.657 JupyterHub base:961] Initiating spawn for USER.Ext
[D 2023-11-01 14:34:46.657 JupyterHub base:965] 0/100 concurrent spawns
[D 2023-11-01 14:34:46.657 JupyterHub base:970] 0 active servers
[I 2023-11-01 14:34:46.689 JupyterHub provider:659] Creating oauth client jupyterhub-user-USER.Ext
[D 2023-11-01 14:34:46.709 JupyterHub user:794] Calling Spawner.start for USER.Ext
[E 2023-11-01 14:34:46.714 JupyterHub user:884] Unhandled error starting USER.Ext's server: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 798, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 1255, in start
        await self.pull_image(image)
      File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 1211, in pull_image
        await self.docker('inspect_image', image)
      File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 947, in _docker
        m = getattr(self.client, method)
      File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 122, in client
        client = docker.APIClient(**kwargs)
      File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 197, in __init__
        self._version = self._retrieve_server_version()
      File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
        raise DockerException(
    docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

[D 2023-11-01 14:34:46.719 JupyterHub user:982] Stopping USER.Ext
[D 2023-11-01 14:34:46.719 JupyterHub dockerspawner:982] Getting container 'jupyter-richard-2e-2e-45xt'
[E 2023-11-01 14:34:46.749 JupyterHub user:894] Failed to cleanup USER.Ext's server that failed to start
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 892, in spawn
        await self.stop(spawner.name)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 986, in stop
        status = await spawner.poll()
      File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 961, in poll
        container = await self.get_object()
      File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 984, in get_object
        obj = await self.docker("inspect_%s" % self.object_type, self.object_name)
      File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 947, in _docker
        m = getattr(self.client, method)
      File "/usr/local/lib/python3.8/dist-packages/dockerspawner/dockerspawner.py", line 122, in client
        client = docker.APIClient(**kwargs)
      File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 197, in __init__
        self._version = self._retrieve_server_version()
      File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
        raise DockerException(
    docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

[E 2023-11-01 14:34:46.751 JupyterHub pages:313] Error starting server USER.Ext: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
    Traceback (most recent call last):
    None: None

[W 2023-11-01 14:34:46.751 JupyterHub web:1869] 500 GET /hub/spawn/USER.Ext (::ffff:IP): Unhandled error starting server USER.Ext
[D 2023-11-01 14:34:46.752 JupyterHub base:1371] No template for 500
[E 2023-11-01 14:34:46.786 JupyterHub log:183] {
      "X-Forwarded-Host": "domain:8000",
      "X-Forwarded-Proto": "https",
      "X-Forwarded-Port": "8000",
      "X-Forwarded-For": "::ffff:10.141.7.222",
      "Cookie": "_xsrf=[secret]; jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]; grafana_session=[secret]; grafana_session_expiry=[secret]; redirect_to=[secret]",
      "Accept-Language": "en-US,en;q=0.9",
      "Accept-Encoding": "gzip, deflate, br",
      "Referer": "https://domain:8000/hub/home",
      "Sec-Fetch-Dest": "document",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Site": "same-origin",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 Edg/118.0.2088.61",
      "Dnt": "1",
      "Upgrade-Insecure-Requests": "1",
      "Sec-Ch-Ua-Platform": "\"Windows\"",
      "Sec-Ch-Ua-Mobile": "?0",
      "Sec-Ch-Ua": "\"Chromium\";v=\"118\", \"Microsoft Edge\";v=\"118\", \"Not=A?Brand\";v=\"99\"",
      "Connection": "keep-alive",
      "Host": "domain:8000"
    }
[E 2023-11-01 14:34:46.786 JupyterHub log:191] 500 GET /hub/spawn/USER.Ext (USER.Ext@::ffff:10.141.7.222) 133.63ms
[D 2023-11-01 14:39:35.184 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
14:39:35.189 [ConfigProxy] info: 200 GET /api/routes
[D 2023-11-01 14:39:35.190 JupyterHub proxy:392] Checking routes
[D 2023-11-01 14:44:35.185 JupyterHub proxy:880] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
14:44:35.188 [ConfigProxy] info: 200 GET /api/routes
[D 2023-11-01 14:44:35.189 JupyterHub proxy:392] Checking routes

config

import os
#from dockerspawner import SystemUserSpawner
from dockerspawner import DockerSpawner
from oauthenticator.azuread import AzureAdOAuthenticator

class CustomAzureAdOAuthenticator(AzureAdOAuthenticator):
    def normalize_username(self, username):
        return username.split('@')[0]
        
# JupyterHub configuration
c.JupyterHub.authenticator_class = CustomAzureAdOAuthenticator
c.Application.log_level = 'DEBUG'

# Azure AD
c.AzureAdOAuthenticator.tenant_id = 
c.AzureAdOAuthenticator.oauth_callback_url = '
c.AzureAdOAuthenticator.client_id = 
c.AzureAdOAuthenticator.client_secret = 
c.AzureAdOAuthenticator.scope = ["openid", "email","profile"]
c.AzureAdOAuthenticator.allow_all = True
#c.AzureAdOAuthenticator.username_claim = 'upn'
c.AzureAdOAuthenticator.username_claim = 'unique_name'
c.AzureAdOAuthenticator.allowed_users = ()


# Spawner
c.JupyterHub.hub_ip = '0.0.0.0'  
c.JupyterHub.hub_connect_ip = '0.0.0.0'  
c.JupyterHub.spawner_class = DockerSpawner
c.DockerSpawner.host_homedir_format_string = "/home/{username}"
c.DockerSpawner.user_workingdir = '/home/{username}'
c.DockerSpawner.image = 'jupyter/minimal-notebook:latest'  
c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.debug = True
c.Spawner.cmd = ['jupyter']
c.Spawner.args = ['lab']
c.NotebookApp.open_browser = False

# Certificates
c.JupyterHub.ssl_cert = '/etc/openssl/cert.cer'
c.JupyterHub.ssl_key = '/etc/openssl/cert.key'

What could cause the error?

Jupyterhub version:4.0.2

pip3 install jupyterhub
Requirement already satisfied: jupyterhub in /usr/local/lib/python3.8/dist-packages (4.0.2)
Requirement already satisfied: alembic>=1.4 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (1.12.1)
Requirement already satisfied: async-generator>=1.9 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (1.10)
Requirement already satisfied: certipy>=0.1.2 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (0.1.3)
Requirement already satisfied: jinja2>=2.11.0 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (3.1.2)
Requirement already satisfied: jupyter-telemetry>=0.1.0 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (0.1.0)
Requirement already satisfied: oauthlib>=3.0 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (3.2.2)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (23.2)
Requirement already satisfied: prometheus-client>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (0.17.1)
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (2.8.2)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from jupyterhub) (2.22.0)
Requirement already satisfied: SQLAlchemy>=1.4 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (2.0.22)
Requirement already satisfied: tornado>=5.1 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (6.3.3)
Requirement already satisfied: traitlets>=4.3.2 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (5.13.0)
Requirement already satisfied: importlib-metadata>=3.6 in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (6.8.0)
Requirement already satisfied: pamela in /usr/local/lib/python3.8/dist-packages (from jupyterhub) (1.1.0)
Requirement already satisfied: Mako in /usr/local/lib/python3.8/dist-packages (from alembic>=1.4->jupyterhub) (1.2.4)
Requirement already satisfied: typing-extensions>=4 in /usr/local/lib/python3.8/dist-packages (from alembic>=1.4->jupyterhub) (4.8.0)
Requirement already satisfied: importlib-resources in /usr/local/lib/python3.8/dist-packages (from alembic>=1.4->jupyterhub) (6.1.0)
Requirement already satisfied: pyopenssl in /usr/local/lib/python3.8/dist-packages (from certipy>=0.1.2->jupyterhub) (23.3.0)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata>=3.6->jupyterhub) (3.17.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2>=2.11.0->jupyterhub) (2.1.3)
Requirement already satisfied: jsonschema in /usr/local/lib/python3.8/dist-packages (from jupyter-telemetry>=0.1.0->jupyterhub) (4.19.1)
Requirement already satisfied: python-json-logger in /usr/local/lib/python3.8/dist-packages (from jupyter-telemetry>=0.1.0->jupyterhub) (2.0.7)
Requirement already satisfied: ruamel.yaml in /usr/local/lib/python3.8/dist-packages (from jupyter-telemetry>=0.1.0->jupyterhub) (0.18.3)
Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.8/dist-packages (from SQLAlchemy>=1.4->jupyterhub) (3.0.1)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil->jupyterhub) (1.14.0)
Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema->jupyter-telemetry>=0.1.0->jupyterhub) (23.1.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.8/dist-packages (from jsonschema->jupyter-telemetry>=0.1.0->jupyterhub) (2023.7.1)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /usr/local/lib/python3.8/dist-packages (from jsonschema->jupyter-telemetry>=0.1.0->jupyterhub) (1.3.10)
Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.8/dist-packages (from jsonschema->jupyter-telemetry>=0.1.0->jupyterhub) (0.30.2)
Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.8/dist-packages (from jsonschema->jupyter-telemetry>=0.1.0->jupyterhub) (0.10.6)
Requirement already satisfied: cryptography<42,>=41.0.5 in /usr/local/lib/python3.8/dist-packages (from pyopenssl->certipy>=0.1.2->jupyterhub) (41.0.5)
Requirement already satisfied: ruamel.yaml.clib>=0.2.7 in /usr/local/lib/python3.8/dist-packages (from ruamel.yaml->jupyter-telemetry>=0.1.0->jupyterhub) (0.2.8)
Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.8/dist-packages (from cryptography<42,>=41.0.5->pyopenssl->certipy>=0.1.2->jupyterhub) (1.16.0)
Requirement already satisfied: pycparser in /usr/local/lib/python3.8/dist-packages (from cffi>=1.12->cryptography<42,>=41.0.5->pyopenssl->certipy>=0.1.2->jupyterhub) (2.21)

mahendrapaipuri · November 1, 2023, 5:32pm

That seems to me either docker is not installed/running or the user does not have access to docker. If you login into the machine via SSH with the same user as you are attempting with JupyterHub and execute docker version does it report back version?

If not, you will have add that user to docker group to access docker socket.

Richard001 · November 2, 2023, 8:11am

Hello, so inside docker image I should have docker installed? I’ve tried that no luck still the same error

mahendrapaipuri · November 2, 2023, 9:23am

Ok, seems like docker is installed on your machine. Did you run that docker -v as root or the user that you are logging in with JupyterHub?

TLDR; that error you are getting saying that the user do not have access to docker socket. So, you need to fix your permissions on the host where docker is installed

Richard001 · November 2, 2023, 10:32am

thanks for reply docker -v ive run with the user that is running jupyterhub, Im login in with azuread user name@host.com it wont create user in linux.

mahendrapaipuri · November 2, 2023, 10:37am

Well, you can use LocalAzureAdOAuthenticator and set c.LocalAuthenticator.create_system_users = True to create the users upon first login. Then you can create a script that creates users and adds them to Docker group and configure that script as add_user_cmd.

This way you will have users created and added to Docker group upon first login.

mahendrapaipuri · November 3, 2023, 8:06am

Looking into your question again, I guess you are running JupyterHub in a docker container. Is that so? If it is the case, you need to mount docker socket into the container for JupyterHub to be able to communicate with docker API on the host. I guess that is the issue you are running into.

Richard001 · November 3, 2023, 10:51am

Thanks for help, yeah that was probably dumb idea to add the docker spawner there I’ve just received the code from colleague that left. Changed it to local spawner and everything works fine.

Richard001 · November 3, 2023, 2:29pm

How exactly should i configure the jupyterhub to create user when logged in through azure ad?

import os
from jupyterhub.spawner import LocalProcessSpawner
from oauthenticator.azuread import AzureAdOAuthenticator

class CustomAzureAdOAuthenticator(AzureAdOAuthenticator):
    def normalize_username(self, username):
        return username.split('@')[0]


# JupyterHub configuration
c.JupyterHub.authenticator_class = CustomAzureAdOAuthenticator
c.Application.log_level = 'DEBUG'

# Azure AD
c.AzureAdOAuthenticator.tenant_id =
c.AzureAdOAuthenticator.oauth_callback_url =
c.AzureAdOAuthenticator.client_id =
c.AzureAdOAuthenticator.client_secret =
c.AzureAdOAuthenticator.scope = ["openid", "email","profile"]
c.AzureAdOAuthenticator.allow_all = True
c.AzureAdOAuthenticator.username_claim = 'upn'
c.AzureAdOAuthenticator.allowed_users = ()
c.LocalAzureAdOAuthenticator.create_system_users = True

It doesnt create user
ive tried
c.LocalAzureAdOAuthenticator.add_user_cmd = [‘useradd’, ‘-m’, ‘{normalized_username}’]
no luck

Thanks

EDIT

also tried this but no luck

import os
from jupyterhub.spawner import LocalProcessSpawner
from oauthenticator.azuread import AzureAdOAuthenticator
from oauthenticator.azuread import LocalAzureAdOAuthenticator
from jupyterhub.auth import LocalAuthenticator

#class CustomAzureAdOAuthenticator(AzureAdOAuthenticator):
#    def normalize_username(self, username):
#        return username.split('@')[0]


class LocalAzureAdOAuthenticator(LocalAuthenticator, AzureAdOAuthenticator):
    def normalize_username(self, username):
        return username.split('@')[0]

# JupyterHub configuration
c.JupyterHub.authenticator_class = LocalAzureAdOAuthenticator

mahendrapaipuri · November 3, 2023, 4:01pm

With a config as below:

class LocalAzureAdOAuthenticator(LocalAuthenticator, AzureAdOAuthenticator):
    def normalize_username(self, username):
        return username.split('@')[0]

# JupyterHub configuration
c.JupyterHub.authenticator_class = LocalAzureAdOAuthenticator
c.LocalAuthenticator.create_system_users = True
c.LocalAuthenticator.add_user_cmd = [‘useradd’, ‘-m’, ‘USERNAME’]

you should be able to create system users with useradd command. The string 'USERNAME' will be replaced by the normalized username that your authenticator passes to JupyterHub. The user will be always known with their normalized username in JupyterHub DB.

Topic		Replies	Views
500 : Internal Server Error azuread JupyterLab community , jupyterlab , jupyterhub , how-to , help-wanted	2	535	August 29, 2023
Spawner issue - Docker container JupyterHub help-wanted	2	220	May 14, 2024
I Need a working Dockerimage or Dockerfile with jupyterhub_config of tmpauthenticator JupyterHub communication , how-to , help-wanted	13	1078	November 3, 2020
Spawn server error discuss jupyterhub , help-wanted	1	387	May 11, 2023
Error 500 unsupported operand type JupyterHub jupyterhub , how-to , help-wanted	3	703	September 22, 2022

Jupyterhub with AzureAD

500 : Internal Server Error

Related topics