Hello! I’m running JupyterHub on an EC2. I’m able to authenticate and enter remotely, but everything crashes when I try to spin up a singleuser container. I saw a very similar post whose solution was to examine the container’s log, and then they traced it back to the dockerfile). Unfortunately, I’m unable to generate these exact logs for some reason. It just makes an empty txt file. I can attach some logs/start-up files as well if that’s helpful.
Here is my jupyterhub log (user: yzuriaw):
[I 2024-05-30 18:05:54.432 JupyterHub app:2859] Running JupyterHub version 4.0.2
[I 2024-05-30 18:05:54.433 JupyterHub app:2889] Using Authenticator: oauthenticator.github.GitHubOAuthenticator-16.2.1
[I 2024-05-30 18:05:54.433 JupyterHub app:2889] Using Spawner: builtins.DockerSpawner
[I 2024-05-30 18:05:54.433 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.2
[I 2024-05-30 18:05:54.451 JupyterHub app:1664] Loading cookie_secret from /home/ubuntu/jupyterhub_cookie_secret
[I 2024-05-30 18:05:54.525 JupyterHub proxy:556] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2024-05-30 18:05:54.665 JupyterHub app:2928] Initialized 0 spawners in 0.003 seconds
[I 2024-05-30 18:05:54.672 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2024-05-30 18:05:54.673 JupyterHub metrics:278] Found 2 active users in the last ActiveUserPeriods.seven_days
[I 2024-05-30 18:05:54.674 JupyterHub metrics:278] Found 2 active users in the last ActiveUserPeriods.thirty_days
[W 2024-05-30 18:05:54.675 JupyterHub proxy:746] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else...
[I 2024-05-30 18:05:54.675 JupyterHub proxy:750] Starting proxy @ http://:8000
[E 2024-05-30 18:05:54.686 JupyterHub utils:91] Unexpected error connecting to ip-172-31-62-22:8000 [Errno 22] Invalid argument
[E 2024-05-30 18:05:54.831 JupyterHub utils:91] Unexpected error connecting to ip-172-31-62-22:8000 [Errno 22] Invalid argument
[E 2024-05-30 18:05:54.870 JupyterHub utils:91] Unexpected error connecting to ip-172-31-62-22:8000 [Errno 22] Invalid argument
18:05:55.038 [ConfigProxy] info: Proxying http://*:8000 to (no default)
18:05:55.041 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
18:05:55.118 [ConfigProxy] info: 200 GET /api/routes
[I 2024-05-30 18:05:55.118 JupyterHub app:3178] Hub API listening on http://172.17.0.1:8081/hub/
18:05:55.119 [ConfigProxy] info: 200 GET /api/routes
[I 2024-05-30 18:05:55.120 JupyterHub proxy:477] Adding route for Hub: / => http://172.17.0.1:8081
18:05:55.122 [ConfigProxy] info: Adding route / -> http://172.17.0.1:8081
18:05:55.123 [ConfigProxy] info: Route added / -> http://172.17.0.1:8081
18:05:55.124 [ConfigProxy] info: 201 POST /api/routes/
[I 2024-05-30 18:05:55.125 JupyterHub app:3245] JupyterHub is now running at http://:8000
[I 2024-05-30 18:06:06.182 JupyterHub log:191] 302 GET / -> /hub/ (@::ffff:73.201.170.13) 1.14ms
[I 2024-05-30 18:06:06.231 JupyterHub log:191] 302 GET /hub/ -> /hub/spawn (yzuriaw@::ffff:73.201.170.13) 23.64ms
[I 2024-05-30 18:06:06.285 JupyterHub provider:659] Creating oauth client jupyterhub-user-yzuriaw
[I 2024-05-30 18:06:06.329 JupyterHub dockerspawner:1033] Container 'jupyter-yzuriaw' is gone
[E 2024-05-30 18:06:06.330 JupyterHub user:884] Unhandled error starting yzuriaw's server: Replacement index 0 out of range for positional args tuple
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/lib/python3.8/site-packages/jupyterhub/user.py", line 798, in spawn
url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
File "/home/ubuntu/anaconda3/lib/python3.8/site-packages/dockerspawner/dockerspawner.py", line 1309, in start
obj = await self.create_object()
File "/home/ubuntu/anaconda3/lib/python3.8/site-packages/dockerspawner/dockerspawner.py", line 1162, in create_object
volumes=self.volume_mount_points,
File "/home/ubuntu/anaconda3/lib/python3.8/site-packages/dockerspawner/dockerspawner.py", line 867, in volume_mount_points
return sorted([value["bind"] for value in self.volume_binds.values()])
File "/home/ubuntu/anaconda3/lib/python3.8/site-packages/dockerspawner/dockerspawner.py", line 882, in volume_binds
binds = self._volumes_to_binds(self.volumes, {})
File "/home/ubuntu/anaconda3/lib/python3.8/site-packages/dockerspawner/dockerspawner.py", line 1456, in _volumes_to_binds
binds[_fmt(k)] = {"bind": _fmt(v), "mode": m}
File "/home/ubuntu/anaconda3/lib/python3.8/site-packages/dockerspawner/dockerspawner.py", line 1448, in _fmt
return self.format_volume_name(v, self)
File "/home/ubuntu/anaconda3/lib/python3.8/site-packages/dockerspawner/volumenamingstrategy.py", line 2, in default_format_volume_name
return template.format(**spawner.template_namespace())
IndexError: Replacement index 0 out of range for positional args tuple
[I 2024-05-30 18:06:06.342 JupyterHub dockerspawner:1033] Container 'jupyter-yzuriaw' is gone
[W 2024-05-30 18:06:06.342 JupyterHub dockerspawner:1008] Container not found: jupyter-yzuriaw
[E 2024-05-30 18:06:06.384 JupyterHub pages:313] Error starting server yzuriaw: Replacement index 0 out of range for positional args tuple
Traceback (most recent call last):
None: None
[W 2024-05-30 18:06:06.384 JupyterHub web:1787] 500 GET /hub/spawn (::ffff:73.201.170.13): Unhandled error starting server yzuriaw
[E 2024-05-30 18:06:06.420 JupyterHub log:183] {
"X-Forwarded-Host": "23.22.250.249:8000",
"X-Forwarded-Proto": "http",
"X-Forwarded-Port": "8000",
"X-Forwarded-For": "::ffff:73.201.170.13",
"Cookie": "jupyterhub-hub-login=[secret]; _xsrf=[secret]",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate",
"Referer": "http://23.22.250.249:8000/hub/admin",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0",
"Dnt": "1",
"Upgrade-Insecure-Requests": "1",
"Connection": "keep-alive",
"Host": "23.22.250.249:8000"
}
[E 2024-05-30 18:06:06.420 JupyterHub log:191] 500 GET /hub/spawn (yzuriaw@::ffff:73.201.170.13) 165.78ms
My dockerfile:
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
FROM jupyter/minimal-notebook
LABEL maintainer="Jupyter Project <jupyter@googlegroups.com>"
ARG JUPYTERLAB_VERSION=4.2.1
ARG JUPYTERHUB_VERSION=4.0.2
USER root
RUN apt-get update --yes && \
apt-get install --yes --no-install-recommends ffmpeg dvipng cm-super && \
apt-get clean && rm -rf /var/lib/apt/lists/*
RUN apt update
RUN apt install -y --fix-missing \
chromium-chromedriver \
git \
nodejs \
npm \
wget \
curl \
pkg-config \
&& apt clean && rm -rf /tmp/* /var/tmp/*
USER ${NB_UID}
RUN pip3 install --no-cache \
wbdata \
socrata-py \
sodapy \
faodata \
notebook==7.2.0 \
jupyterlab==$JUPYTERLAB_VERSION \
jupyterhub==$JUPYTERHUB_VERSION
RUN pip3 install --no-cache \
wheel \
reportlab \
colour \
pyproj \
holoviews \
country_converter \
lxml \
pyxlsb \
python-pptx \
openpyxl \
shapely \
geopandas \
jupyterlab_iframe \
pandas \
selenium \
seaborn \
openai \
pypdf \
xlsxwriter
RUN conda install --quiet --yes -c conda-forge \
jupyterlab_pygments \
altair \
bottleneck \
cloudpickle \
cython \
dask \
dill \
h5py \
ipympl \
ipywidgets \
matplotlib-base \
numba \
numexpr \
patsy \
protobuf \
pytables \
scikit-image \
scikit-learn \
scipy \
statsmodels \
sympy \
widgetsnbextension \
xlrd && \
conda clean --all -f -y && \
fix-permissions "${CONDA_DIR}" && \
fix-permissions "/home/${NB_USER}"
RUN pip3 install jupyterlab_git \
lckr_jupyterlab_variableinspector \
jupyterlab-spellchecker \
jupyter-ai \
jupyter-server-proxy
RUN pip install --upgrade jupyter-server
USER ${NB_UID}
CMD ["jupyterhub-singleuser"]
ENV XDG_CACHE_HOME="/home/${NB_USER}/.cache/"
RUN MPLBACKEND=Agg python -c "import matplotlib.pyplot" && \
fix-permissions "/home/${NB_USER}"
USER ${NB_UID}
WORKDIR "${HOME}"
And my config file:
import os
import pandas as pd
import glob
# configure GitHub Authentication
from oauthenticator.github import GitHubOAuthenticator
c.JupyterHub.authenticator_class = GitHubOAuthenticator
users = pd.read_csv('/mnt/c/Work/JupyterHub/test/GitHub_Authentication.csv')
names = users['github_name'].values
volume_name = '/storage/volume_nyc3_02'
# generate list of whitelisted users and administrators
c.Authenticator.allowed_users = set(names)
# set of users who can administer the hub
c.Authenticator.admin_users = {'xxxxxxx', 'xxxxxxxx'}
# generate authentication options (change the 'My' to Github
c.GitHubOAuthenticator.oauth_callback_url = 'xxxxxxxxxxxxxxxxxxxxx'
c.GitHubOAuthenticator.client_id = 'xxxxxxxxxxxxxxxxxxxxx'
c.GitHubOAuthenticator.client_secret = 'xxxxxxxxxxxxxxxxxxxxx'
#****************************************************************************************
#****************************************************************************************
#****************************************************************************************
# set docker as the spawner class
import dockerspawner
def find_teams(df):
'''Generate dictionary with list of types as values'''
def clean(string):
s = string.split(',')
return [val.strip() for val in s]
return {name: clean(teams) for name, teams in zip(df['github_name'], df['groups'])}
def set_esdb_dirs(direc, name):
'''Given a dictionary of jupyter folder paths, add the esdb source folders in a useful format.'''
# set the database to the usual structure
direc['{}/Task-8/esdb-database/'.format(volume_name)] = '/home/jovyan/Home/esdb-database'
# get a list of the folders - sources
def clean(string):
return string.rsplit('/',1)[1]
folders = [clean(string) for string in glob.glob('{}/Task-8/esdb-source-data/*'.format(volume_name))]
folders = folders
for d in folders:
direc['/volumes/Users/{}/esdb-source-programs/'.format(name)] = '/home/jovyan/Home/esdb-source-programs/'
for folder in ['/Raw data','/Edited data','/Final data', '/Mapping']:
direc['{}/Task-8/esdb-source-data/{}{}/'.format(volume_name, d, folder)] = '/home/jovyan/Home/esdb-source-programs/{}{}'.format(d, folder)
direc['{}/Task-8/esdb-archive/{}/'.format(volume_name, d)] = '/home/jovyan/Home/esdb-source-programs/{}/z_archive'.format(d)
return direc
def set_hub_directory(name):
"""Given a username, define the folders the users can see."""
directory = {
# define the default user home dir
'/volumes/Users/{}/'.format(name): '/home/jovyan/Home'
""",
'{}/Users/{}'.format(volume_name, name): '/home/jovyan/Home/Storage',
# define the default DevTech-Shared folder
'/volumes/DevTech-Shared/': '/home/jovyan/DevTech-Shared',
'/volumes/Packages/': '/home/jovyan/DevTech-Shared/Packages',
'{}/DevTech-Shared/'.format(volume_name): '/home/jovyan/DevTech-Shared/Storage',
# set fonts for use
'/usr/share/fonts/abel': '/usr/share/fonts/abel'
"""
}
# for each team the user is a part of, add the teams folder to the volume path.
for team in teams[name]:
directory['/volumes/{}/'.format(team)] = '/home/jovyan/{}'.format(team)
# directory['{}/{}/'.format(volume_name, team)] = '/home/jovyan/{}/Storage'.format(team)
if team == 'Task-8':
directory = set_esdb_dirs(directory, name)
return directory
'''
load data
'''
teams = find_teams(users)
team_options = list(users['groups'].unique())
'''
Create new DockerSpawner object
'''
# Spawn single-user servers as Docker containers
class DockerSpawner(dockerspawner.DockerSpawner):
'''add an additional functions to the dockerspawner class which specifies the directories'''
def start(self):
# username is self.user.name
self.volumes = set_hub_directory(self.user.name)
return super().start()
c.JupyterHub.spawner_class = DockerSpawner
# tell docker where to look for ip addresses
import netifaces
docker0 = netifaces.ifaddresses('docker0') # netifaces.ifaddresses('docker0')
docker0_ipv4 = docker0[netifaces.AF_INET][0]
c.JupyterHub.hub_ip = docker0_ipv4['addr']
c.DockerSpawner.network_name = 'bridge'
# set image to custom image
c.DockerSpawner.image = 'advise-image-2.0:1'
c.DockerSpawner.use_internal_ip = True
c.DockerSpawner.remove = True
c.DockerSpawner.debug = True
'''
JupyterLab Settings
'''
# configure to run on jupterlab
c.Spawner.cmd = ['jupyter-labhub']
Been stuck on this for days, so any and all help is appreciated!