Jupyterhub 500 : Internal Server Error

Hi team,

In our HDP cluster the jupyterhub was working fine. But due to some package uninstallation, we installed Jhub and other packages again. But now it is not working. I am getting below error after I login to Jhub UI

500 : Internal Server Error

"Failed to start your server on the last attempt. Please contact admin if the issue persists.

You can try restarting your server from the home page."

Below is the error trace from jupyterhub
Error trace:
‘’’
[E 2021-06-01 22:16:02.266 JupyterHub web:1670] Uncaught exception GET /hub/user// (::ffff:)
HTTPServerRequest(protocol=‘https’, host=’:8000’, method=‘GET’, uri=’/hub/user//’, version=‘HTTP/1.1’, remote_ip=’::ffff:’)
Traceback (most recent call last):
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/tornado/web.py”, line 1592, in _execute
result = yield result
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/jupyterhub/handlers/base.py”, line 1052, in get
await self.spawn_single_user(user)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/jupyterhub/handlers/base.py”, line 705, in spawn_single_user
timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/jupyterhub/handlers/base.py”, line 626, in finish_user_spawn
await spawn_future
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/jupyterhub/user.py”, line 489, in spawn
raise e
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/jupyterhub/user.py”, line 409, in spawn
url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/jupyterhub/spawner.py”, line 1110, in start
self.proc = Popen(cmd, **popen_kwargs)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/subprocess.py”, line 729, in init
restore_signals, start_new_session)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/subprocess.py”, line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: ‘/usr/share/miniconda2/envs/py36/bin/jupyterhub-singleuser’

[E 2021-06-01 22:16:02.268 JupyterHub log:150] {
“X-Forwarded-Host”: “:8000”,
“X-Forwarded-Proto”: “https”,
“X-Forwarded-Port”: “8000”,
“X-Forwarded-For”: “::ffff:10.216.59.96”,
“Cookie”: “jupyterhub-hub-login=“2|1:0|10:1622554098|20:jupyterhub-hub-login|xx”; jupyterhub-session-id=xx”,
“Accept-Language”: “en-US,en;q=0.9”,
“Accept-Encoding”: “gzip, deflate, br”,
“Referer”: “https://:8000/hub/home”,
“Sec-Ch-Ua-Mobile”: “?0”,
“Sec-Ch-Ua”: “” Not A;Brand";v=“99”, “Chromium”;v=“90”, “Google Chrome”;v=“90"”,
“Sec-Fetch-Dest”: “document”,
“Sec-Fetch-User”: “?1”,
“Sec-Fetch-Mode”: “navigate”,
“Sec-Fetch-Site”: “same-origin”,
“Accept”: “text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9”,
“User-Agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36”,
“Upgrade-Insecure-Requests”: “1”,
“Connection”: “close”,
“Host”: “:8000”
}
[E 2021-06-01 22:16:02.268 JupyterHub log:158] 500 GET /hub/user// (@::ffff:) 88.81ms
‘’’
I updated 755 chmod to “/usr/share/miniconda2/envs/py36/bin/jupyterhub-singleuser” and also the kernel speck with proper values.

Kindly help with this issue. I am stuck

Thanks

Hi!

Please could you tell us exactly what you did?

Can you show us the full permissions on this file (ls -l)? It may be easiest to delete the environment and create a new one.

Hi @manics Manics,
I removed the conda environment and created same environment.

This is the command that I used
sudo /usr/share/miniconda2/bin/conda create -n py36 python=3.6 anaconda

And then I had previous working version of pip list. I used the same list and installed all the packages.
I installed all the packages from the previous working pip list. That includes the packages that are installed when I created the environment. Is that OK or I should exclude them?

Permission: This same permission works fine in other environments.
ls -l /usr/share/miniconda2/envs/py36/bin/jupyterhub-singleuser
-rwxr-xr-x 1 root root 124 Jun 1 20:21 /usr/share/miniconda2/envs/py36/bin/jupyterhub-singleuser

Note: Previous admin advised to use jupyterhub-ldapauthenticator==1.1 but if i use this version when I start jupyterhub I am getting this below error. If I use 1.2.2 I am not getting this error and Jhub is starting but getting the above mentioned 500 internal server issue

Error Trace
[W 2021-06-01 23:18:41.821 JupyterHub configurable:168] Config option delete_invalid_users not recognized by LDAPAuthenticator.
[E 2021-06-01 23:18:41.821 JupyterHub app:1958]
Traceback (most recent call last):
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/jupyterhub/app.py”, line 1955, in launch_instance_async
await self.initialize(argv)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/jupyterhub/app.py”, line 1684, in initialize
await self.init_users()
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/jupyterhub/app.py”, line 1148, in init_users
if self.authenticator.enable_auth_state:
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/traitlets.py”, line 556, in get
return self.get(obj, cls)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/traitlets.py”, line 535, in get
value = self._validate(obj, dynamic_default())
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/jupyterhub/app.py”, line 675, in _authenticator_default
return self.authenticator_class(parent=self, db=self.db)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/config/configurable.py”, line 84, in init
self.config = config
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/traitlets.py”, line 585, in set
self.set(obj, value)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/traitlets.py”, line 574, in set
obj._notify_trait(self.name, old_value, new_value)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/traitlets.py”, line 1139, in _notify_trait
type=‘change’,
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/traitlets.py”, line 1176, in notify_change
c(change)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/traitlets.py”, line 819, in compatible_observer
return func(self, change)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/config/configurable.py”, line 186, in _config_changed
self._load_config(change.new, traits=traits, section_names=section_names)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/config/configurable.py”, line 153, in _load_config
setattr(self, name, deepcopy(config_value))
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/traitlets.py”, line 585, in set
self.set(obj, value)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/traitlets.py”, line 559, in set
new_value = self._validate(obj, value)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/traitlets.py”, line 591, in _validate
value = self.validate(obj, value)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/traitlets.py”, line 2054, in validate
self.error(obj, value)
File “/usr/share/miniconda2/envs/py36/lib/python3.6/site-packages/traitlets/traitlets.py”, line 625, in error
raise TraitError(e)
traitlets.traitlets.TraitError: The ‘bind_dn_template’ trait of a LDAPAuthenticator instance must be a unicode string, but a value of [‘string value’] <class ‘list’> was specified.

Attaching my current pip list
absl-py 0.11.0
alabaster 0.7.12
alembic 1.0.7
amply 0.1.4
anaconda-client 1.7.2
anaconda-project 0.10.0
anyio 2.2.0
appdirs 1.4.4
argh 0.26.2
argon2-cffi 20.1.0
arviz 0.10.0
ASLPAw 2.0.0
asn1crypto 0.24.0
astor 0.8.1
astroid 2.1.0
astropy 3.1
astunparse 1.6.3
async-generator 1.10
atomicwrites 1.2.1
attrs 21.2.0
autopep8 1.5.4
Babel 2.6.0
backcall 0.1.0
backports.os 0.1.1
backports.shutil-get-terminal-size 1.0.0
beautifulsoup4 4.6.3
bimlpa 0.1.2
bitarray 0.8.3
bkcharts 0.2
black 19.10b0
blaze 0.10.1
bleach 3.2.1
blis 0.4.1
bokeh 2.3.0
boto 2.49.0
boto3 1.14.17
botocore 1.17.17
Bottleneck 1.2.1
Brotli 1.0.9
brotlipy 0.7.0
bson 0.5.8
CacheControl 0.12.6
cached-property 1.5.2
cachetools 4.2.0
cachy 0.3.0
catalogue 1.0.0
catboost 0.13.1
cdlib 0.2.0
certifi 2018.11.29
cffi 1.11.5
cftime 1.3.0
chardet 3.0.4
chinese-whispers 0.7.4
cleo 0.8.1
Click 7.0
clikit 0.6.2
cloudera-director-python-client 6.3.0
cloudpickle 1.6.0
clyent 1.2.2
colorama 0.4.1
colorcet 2.0.6
conda 4.3.16
conda-build 2.1.5
conda-pack 0.6.0
contextlib2 0.5.5
contextvars 2.4
count-dict 1.0.2
crashtest 0.3.1
cryptography 2.4.2
cursor 1.3.4
cvae 0.0.3
cycler 0.10.0
cymem 2.0.3
Cython 0.29.2
cytoolz 0.9.0.1
dash 0.39.0
dash-core-components 0.44.0
dash-cytoscape 0.1.1
dash-html-components 0.14.0
dash-renderer 0.20.0
dash-table 3.6.0
dash-table-experiments 0.6.0
dask 2.30.0
dask-glm 0.2.0
dask-lightgbm 0.1.0
dask-ml 1.7.0
dask-xgboost 0.1.11
dataclasses 0.7
datashader 0.12.1
datashape 0.5.2
decorator 4.3.0
defusedxml 0.5.0
demon 2.0.5
dgl 0.6.1
diff-match-patch 20200713
distlib 0.3.1
distributed 2.30.1
docopt 0.6.2
docutils 0.14
dynetx 0.2.4
entrypoints 0.3
enum34 1.1.6
et-xmlfile 1.0.1
eva-lcd 0.1.0
factor-analyzer 0.3.1
fastai 2.0.7
fastcache 1.0.2
fastcore 1.0.0
fastprogress 1.0.0
filelock 3.0.10
flake8 3.9.0
Flask 1.0.2
Flask-Compress 1.4.0
Flask-Cors 3.0.9
flatbuffers 1.12
fsspec 0.8.4
funcy 1.14
future 0.18.2
fuzzywuzzy 0.17.0
gast 0.3.3
gensim 3.8.3
gevent 1.4.0
glob2 0.6
gmpy2 2.0.8
google-auth 1.23.0
google-auth-oauthlib 0.4.2
google-pasta 0.2.0
graphviz 0.14.1
greenlet 0.4.15
grpcio 1.32.0
gssapi 1.6.1
h5py 2.10.0
HeapDict 1.0.0
holoviews 1.14.2
html5lib 1.0.1
icc-rt 2020.0.133
idna 2.8
imageio 2.4.1
imagesize 1.1.0
imbalanced-learn 0.7.0
immutables 0.14
implicit 0.4.2
importlib-metadata 1.7.0
importlib-resources 3.3.0
impyla 0.16.2
iniconfig 1.1.1
install 1.3.3
intel-openmp 2020.0.133
interpret 0.1.8
intervaltree 3.1.0
ipykernel 5.1.1
ipython 7.5.0
ipython-genutils 0.2.0
ipywidgets 7.4.2
isort 4.3.4
itsdangerous 1.1.0
JayDeBeApi 1.2.3
jdcal 1.4
jedi 0.13.2
jeepney 0.6.0
jenkspy 0.1.5
Jinja2 2.11.2
jmespath 0.10.0
joblib 0.13.2
JPype1 1.2.1
json5 0.9.5
jsonschema 2.6.0
jupyter 1.0.0
jupyter-client 5.2.4
jupyter-console 6.0.0
jupyter-contrib-core 0.3.3
jupyter-contrib-nbextensions 0.5.1
jupyter-core 4.4.0
jupyter-highlight-selected-word 0.2.0
jupyter-latex-envs 1.4.6
jupyter-nbextensions-configurator 0.4.1
jupyter-packaging 0.7.12
jupyter-server 0.0.5
jupyterhub 0.9.4
jupyterhub-ldapauthenticator 1.1
jupyterlab 0.35.3
jupyterlab-pygments 0.1.2
jupyterlab-server 0.2.0
jupyterlab-widgets 1.0.0
kafka-python 1.4.6
karateclub 1.0.24
Keras-Applications 1.0.8
keras-nightly 2.5.0.dev2021032900
Keras-Preprocessing 1.1.2
keyring 21.5.0
kiwisolver 1.0.1
koalas 1.3.0
lazy-object-proxy 1.3.1
ldap3 2.5.2
leidenalg 0.8.4
libarchive-c 2.9
Library 0.0.0
lief 0.9.0
lightgbm 2.3.1
lime 0.1.1.34
llvmlite 0.36.0
locket 0.2.0
lockfile 0.12.2
lxml 4.2.5
Mako 1.0.7
Markdown 3.3.3
markov-clustering 0.0.6.dev0
MarkupSafe 1.1.0
matplotlib 3.0.2
mccabe 0.6.1
mecab-python3 0.996.2
mistune 0.8.4
mkl 2019.0
mkl-fft 1.0.6
mkl-random 1.1.1
mkl-service 2.3.0
mlens 0.2.3
mock 4.0.3
more-itertools 4.3.0
mpmath 1.1.0
msgpack 1.0.0
multipledispatch 0.6.0
multivalued-dict 1.7.1
murmurhash 1.0.2
mypy-extensions 0.4.3
nbclassic 0.2.6
nbclient 0.5.3
nbconvert 5.5.0
nbformat 4.4.0
neo4j 4.2.1
neo4j-driver 1.6.3
neobolt 1.7.13
neomodel 3.3.1
neotime 1.7.4
nest-asyncio 1.5.1
netCDF4 1.5.4
networkx 2.5.1
nf1 0.0.3
nltk 3.4
nose 1.3.7
notebook 5.7.8
numba 0.53.1
numexpr 2.6.8
numpy 1.19.4
numpydoc 0.8.0
oauthlib 3.1.0
odo 0.5.0
olefile 0.46
omega-index-py3 0.3
openpyxl 2.5.12
opt-einsum 3.3.0
packaging 20.7
pamela 1.0.0
pandas 1.1.0
pandocfilters 1.4.2
panel 0.11.1
param 1.10.1
parso 0.8.0
partd 1.1.0
pastel 0.2.1
path 15.1.2
path.py 11.5.0
pathlib2 2.3.3
pathspec 0.7.0
pathtools 0.1.2
patsy 0.5.1
pep8 1.7.1
pexpect 4.8.0
pickleshare 0.7.5
Pillow 8.0.1
pip 21.1.1
pipdeptree 1.0.0
pkginfo 1.4.2
plac 1.1.3
plotly 3.10.0
pluggy 0.12.0
ply 3.11
poetry-core 1.0.0
pooch 1.3.0
pquality 0.0.7
preshed 3.0.2
prometheus-client 0.5.0
prompt-toolkit 2.0.7
protobuf 3.14.0
psutil 5.6.2
ptyprocess 0.6.0
PuLP 2.4
py 1.7.0
py2neo 4.3.0
py4j 0.10.8.1
pyarrow 3.0.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycodestyle 2.6.0
pycosat 0.6.3
pycparser 2.19
pycrypto 2.6.1
pyct 0.4.8
pycurl 7.43.0.6
pydocstyle 6.1.1
pyflakes 2.0.0
Pygments 2.3.1
PyGSP 0.5.1
PyHive 0.6.3
pyhs2 0.6.0
pyLDAvis 2.1.2
pylev 1.3.0
pylint 2.2.2
pyls-black 0.4.6
pyls-spyder 0.3.2
pynndescent 0.5.2
pyodbc 4.0.25
pyOpenSSL 18.0.0
pyparsing 2.3.0
PyQt5 5.9.2
pyrsistent 0.17.3
PyScaffold 3.0.3
PySocks 1.6.8
pystan 2.18.0.0
pytest 4.6.2
pytest-arraydiff 0.3
pytest-astropy 0.5.0
pytest-doctestplus 0.2.0
pytest-openfiles 0.3.1
pytest-remotedata 0.3.1
pytest-runner 5.1
python-dateutil 2.7.5
python-editor 1.0.4
python-igraph 0.9.1
python-jsonrpc-server 0.4.0
python-language-server 0.36.2
python-Levenshtein 0.12.0
python-louvain 0.14
python-oauth2 1.1.0
pytz 2021.1
pyviz-comms 2.0.1
PyWavelets 1.1.1
pyxdg 0.27
PyYAML 5.3.1
pyzmq 17.1.2
QDarkStyle 2.8.1
QtAwesome 0.5.3
qtconsole 4.4.3
QtPy 1.5.2
regex 2021.4.4
requests 2.24.0
requests-oauthlib 1.3.0
requests-toolbelt 0.9.1
retrying 1.3.3
rope 0.11.0
rsa 4.6
Rtree 0.9.7
ruamel-yaml-conda 0.15.100
ruamel.yaml 0.15.46
ruamel.yaml.clib 0.2.2
s3transfer 0.3.3
SALib 1.3.7
sasl 0.2.1
scikeras 0.2.1
scikit-image 0.17.2
scikit-learn 0.23.2
scikit-optimize 0.5.2
scikit-plot 0.3.7
scipy 1.4.1
scorch 0.0.19
seaborn 0.10.0
SecretStorage 3.3.0
Send2Trash 1.5.0
setuptools 54.2.0
shap 0.28.5
shellescape 3.8.1
shellingham 1.3.2
shuffle-graph 1.1.1
simplegeneric 0.8.1
singledispatch 3.4.0.3
sip 4.19.8
six 1.15.0
skope-rules 1.0.0
slicer 0.0.7
smart-open 2.1.0
sniffio 1.2.0
snowballstemmer 1.2.1
sortedcollections 1.0.1
sortedcontainers 2.1.0
soupsieve 2.2.1
spacy 2.3.0
Sphinx 1.8.2
sphinxcontrib-applehelp 1.0.2
sphinxcontrib-devhelp 1.0.2
sphinxcontrib-htmlhelp 1.0.3
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.3
sphinxcontrib-serializinghtml 1.1.4
sphinxcontrib-websupport 1.1.0
spyder 3.3.2
spyder-kernels 0.3.0
SQLAlchemy 1.3.0
srsly 1.0.2
statsmodels 0.10.0rc2
stellargraph 1.2.1
sympy 1.3
tables 3.4.4
tbb 2021.2.0
tbb4py 2021.2.0
tblib 1.7.0
tensorboard 2.4.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.7.0
tensorflow 2.4.0
tensorflow-estimator 2.4.0
termcolor 1.1.0
terminado 0.8.1
testpath 0.4.2
textdistance 4.2.1
texttable 1.6.3
thinc 7.4.1
threadpoolctl 2.1.0
three-merge 0.1.1
thrift 0.13.0
thrift-sasl 0.4.2
thriftpy2 0.4.11
tifffile 2020.9.3
toml 0.10.2
tomlkit 0.7.0
toolz 0.10.0
torch 1.6.0
torchvision 0.7.0
tornado 5.0.2
tqdm 4.47.0
traitlets 4.3.2
typed-ast 1.1.0
typing-extensions 3.7.4.3
ujson 4.0.2
umap-learn 0.5.1
unicodecsv 0.14.1
urllib3 1.24.3
varclushi 0.1.0
virtualenv 20.2.1
virtualenv-clone 0.5.4
wasabi 0.7.0
watchdog 1.0.2
wcwidth 0.1.7
webencodings 0.5.1
Werkzeug 1.0.1
wheel 0.36.2
widgetsnbextension 3.4.2
wrapt 1.12.1
wurlitzer 1.0.2
xarray 0.16.1
xgboost 0.82
xlrd 1.2.0
XlsxWriter 1.1.2
xlwt 1.3.0
yapf 0.31.0
zict 0.1.3
zipp 0.5.1
zope.event 4.5.0
zope.interface 5.3.0

What user are you running JupyterHub as, how are you running it (manually? as a service?) and what’s the system user who’s singleuser server isn’t launching? If you switch to that user using su <singleuser-user> are you able to run /usr/share/miniconda2/envs/py36/bin/jupyterhub-singleuser?

I’ve noticed that I’m getting a 500 server error on a standalone jupyter notebook server in a Docker container that installs ou-tm351-py · PyPI

There’s some package in there that seems to break the notebook server. I’d try to chase it further but it’s been a long day and too early hours of the morning to spend any more time on it tonight…

UPDATE: it just seems to be classic notebook that gives the 500 in my case. JupyterLab still seems to work. I’m mended to treat this as a weak signal that classic nb rot may be starting to set in… UPDATE UPDATE: found my error, I think - jupyter-docx-bundler; I guess what I’m wary of there is a wide ecosystem of packages out there and other things may start to break classic notebook…