Failed to start the single-user server in JupyterHub

Hi, I recently encountered an error indicating spawn failure due to the python module issue. From the error log it suggested /opt/conda/bin/jupyterhub-singleuser is trying to load the ‘alembic’ package from /home/jovyan/.local instead of from /opt/conda/lib.

What is strange to me is that all the other people can successfully launch their server except that specific user - seem all the other users have no ‘alembic’ package installed under their home directory at all (/home/jovyan). I believe this error might be caused by the incorrect version of ‘alembic’ under /home/jovyan.

I further examine ‘sys.path’ under a successfully spawned server and it showed the following:

import sys
[‘’, ‘/opt/conda/lib/’, ‘/opt/conda/lib/python3.8’, ‘/opt/conda/lib/python3.8/lib-dynload’, ‘/home/jovyan/.local/lib/python3.8/site-packages’, ‘/home/jovyan/.local/lib/python3.8/site-packages/Swiss_Re_Analytics-0.0.5-py3.8.egg’, ‘/opt/conda/lib/python3.8/site-packages’]

From the output I can see /home/jovyan has higher priority over /opt/conda in terms of module search order. I believe the jupyterhub-singleuser process is supposed to load ‘alembic’ from /opt/conda instead of from /home/jovyan.

Now the question is how can I change the order in sys.path? I’ve tried to modify sys.path in the config.yaml file (leveraged by Helm) but got no luck.

P.S. The platform is deployed on a K8S cluster and releases are managed by Helm.

Traceback (most recent call last):
File “/opt/conda/bin/jupyterhub-singleuser”, line 7, in
from jupyterhub.singleuser import main
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser/”, line 5, in
from .app import main
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser/”, line 10, in
from .mixins import make_singleuser_app
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/singleuser/”, line 43, in
from …log import log_request
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/”, line 15, in
from .handlers.pages import HealthCheckHandler
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/handlers/”, line 1, in
from . import base
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/handlers/”, line 32, in
from … import orm
File “/opt/conda/lib/python3.8/site-packages/jupyterhub/”, line 11, in
import alembic.command
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/”, line 3, in
from . import context # noqa
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/”, line 1, in
from .runtime.environment import EnvironmentContext
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/runtime/”, line 1, in
from .migration import MigrationContext
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/runtime/”, line 15, in
from … import ddl
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/ddl/”, line 1, in
from . import mssql # noqa
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/ddl/”, line 8, in
from .base import AddColumn
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/ddl/”, line 11, in
from …util import sqla_compat
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/util/”, line 26, in
from .sqla_compat import has_computed # noqa
File “/home/jovyan/.local/lib/python3.8/site-packages/alembic/util/”, line 12, in
from sqlalchemy.sql.expression import _BindParamClause
ImportError: cannot import name ‘_BindParamClause’ from ‘sqlalchemy.sql.expression’ (/home/jovyan/.local/lib/python

Not sure if my assumption is right but please let me know if you have any other thoughts. Thank you.

I resolved this issue by upgrading the ‘alembic’ package to the latest version. Just let you aware that I manually attached the cloud disk, where the package was installed, to my own Pod - in this way the upgrade can be conducted there.