received signal 15, stopping
SingleUserLabApp notebookapp:2052] Shutting down 0 kernels
SingleUserLabApp notebookapp:2067] Shutting down 0 terminals
restarting the user’s container and the spawn process shut it down again.
Thanks.
You might try updating JupyterHub. 1.1 is a long time ago (current is 2.2.2, but you can use 1.5.x).
It’s important to include logs from the hub surrounding events in order to track down errors. Make sure to include:
conda env export or pip freeze)jupyterhub_config.py (with sensitive info redacted)I tried to upgrade but it broke jupyterhub, and reported:FileNotFoundError: [Errno 2] No such file or directory: ‘alembic’: ‘alembic’. not sure if it is compatible with the enviroment.
Jupyterhub logs:
302 GET / → /hub/
302 GET /hub/ → /hub/spawn
when spawn from jupyterhub restart
docker logs:
received signal 15, stopping
Shutting down 0 kernels
Shutting down 0 terminals
python3 -m pip freeze
alembic==1.4.3
argon2-cffi==20.1.0
asn1crypto==0.24.0
async-generator==1.10
attrs==20.2.0
backcall==0.2.0
beautifulsoup4==4.6.3
bleach==3.2.1
certifi==2020.6.20
certipy==0.1.3
cffi==1.14.3
chardet==3.0.4
configobj==5.0.6
cryptography==3.1.1
cssselect==0.9.2
dbus-python==1.2.4
decorator==4.4.2
defusedxml==0.6.0
docker==4.3.1
dockerspawner==0.11.1
entrypoints==0.3
escapism==1.0.1
ethtool==0.14
future==0.16.0
gpg==1.13.1
html5lib==0.999999999
idna==2.10
importlib-metadata==2.0.0
iniparse==0.4
ipykernel==5.3.4
ipython==7.16.1
ipython-genutils==0.2.0
isc==2.0
jedi==0.17.2
Jinja2==2.11.2
jsonschema==3.2.0
jupyter-client==6.1.7
jupyter-core==4.6.3
jupyter-telemetry==0.1.0
jupyterhub==1.1.0
jupyterlab-pygments==0.1.1
libcomps==0.1.16
lxml==4.2.3
Mako==1.1.3
MarkupSafe==1.1.1
mistune==0.8.4
nbclient==0.5.0
nbconvert==6.0.5
nbformat==5.0.7
nest-asyncio==1.4.0
netifaces==0.10.6
nftables==0.1
notebook==6.1.4
oauthlib==3.1.0
packaging==20.4
pamela==1.0.0
pandocfilters==1.4.2
parso==0.7.1
pciutils==2.3.6
perf==0.1
pexpect==4.8.0
pickleshare==0.7.5
ply==3.9
prometheus-client==0.8.0
prompt-toolkit==3.0.7
psutil==5.7.2
psycopg2==2.8.6
ptyprocess==0.6.0
pycairo==1.16.3
pycparser==2.20
pycurl==7.43.0.6
pydbus==0.6.0
Pygments==2.7.1
PyGObject==3.28.3
pyinotify==0.9.6
pyOpenSSL==19.1.0
pyparsing==2.1.10
pyrsistent==0.17.3
PySocks==1.6.8
python-dateutil==2.8.1
python-dmidecode==3.12.2
python-editor==1.0.4
python-json-logger==2.0.0
python-linux-procfs==0.6.3
python-qpid-proton==0.28.0
pyudev==0.21.0
PyYAML==3.12
pyzmq==19.0.2
requests==2.24.0
rhnlib==2.8.6
rpm==4.14.3
ruamel.yaml==0.16.12
ruamel.yaml.clib==0.2.2
schedutils==0.6
selinux==2.9
Send2Trash==1.5.0
sepolicy==1.1
setools==4.3.0
setroubleshoot==1.1
six==1.15.0
slip==0.6.4
slip.dbus==0.6.4
SQLAlchemy==1.3.19
SSSDConfig==2.5.2
subscription-manager==1.28.21
syspurpose==1.28.21
systemd-python==234
terminado==0.9.1
testpath==0.4.4
tornado==6.0.4
traitlets==4.3.3
urllib3==1.25.10
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==0.57.0
zipp==3.2.0
jupyterhub_config.py
# Configuration file for jupyterhub.
import os
from jupyterhub.utils import random_port
from dockerspawner import DockerSpawner
class custom_spawner(DockerSpawner):
@property
def internal_hostname (self):
# Set FQDN or localhost
return 'localhost’
def _port_default (self):
#Do NOT set c.DockerSpawner.port in the config file
return random_port()
c.JupyterHub.bind_url = 'http://127.0.0.1:8000’
# DockerSpawner chosen as spawner for good isolation between users, particularly filesystem isolation
#c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner’
c.JupyterHub.spawner_class = custom_spawner
# Running Docker Stacks scipy-notebook image (hash specified for version known compatible
# with this jupyterhub install)
# Selecting an Image — Docker Stacks documentation
#c.DockerSpawner.image = 'jupyter/scipy-notebook:612aa5710bf9’
#c.DockerSpawner.image = 'jupyter/tensorflow-notebook:feacdbfc2e89’
c.DockerSpawner.image = '###'
# The docker instances need access to the Hub and can’t use the default loopback address - they
# need to use the public IP address of the Hub. A firewall rule is in place to allow access
# through the default port 8081.
from jupyter_client.localinterfaces import public_ips
c.JupyterHub.hub_ip = public_ips()[0]
# Explicitly set notebook directory because we’ll be mounting a host volume to
*# it. Most jupyter/docker-stacks -notebook images run the Notebook server as
# user jovyan, and set the notebook directory to /home/jovyan/work.
notebook_dir = os.environ.get( ‘DOCKER_NOTEBOOK_DIR’ ) or '/home/jovyan/work’
c.DockerSpawner.notebook_dir = notebook_dir
# Use a previously created docker network bridge
# network_name = 'jupyter’
# c.DockerSpawner.network_name = network_name
# c.DockerSpawner.extra_host_config = {‘network_mode’: network_name }
c.DockerSpawner.extra_host_config = { ‘network_mode’ : ‘host’ }
c.DockerSpawner.use_internal_hostname = True
c.DockerSpawner.network_name = 'host’
c.DockerSpawner.host_ip = '###'
# Using docker volume for persistance of user data. Mount the real user’s Docker
# volume on the host to the notebook user’s notebook directory in the container
c.DockerSpawner.volumes = { ‘jupyterhub-user-{username}’ : notebook_dir }
# And use a docker bind mount to get readonly access to course material in host
# directory.
c.DockerSpawner.read_only_volumes = { ‘/home/users/files’ : notebook_dir + “/files” }
# Set a user memory limit
c.DockerSpawner.mem_limit = '2G’
# be provided by jupyterhub database.
c.Authenticator.admin_users = set()
c.Authenticator.admin_users.add( ‘###’ )
c.Authenticator.whitelist = set()
c.Authenticator.whitelist.add( ‘###’ )
# Allow admin users to access other users’ notebooks
c.JupyterHub.admin_access = True
c.ConfigurableHTTPProxy.auth_token = '###'
c.JupyterHub.cookie_secret = b '###'
# We use a modified pam config which omits pam_access access list check - that list is used to
# restrict non-jupyterhub logins to this server, and the jupyterhub whitelist is used for hub
# logins.
c.PAMAuthenticator.service = 'jupyterhub’
#pg_pass = os.getenv(‘POSTGRES_ENV_JPY_PSQL_PASSWORD’)
pg_pass = '###'
#pg_host = os.getenv(‘POSTGRES_PORT_5432_TCP_ADDR’)
pg_host = '127.0.0.1’
c.JupyterHub.db_url = ‘postgresql://jupyterhub:{}@{}:5432/jupyterhub’ .format(
pg_pass, pg_host
)
c.Spawner.cmd=[ “jupyter-labhub” ]
#c.Spawner.http_timeout = 60
#c.Spawner.start_timeout = 60
I have the same issue on jupyterhub version 2.2.2. For me the default image of jupyterhub/singleuser:2.2 works fine, but whenever I try a different docker stack; such as jupyter/datascience-notebook:hub-2.2.2 I receive signal 15 and it shuts down (swarm spawner). I can post my logs if it would help triangulate the issue, but I don’t want to thread hijack.
Thanks @Actuary ! Maybe a shorter version of the logs?
My setup is domain name → router (192.168.86.1) → VM with reverse proxy to terminate SSL (192.168.86.100) → Arch VM with jupyterhub (192.168.86.101).
Proxy
swarm_proxy.1.wt2ok5xvj841@jupyter | 00:39:29.710 [ConfigProxy] info: Adding route / -> http://jupyterhub:8081
swarm_proxy.1.wt2ok5xvj841@jupyter | 00:39:29.729 [ConfigProxy] info: Proxying http://*:8000 to http://jupyterhub:8081
swarm_proxy.1.wt2ok5xvj841@jupyter | 00:39:29.730 [ConfigProxy] info: Proxy API at http://0.0.0.0:8001/api/routes
swarm_proxy.1.wt2ok5xvj841@jupyter | 00:39:29.733 [ConfigProxy] info: Route added / -> http://jupyterhub:8081
swarm_proxy.1.wt2ok5xvj841@jupyter | 00:39:33.105 [ConfigProxy] info: 200 GET /api/routes
swarm_proxy.1.wt2ok5xvj841@jupyter | 00:39:33.109 [ConfigProxy] info: 200 GET /api/routes
swarm_proxy.1.wt2ok5xvj841@jupyter | 00:39:33.113 [ConfigProxy] info: Adding route / -> http://jupyterhub:8081
swarm_proxy.1.wt2ok5xvj841@jupyter | 00:39:33.113 [ConfigProxy] info: Route added / -> http://jupyterhub:8081
swarm_proxy.1.wt2ok5xvj841@jupyter | 00:39:33.115 [ConfigProxy] info: 201 POST /api/routes/
swarm_proxy.1.wt2ok5xvj841@jupyter | 00:44:33.130 [ConfigProxy] info: 200 GET /api/routes
swarm_proxy.1.wt2ok5xvj841@jupyter | 00:49:33.124 [ConfigProxy] info: 200 GET /api/routes
Dockerfile - swarm_hub:2.2.2
RUN pip install --upgrade pip
RUN pip install oauthenticator
RUN pip install jupyterlab
RUN pip install --no-cache dockerspawner```
Other files are a bit larger so I threw them on pastebin for easier reading.
[jupyterhub_config.py](https://pastebin.com/ehKKLEfs)
[yaml](https://pastebin.com/NRZGHr2P)
[Notebook log](https://pastebin.com/jSkmbPQG)
[Hub log](https://pastebin.com/eWR0DXL7)Here are my full logs, I tossed them on pastebin for easier reading.
@Actuary A healthcheck was recently added to the jupyter docker images:
Do you still have a problem if you use an older image from before that PR was merged, or if you somehow disable the healthcheck?
Thanks for the suggestion manics! I think you may have found the issue!
After using version 2.2.0 for both the hub and notebook image, I was able to get it working. As a check, I swapped them both to 2.2.2 and confirmed that I was still receiving the signal 15 issue; therefore, my configuration must be having an issue with a change between those version numbers.
I am not sure why, but when I tried to set the following parameter in jupyterhub_config.py, it was not able to find the associated image despite it being on dockerhub. Maybe I made a typo and did not realize it, but I felt like I should mention it just in case.
c.SwarmSpawner.image = 'jupyter/datascience-notebook:hub-2.2.0'
Making the following image and then using it in the above parameter worked though.
swarm_notebook:2.2.0
From jupyter/datascience-notebook:hub-2.2.0
ARG JUPYTERHUB_VERSION=2.2.0
RUN pip3 install --no-cache \ jupyterhub==$JUPYTERHUB_VERSION
swarm_hub:2.2.0
FROM jupyterhub/jupyterhub:2.2.0
RUN pip install --upgrade pip
RUN pip install oauthenticator
RUN pip install --no-cache dockerspawner
Thanks Both! @Actuary Do you mean you don’t have signal 15 issue on version 2.2.0 but the issue is there when using version 2.2.2?
That’s correct! Were you able to bypass the signal 15 issue when using the default image?
Thanks! I have not tried the default image as we need a customised image… I also have problem upgrading to 2.2.2 from 1.1.0 , I will hold on trying to upgrade for now. Thanks.
I updated to 2.3.0 and the issue is no longer present for me, just thought I’d write back and let you know!