Trouble Configuring Internal SSL with a custom service running

I have a Jupyter Hub deployment and currently I am trying to set up internal SSL. However, I think I am having issues because of a custom service that I wrote. I’m a bit green when it comes to certificate configuration, so any help is appreciated.

Here is a snippet from my config file showing how I set up my service and how I have enabled internal SSL. :

c.JupyterHub.services = [
    {
        "name": "<my_website>",
        "url": "https://127.0.0.1:8899",
        "command": ["python3", "/opt/tljh/services/<my_website>_proxy.py"],
        "oauth_no_confirm": True,
    }
]

c.JupyterHub.trusted_alt_names = [
    "DNS:<domain jupyter hub is being served at>",
    "IP:127.0.0.1",
    "DNS:localhost",
    "DNS:<domain the website I am trying to proxy is served at>",
]

c.JupyterHub.internal_certs_location = "/etc/jupyterhub/internal-ssl"
c.JupyterHub.internal_ssl = True

The <my_website>_proxy.py runs a proxy to a private website through a Tornado HTTP server. I am using this in conjunction with custom JupyterLab Launcher Cards to embed the website in an i-frame with in the analysis environment.

When I run tljh-config reload, the certifcates are generated along with the JSON below that shows how they are related:

{
    "hub-ca": {
        "serial": 0,
        "is_ca": true,
        "parent_ca": "",
        "signees": {
            "hub-internal": 1
        },
        "files": {
            "key": "/etc/jupyterhub/internal-ssl/hub-ca/hub-ca.key",
            "cert": "/etc/jupyterhub/internal-ssl/hub-ca/hub-ca.crt",
            "ca": ""
        }
    },
    "notebooks-ca": {
        "serial": 0,
        "is_ca": true,
        "parent_ca": "",
        "signees": {
            "user-service": 1
        },
        "files": {
            "key": "/etc/jupyterhub/internal-ssl/notebooks-ca/notebooks-ca.key",
            "cert": "/etc/jupyterhub/internal-ssl/notebooks-ca/notebooks-ca.crt",
            "ca": ""
        }
    },
    "services-ca": {
        "serial": 0,
        "is_ca": true,
        "parent_ca": "",
        "signees": null,
        "files": {
            "key": "/etc/jupyterhub/internal-ssl/services-ca/services-ca.key",
            "cert": "/etc/jupyterhub/internal-ssl/services-ca/services-ca.crt",
            "ca": ""
        }
    },
    "proxy-api-ca": {
        "serial": 0,
        "is_ca": true,
        "parent_ca": "",
        "signees": {
            "proxy-api": 1
        },
        "files": {
            "key": "/etc/jupyterhub/internal-ssl/proxy-api-ca/proxy-api-ca.key",
            "cert": "/etc/jupyterhub/internal-ssl/proxy-api-ca/proxy-api-ca.crt",
            "ca": ""
        }
    },
    "proxy-client-ca": {
        "serial": 0,
        "is_ca": true,
        "parent_ca": "",
        "signees": {
            "proxy-client": 1
        },
        "files": {
            "key": "/etc/jupyterhub/internal-ssl/proxy-client-ca/proxy-client-ca.key",
            "cert": "/etc/jupyterhub/internal-ssl/proxy-client-ca/proxy-client-ca.crt",
            "ca": ""
        }
    },
    "hub-internal": {
        "serial": 0,
        "is_ca": false,
        "parent_ca": "hub-ca",
        "signees": null,
        "files": {
            "key": "/etc/jupyterhub/internal-ssl/hub-internal/hub-internal.key",
            "cert": "/etc/jupyterhub/internal-ssl/hub-internal/hub-internal.crt",
            "ca": "/etc/jupyterhub/internal-ssl/hub-ca/hub-ca.crt"
        }
    },
    "proxy-api": {
        "serial": 0,
        "is_ca": false,
        "parent_ca": "proxy-api-ca",
        "signees": null,
        "files": {
            "key": "/etc/jupyterhub/internal-ssl/proxy-api/proxy-api.key",
            "cert": "/etc/jupyterhub/internal-ssl/proxy-api/proxy-api.crt",
            "ca": "/etc/jupyterhub/internal-ssl/proxy-api-ca/proxy-api-ca.crt"
        }
    },
    "proxy-client": {
        "serial": 0,
        "is_ca": false,
        "parent_ca": "proxy-client-ca",
        "signees": null,
        "files": {
            "key": "/etc/jupyterhub/internal-ssl/proxy-client/proxy-client.key",
            "cert": "/etc/jupyterhub/internal-ssl/proxy-client/proxy-client.crt",
            "ca": "/etc/jupyterhub/internal-ssl/proxy-client-ca/proxy-client-ca.crt"
        }
    },
    "user-service": {
        "serial": 669,
        "is_ca": false,
        "parent_ca": "notebooks-ca",
        "signees": null,
        "files": {
            "key": "/etc/jupyterhub/internal-ssl/user-service/user-service.key",
            "cert": "/etc/jupyterhub/internal-ssl/user-service/user-service.crt",
            "ca": "/etc/jupyterhub/internal-ssl/notebooks-ca/notebooks-ca.crt"
        }
    }
}

Now I am a little confused on how to make my custom proxy server utilize HTTPS and I think I am messing up on how I am creating the SSL context.

def create_ssl_context_for_server():
    """Create SSL context for this service to present its certificate"""
    context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
    context.load_cert_chain(/etc/jupyterhub/internal-ssl/<my_website>_service.crt, /etc/jupyterhub/internal-ssl/<my_website>_service.key)
    
    # Load the services CA to verify client certificates if needed
    if os.path.exists(/etc/jupyterhub/internal-ssl/services-ca.crt):
        context.load_verify_locations(/etc/jupyterhub/internal-ssl/services-ca.crt)
        context.verify_mode = ssl.CERT_OPTIONAL  
    
    return context

The <my_website>_service.crt and <my_website>_service.key is generated with Opelssl from the services-ca, but I can’t reload Jupyter Hub as it’s throwing the error(s):

Jul 28 23:23:05 ip-172-16-0-86 python3[34817]: 2025-07-28 23:23:05,291 - WARNING - SSL Error on 8 ('127.0.0.1', 60000): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)
Jul 28 23:23:05 ip-172-16-0-86 python3[34798]: [W 2025-07-28 23:23:05.292 JupyterHub iostream:779] error on read: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2559)

These are just warnings which I assume coming from health check of the services. This should not block your JupyterHub startup. Could you please share full logs right from the begining of the hub start?

2 Likes

Since I configured internal_ssl I have not been able to reload the config until I set internal_ssl back to False.

When I run tljh-config reload I get the following (note the 500 error code starts only after two of the 502s and then continues forever. I am unsure why.):

active
/opt/tljh/hub/lib/python3.12/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Hub not ready: (HTTP status 502)
/opt/tljh/hub/lib/python3.12/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Hub not ready: (HTTP status 502)
/opt/tljh/hub/lib/python3.12/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Hub not ready: (HTTP status 500)
/opt/tljh/hub/lib/python3.12/site-packages/urllib3/connectionpool.py:1097: InsecureRequestWarning: Unverified HTTPS request is being made to host '127.0.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings

When I check the logs after running this with journalctl -u jupyterhub, this is the output (all the logs with INFO in the output are debug statements from <my_service>_proxy.py:

Jul 30 16:17:44 ip-172-16-0-86 systemd[1]: Started jupyterhub.service.
Jul 30 16:17:45 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:45.869 JupyterHub app:3346] Running JupyterHub version 5.2.1
Jul 30 16:17:45 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:45.870 JupyterHub app:3376] Using Authenticator: oauthenticator.google.GoogleOAuthenticator-17.3.0
Jul 30 16:17:45 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:45.870 JupyterHub app:3376] Using Spawner: tljh.user_creating_spawner.UserCreatingSpawner
Jul 30 16:17:45 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:45.870 JupyterHub app:3376] Using Proxy: jupyterhub_traefik_proxy.fileprovider.TraefikFileProviderProxy-2.0.0
Jul 30 16:17:45 ip-172-16-0-86 python3[2069]: /opt/tljh/hub/lib/python3.12/site-packages/jupyter_events/schema.py:68: JupyterEventsVersionWarning: The `version` property of an event schema must be a string. It has been type coerced, but in a future version of this library, it will fail to validate. Please update schema: https://schema.jupyter.org/jupyterhub/events/server-action
Jul 30 16:17:45 ip-172-16-0-86 python3[2069]:   validate_schema(_schema)
Jul 30 16:17:45 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:45.880 JupyterHub app:1831] Loading cookie_secret from /opt/tljh/state/jupyterhub_cookie_secret
Jul 30 16:17:45 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:45.904 JupyterHub app:1945] Using existing hub-internal CA
Jul 30 16:17:45 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:45.904 JupyterHub app:1965] Using existing proxy-api CA
Jul 30 16:17:45 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:45.904 JupyterHub app:1965] Using existing proxy-client CA
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:46.127 JupyterHub app:2881] Creating service <my_service> with oauth_client_id=<my_service>
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:46.135 JupyterHub provider:663] Updating oauth client service-<my_service>
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:46.189 JupyterHub app:2597] Setting admin=True on <jupyterhub.orm.Service object at 0x7a958746b8f0>
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [W 2025-07-30 16:17:46.196 JupyterHub app:3212] Allowing service <my_service> to complete OAuth without confirmation on an authorization web page
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:46.202 JupyterHub fileprovider:99] Creating the dynamic configuration file: /opt/tljh/state/rules/rules.toml
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:46.231 JupyterHub app:3416] Initialized 0 spawners in 0.028 seconds
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:46.236 JupyterHub metrics:373] Found 1 active users in the last ActiveUserPeriods.twenty_four_hours
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:46.237 JupyterHub metrics:373] Found 2 active users in the last ActiveUserPeriods.seven_days
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:46.238 JupyterHub metrics:373] Found 2 active users in the last ActiveUserPeriods.thirty_days
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:46.238 JupyterHub app:3703] Not starting proxy
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:46.285 JupyterHub app:3739] Hub API listening on https://127.0.0.1:15001/hub/
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:46.285 JupyterHub app:3615] Starting managed service <my_service> at https://127.0.0.1:8899
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:46.285 JupyterHub service:423] Starting service '<my_service>': ['python3', '/opt/tljh/services/<my_service>_proxy.py']
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:46.286 JupyterHub spawner:1270] Creating certs for : DNS:localhost;IP:127.0.0.1
Jul 30 16:17:46 ip-172-16-0-86 python3[2069]: [I 2025-07-30 16:17:46.465 JupyterHub service:136] Spawning python3 /opt/tljh/services/<my_service>_proxy.py
Jul 30 16:17:46 ip-172-16-0-86 python3[2079]: 2025-07-30 16:17:46,643 - INFO - JUPYTERHUB_API_TOKEN set: YES
Jul 30 16:17:46 ip-172-16-0-86 python3[2079]: 2025-07-30 16:17:46,643 - INFO - Found <my_service> Service Certificate: /etc/jupyterhub/internal-ssl/<my_service>-service/<my_service>-service.crt
Jul 30 16:17:46 ip-172-16-0-86 python3[2079]: 2025-07-30 16:17:46,644 - INFO - Found <my_service> Service Key: /etc/jupyterhub/internal-ssl/<my_service>-service/<my_service>-service.key
Jul 30 16:17:46 ip-172-16-0-86 python3[2079]: 2025-07-30 16:17:46,644 - INFO - Found <my_service> Service CA: /etc/jupyterhub/internal-ssl/services-ca/services-ca.crt
Jul 30 16:17:46 ip-172-16-0-86 python3[2079]: 2025-07-30 16:17:46,644 - INFO - Found JupyterHub Client Certificate: /etc/jupyterhub/internal-ssl/proxy-client/proxy-client.crt
Jul 30 16:17:46 ip-172-16-0-86 python3[2079]: 2025-07-30 16:17:46,644 - INFO - Found JupyterHub Client Key: /etc/jupyterhub/internal-ssl/proxy-client/proxy-client.key
Jul 30 16:17:46 ip-172-16-0-86 python3[2079]: 2025-07-30 16:17:46,644 - INFO - Found JupyterHub CA Certificate: /etc/jupyterhub/internal-ssl/proxy-client-ca/proxy-client-ca.crt
Jul 30 16:17:46 ip-172-16-0-86 python3[2079]: 2025-07-30 16:17:46,858 - INFO - Backend <my_service> service is reachable at https://<my_service>
Jul 30 16:17:46 ip-172-16-0-86 python3[2079]: 2025-07-30 16:17:46,865 - DEBUG - Using selector: EpollSelector
Jul 30 16:17:46 ip-172-16-0-86 python3[2079]: 2025-07-30 16:17:46,869 - INFO - <my_service> proxy running on https://127.0.0.1:8899/services/<my_service>(HTTPS)
Jul 30 16:17:46 ip-172-16-0-86 python3[2079]: 2025-07-30 16:17:46,869 - INFO - JupyterHub API URL: https://127.0.0.1:15001/hub/api
Jul 30 16:17:46 ip-172-16-0-86 python3[2079]: 2025-07-30 16:17:46,869 - INFO - <my_service> Service URL: https://<my_service>
Jul 30 16:17:47 ip-172-16-0-86 python3[2079]: 2025-07-30 16:17:47,270 - WARNING - SSL Error on 8 ('127.0.0.1', 58586): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)
Jul 30 16:17:47 ip-172-16-0-86 python3[2069]: [W 2025-07-30 16:17:47.270 JupyterHub iostream:779] error on read: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2559)

Then when I try to visit JupyterHub from the browser, it shows Internal Server Error on a white screen.

There are no errors in the logs you shared. Could you please turn on DEBUG logging on JupyterHub and follow journal logs in one terminal `journalctl -f -u jupyterhub` and reload TLJH in another terminal so that we will see any errors in JupyterHub logs?

I added c.JupyterHub.log_level = ‘DEBUG’ to the config file and ran reload again.

Here are the logs:

Jul 31 22:29:26 ip-172-16-0-86 systemd[1]: Stopping jupyterhub.service…
Jul 31 22:29:26 ip-172-16-0-86 systemd[1]: jupyterhub.service: Deactivated successfully.
Jul 31 22:29:26 ip-172-16-0-86 systemd[1]: Stopped jupyterhub.service.
Jul 31 22:29:26 ip-172-16-0-86 systemd[1]: jupyterhub.service: Consumed 2.278s CPU time.
Jul 31 22:29:26 ip-172-16-0-86 systemd[1]: Started jupyterhub.service.
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:27.671 JupyterHub app:3346] Running JupyterHub version 5.2.1
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:27.671 JupyterHub app:3376] Using Authenticator: oauthenticator.google.GoogleOAuthenticator-17.3.0
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:27.671 JupyterHub app:3376] Using Spawner: tljh.user_creating_spawner.UserCreatingSpawner
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:27.671 JupyterHub app:3376] Using Proxy: jupyterhub_traefik_proxy.fileprovider.TraefikFileProviderProxy-2.0.0
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: /opt/tljh/hub/lib/python3.12/site-packages/jupyter_events/schema.py:68: JupyterEventsVersionWarning: The version property of an event schema must be a string. It has been type coerced, but in a future version of this library, it will fail to validate. Please update schema: https://schema.jupyter.org/jupyterhub/events/server-action
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: validate_schema(_schema)
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:27.681 JupyterHub app:1831] Loading cookie_secret from /opt/tljh/state/jupyterhub_cookie_secret
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:27.704 JupyterHub app:1945] Using existing hub-internal CA
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:27.704 JupyterHub app:1965] Using existing proxy-api CA
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:27.704 JupyterHub app:1965] Using existing proxy-client CA
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:27.747 JupyterHub app:1998] Connecting to db: sqlite:///jupyterhub.sqlite
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:27.764 JupyterHub orm:1509] database schema version found: 4621fec11365
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:27.771 JupyterHub orm:1509] database schema version found: 4621fec11365
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:27.844 JupyterHub app:2338] Loading roles into database
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:27.912 JupyterHub app:2881] Creating service <my_service> with oauth_client_id=service-<my_service>
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:27.920 JupyterHub provider:663] Updating oauth client service-<my_service>
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:27.943 JupyterHub app:2685] Purging expired APITokens
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:27.950 JupyterHub app:2685] Purging expired OAuthCodes
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:27.952 JupyterHub app:2685] Purging expired Shares
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:27.953 JupyterHub app:2685] Purging expired ShareCodes
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:27.957 JupyterHub app:2459] Loading role assignments from config
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:27.959 JupyterHub app:2570] Assigning allowed_users to the user role
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:27.964 JupyterHub app:2578] Assigning 0 allowed_users to the user role
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:27.973 JupyterHub app:2597] Setting admin=True on <jupyterhub.orm.Service object at 0x76d8460b6030>
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [W 2025-07-31 22:29:27.981 JupyterHub app:3212] Allowing service <my_service> to complete OAuth without confirmation on an authorization web page
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:27.987 JupyterHub fileprovider:99] Creating the dynamic configuration file: /opt/tljh/state/rules/rules.toml
Jul 31 22:29:27 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:27.987 JupyterHub proxy:510] Setting up traefik’s dynamic config…
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:28.004 JupyterHub fileprovider:95] Writing dynamic config {‘http’: {‘routers’: {‘route_api’: {‘rule’: ‘Host(localhost) && PathPrefix(/api)’, ‘entryPoints’: [‘auth_api’], ‘service’: ‘api@internal’, ‘middlewares’: [‘auth_api’]}, ‘router__2F’: {‘service’: ‘service__2F’, ‘rule’: ‘PathPrefix(/)’, ‘entryPoints’: [‘https’]}, ‘router__2Fservices_2F<my_service>_2F’: {‘service’: ‘service__2Fservices_2F<my_service>_2F’, ‘rule’: ‘( PathPrefix(/services/<my_service>/) || Path(/services/<my_service>) )’, ‘entryPoints’: [‘https’]}, ‘router__2Fuser_2F<my_user>_40<my_domain>_2Eco_2Ftest_2F’: {‘service’: ‘service__2Fuser_2F<my_user>_40<my_domain>_2Eco_2Ftest_2F’, ‘rule’: ‘( PathPrefix(/user/<my_user>@<my_domain>.co/test/) || Path(/user/<my_user>@<my_domain>.co/test) )’, ‘entryPoints’: [‘https’]}}, ‘middlewares’: {‘auth_api’: {‘basicAuth’: {‘users’: [‘api_admin:/’]}}}, ‘services’: {‘service__2F’: {‘loadBalancer’: {‘passHostHeader’: True, ‘servers’: [{‘url’: ‘https://127.0.0.1:15001’}]}}, ‘service__2Fservices_2F<my_service>_2F’: {‘loadBalancer’: {‘passHostHeader’: True, ‘servers’: [{‘url’: ‘https://127.0.0.1:8899’}]}}, ‘service__2Fuser_2F<my_user>_40<my_domain>_2Eco_2Ftest_2F’: {‘loadBalancer’: {‘passHostHeader’: True, ‘servers’: [{‘url’: ‘http://127.0.0.1:48609’}]}}}}, ‘jupyterhub’: {‘routes’: {‘router__2F’: {‘routespec’: ‘/’, ‘target’: ‘https://127.0.0.1:15001’, ‘router’: ‘router__2F’, ‘service’: ‘service__2F’, ‘data’: {‘hub’: True}}, ‘router__2Fservices_2F<my_service>_2F’: {‘routespec’: ‘/services/<my_service>/’, ‘target’: ‘https://127.0.0.1:8899’, ‘router’: ‘router__2Fservices_2F<my_service>_2F’, ‘service’: ‘service__2Fservices_2F<my_service>_2F’, ‘data’: {‘service’: ‘<my_service>’}}, ‘router__2Fuser_2F<my_user>_40<my_domain>_2Eco_2Ftest_2F’: {‘routespec’: ‘/user/<my_user>@<my_domain>.co/test/’, ‘target’: ‘http://127.0.0.1:48609’, ‘router’: ‘router__2Fuser_2F<my_user>_40<my_domain>_2Eco_2Ftest_2F’, ‘service’: ‘service__2Fuser_2F<my_user>_40<my_domain>_2Eco_2Ftest_2F’, ‘data’: {‘user’: ‘<my_user>@<my_domain>.co’, ‘server_name’: ‘test’}}}}}
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:28.005 JupyterHub app:2970] Initializing spawners
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:28.013 JupyterHub app:3120] Loaded users:
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]:
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:28.014 JupyterHub proxy:362] Fetching traefik api http://localhost:8099/api/overview
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:28.015 JupyterHub app:3416] Initialized 0 spawners in 0.029 seconds
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:28.021 JupyterHub metrics:373] Found 0 active users in the last ActiveUserPeriods.twenty_four_hours
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:28.022 JupyterHub metrics:373] Found 2 active users in the last ActiveUserPeriods.seven_days
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:28.023 JupyterHub metrics:373] Found 2 active users in the last ActiveUserPeriods.thirty_days
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:28.023 JupyterHub app:3703] Not starting proxy
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:28.025 JupyterHub proxy:372] 200 GET http://localhost:8099/api/overview
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:28.025 JupyterHub proxy:362] Fetching traefik api http://localhost:8099/api/entrypoints/https
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:28.027 JupyterHub proxy:372] 200 GET http://localhost:8099/api/entrypoints/https
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:28.072 JupyterHub app:3739] Hub API listening on https://127.0.0.1:15001/hub/
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:28.072 JupyterHub app:3615] Starting managed service <my_service> at https://127.0.0.1:8899
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:28.072 JupyterHub service:423] Starting service ‘<my_service>’: [‘python3’, ‘/opt/tljh/services/<my_service>_proxy.py’]
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:28.073 JupyterHub spawner:1270] Creating certs for : DNS:localhost;IP:127.0.0.1
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [I 2025-07-31 22:29:28.185 JupyterHub service:136] Spawning python3 /opt/tljh/services/<my_service>_proxy.py
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:28.186 JupyterHub spawner:1486] Polling subprocess every 30s
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:28.186 JupyterHub utils:292] Waiting 1s for server at https://127.0.0.1:8899/services/<my_service>
Jul 31 22:29:28 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:28,364 - INFO - JUPYTERHUB_API_TOKEN set: YES
Jul 31 22:29:28 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:28,364 - INFO - Found <my_service> Service Certificate: /etc/jupyterhub/internal-ssl/<my_service>-service/<my_service>-service.crt
Jul 31 22:29:28 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:28,364 - INFO - Found <my_service> Service Key: /etc/jupyterhub/internal-ssl/<my_service>-service/<my_service>-service.key
Jul 31 22:29:28 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:28,364 - INFO - Found <my_service> Service CA: /etc/jupyterhub/internal-ssl/services-ca/services-ca.crt
Jul 31 22:29:28 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:28,364 - INFO - Found JupyterHub Client Certificate: /etc/jupyterhub/internal-ssl/proxy-client/proxy-client.crt
Jul 31 22:29:28 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:28,364 - INFO - Found JupyterHub Client Key: /etc/jupyterhub/internal-ssl/proxy-client/proxy-client.key
Jul 31 22:29:28 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:28,364 - INFO - Found JupyterHub CA Certificate: /etc/jupyterhub/internal-ssl/proxy-client-ca/proxy-client-ca.crt
Jul 31 22:29:28 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:28,579 - INFO - Backend <my_service> service is reachable at https://<my_service>.<my_domain>.co
Jul 31 22:29:28 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:28,586 - DEBUG - Using selector: EpollSelector
Jul 31 22:29:28 ip-172-16-0-86 python3[2043]: [W 2025-07-31 22:29:28.589 JupyterHub iostream:1382] SSL Error on 8 (‘127.0.0.1’, 36270): [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1000)
Jul 31 22:29:28 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:28,591 - INFO - <my_service> proxy running on https://127.0.0.1:8899/services/<my_service> (HTTPS)
Jul 31 22:29:28 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:28,592 - INFO - JupyterHub API URL: https://127.0.0.1:15001/hub/api
Jul 31 22:29:28 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:28,592 - INFO - <my_service> Service URL: https://<my_service>.<my_domain>.co
Jul 31 22:29:29 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:29,172 - WARNING - SSL Error on 8 (‘127.0.0.1’, 38658): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)
Jul 31 22:29:29 ip-172-16-0-86 python3[2043]: [W 2025-07-31 22:29:29.173 JupyterHub iostream:779] error on read: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2559)
Jul 31 22:29:29 ip-172-16-0-86 python3[2043]: [W 2025-07-31 22:29:29.173 JupyterHub utils:317] Failed to connect to https://127.0.0.1:8899/services/<my_service> ([SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2559))
Jul 31 22:29:29 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:29,179 - WARNING - SSL Error on 8 (‘127.0.0.1’, 38662): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)
Jul 31 22:29:29 ip-172-16-0-86 python3[2043]: [W 2025-07-31 22:29:29.179 JupyterHub iostream:779] error on read: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2559)
Jul 31 22:29:29 ip-172-16-0-86 python3[2043]: [W 2025-07-31 22:29:29.179 JupyterHub utils:317] Failed to connect to https://127.0.0.1:8899/services/<my_service> ([SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2559))
Jul 31 22:29:29 ip-172-16-0-86 python3[2043]: [D 2025-07-31 22:29:29.180 JupyterHub utils:292] Waiting 1s for server at https://127.0.0.1:8899/services/<my_service>
Jul 31 22:29:29 ip-172-16-0-86 python3[2072]: 2025-07-31 22:29:29,185 - WARNING - SSL Error on 8 (‘127.0.0.1’, 38678): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)
Jul 31 22:29:29 ip-172-16-0-86 python3[2043]: [W 2025-07-31 22:29:29.185 JupyterHub iostream:779] error on read: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2559)
Jul 31 22:29:29 ip-172-16-0-86 python3[2043]: [W 2025-07-31 22:29:29.185 JupyterHub utils:317] Failed to connect to https://127.0.0.1:8899/services/<my_service> ([SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2559))

Jul 31 22:29:33 ip-172-16-0-86 systemd[1]: jupyterhub.service: Main process exited, code=exited, status=1/FAILURE
Jul 31 22:29:33 ip-172-16-0-86 systemd[1]: jupyterhub.service: Failed with result ‘exit-code’.
Jul 31 22:29:33 ip-172-16-0-86 systemd[1]: jupyterhub.service: Consumed 2.742s CPU time.
Jul 31 22:29:33 ip-172-16-0-86 systemd[1]: jupyterhub.service: Scheduled restart job, restart counter is at 1.

Is the error on read lines in the previous reply and the logs above not an error? To me that seems like the reason the service won’t reload…but obvioulsy I am not the expert lol

Well, they are all warnings from JupyterHub. I think the service is failing to launch or JupyterHub is not able to contact the service. It is strange because we do not see any concrete error message or traceback from JupyterHub before the process exited.

Anyways, one thing I have noticed is that you are not using correct CA for configuring your service. You should use internal_ssl/services-ca_trust.crt. In your origin post you are using /etc/jupyterhub/internal-ssl/services-ca.crt which I think does not exist.