500 error on /api/metrics/v1?

I’ve upgraded to Jupyterhub 5.0 and Notebook 6.5.7, and I’m trying to get jupyter-resource-usage 0.7.1 working. However, I get constant 500 server errors when /api/metrics/v1 is called.

[W 2024-08-02 14:49:07.055 SingleUserNotebookApp handlers:774] wrote error: 'Unhandled error'
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.11/site-packages/tornado/web.py", line 1769, in _execute
        result = await result  # type: ignore
                 ^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/jupyter_server/base/handlers.py", line 751, in prepare
        await super().prepare()
      File "/opt/conda/lib/python3.11/site-packages/jupyter_server/base/handlers.py", line 618, in prepare
        user = User(self.get_current_user())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<string>", line 9, in __init__
      File "/opt/conda/lib/python3.11/site-packages/jupyter_server/auth/identity.py", line 59, in __post_init__
        self.fill_defaults()
      File "/opt/conda/lib/python3.11/site-packages/jupyter_server/auth/identity.py", line 72, in fill_defaults
        raise ValueError(msg)
    ValueError: user.username must not be empty: User(username=None, name='', display_name='', initials=None, avatar_url=None, color=None)

The weird thing is that I can open this url directly from my browser and it works fine:

https://xxx/user/yyy/api/metrics/v1
{"rss": 319979520, "limits": {"memory": {"rss": 6442450944, "pss": 6442450944, "warn": false}}, "pss": 280208384}

I’m not sure what is causing this user.username error, do I need to be setting that somewhere? I use a custom authenticator which returns name.

async def authenticate(self, handler, data=None):
 return {
                'name': data['username'],
                'auth_state': xxx
                }

I don’t have a problem with authentication or the notebook executing cells, just with /api/metrics called from within the notebook, or even using curl from the notebook terminal. I’m not quite sure what could be the problem.

Thanks,
Mark

I’m not sure if this is related, but I used to be able to interact with local files through requests in notebook, but they now lead to a authorization redirect loop:

GET https://xxx/user/yyy/notebooks/experiments/dibromoethane.molden
GET https:/xxx/hub/api/oauth2/authorize?client_id=jupyterhub-user-yyy&redirect_uri=zzz%2Foauth_callback&response_type=code&state=-www
GET https://xxx/user/yyy/oauth_callback?code=zzz&state=www
<repeats until ERR_TOO_MANY_REDIRECTS>

If I access https://xxx/user/yyy/notebooks/experiments/dibromoethane.molden separately n my browser it works just fine

I suspect this has to do with the combination of versions with notebook < 7 and jupyter-server 2.

Since it looks like you want to use the legacy notebook server, does it work to do:

JUPYTERHUB_SINGLEUSER_APP=notebook

in your spawner environment (c.Spawner.environment)?

How is the request to GET https://xxx/user/yyy/notebooks/experiments/dibromoethane.molden authenticated? And what are the logs for the request from the single-user server?

Thanks, it could be that I need to update to notebook 7, but I was worried about extensions that our users use. And mamba didn’t install notebook 7 when I built this image, so I assume I have another package conflicting somewhere.

I had been setting notebook in CustomFormSpawner:

class CustomFormSpawner(SwarmSpawner):
    def options_from_form(self, formdata):
    self.environment = {
            'JUPYTERHUB_SINGLEUSER_APP':'notebook.notebookapp.NotebookApp',
                'LOG_TERMINAL_OUTPUT':'true',
            }

I changed it to just ‘notebook’, but didn’t see any change

self.environment = {
                'JUPYTERHUB_SINGLEUSER_APP':'notebook',
                'LOG_TERMINAL_OUTPUT':'true',
            }

The GET request for diboromoethane.molden is sent from the jupyter_jsmol plugin, which makes the request using javascript from the user’s browser, so it shows up in developer tools. It’s normally authenticated the same as the notebook, I assume it uses jupyterhub-session-id, which is set and the same value as the call to /api/metrics/v1 and the original request for the notebook itself.

The single user server shows:

[I 2024-08-05 17:42:54.188 SingleUserNotebookApp log:192] 302 GET /jupyterhub_internal/user/xxx/notebooks/experiments/dibromoethane.molden -> /jupyterhub_internal/hub/api/oauth2/authorize?client_id=jupyterhub-user-xxx&redirect_uri=%2Fjupyterhub_internal%2Fuser%2Fxxx%2Foauth_callback&response_type=code&state=[secret] (@::ffff:10.0.0.30) 1.12ms
[I 2024-08-05 17:42:54.389 SingleUserNotebookApp auth:1614] Logged-in user xxx
[I 2024-08-05 17:42:54.390 SingleUserNotebookApp _xsrf_utils:125] Setting new xsrf cookie for b'yyy:bb61644f4c3da3c88235512c14540ac4f75fdc49b9c52190204e5feb5f88d93d' {'path': '/jupyterhub_internal/user/xxx/'}
[I 2024-08-05 17:42:54.390 SingleUserNotebookApp log:192] 302 GET /jupyterhub_internal/user/xxx/oauth_callback?code=[secret]&state=[secret] -> /jupyterhub_internal/user/xxx/notebooks/experiments/dibromoethane.molden (@::ffff:10.0.0.30) 31.93ms
[I 2024-08-05 17:42:54.763 SingleUserNotebookApp log:192] 302 GET /jupyterhub_internal/user/xxx/notebooks/experiments -> /jupyterhub_internal/user/xxx/tree/experiments (xxx@::ffff:10.0.0.30) 1.65ms
[I 2024-08-05 17:42:54.834 SingleUserNotebookApp log:192] 200 GET /jupyterhub_internal/user/xxx/api (@127.0.0.1) 0.71ms
[I 2024-08-05 17:42:54.845 SingleUserNotebookApp log:192] 200 GET /jupyterhub_internal/user/xxx/tree/experiments (xxx@::ffff:10.0.0.30) 2.15ms
[I 2024-08-05 17:42:55.168 SingleUserNotebookApp log:192] 302 GET /jupyterhub_internal/user/xxx/notebooks/experiments/dibromoethane.molden -> /jupyterhub_internal/hub/api/oauth2/authorize?client_id=jupyterhub-user-xxx&redirect_uri=%2Fjupyterhub_internal%2Fuser%2Fxxx%2Foauth_callback&response_type=code&state=[secret] (@::ffff:10.0.0.30) 1.26ms
[I 2024-08-05 17:42:55.420 SingleUserNotebookApp auth:1614] Logged-in user xxx
[I 2024-08-05 17:42:55.420 SingleUserNotebookApp _xsrf_utils:125] Setting new xsrf cookie for b'yyy:818e161864e2f0661a983ab521066a34513803bab3a7b2552f96d7c8b48d8f58' {'path': '/jupyterhub_internal/user/xxx/'}
[I 2024-08-05 17:42:55.421 SingleUserNotebookApp log:192] 302 GET /jupyterhub_internal/user/xxx/oauth_callback?code=[secret]&state=[secret] -> /jupyterhub_internal/user/xxx/notebooks/experiments/dibromoethane.molden (@::ffff:10.0.0.30) 34.52ms
[I 2024-08-05 17:42:55.507 SingleUserNotebookApp log:192] 302 GET /jupyterhub_internal/user/xxx/notebooks/experiments/dibromoethane.molden -> /jupyterhub_internal/hub/api/oauth2/authorize?client_id=jupyterhub-user-xxx&redirect_uri=%2Fjupyterhub_internal%2Fuser%2Fxxx%2Foauth_callback&response_type=code&state=[secret] (@::ffff:10.0.0.30) 1.12ms
[I 2024-08-05 17:42:55.714 SingleUserNotebookApp auth:1614] Logged-in user xxx
[I 2024-08-05 17:42:55.714 SingleUserNotebookApp _xsrf_utils:125] Setting new xsrf cookie for b'yyy:7a626bbaebee00f9471b1871e83810ab9f1fb99dcbd7dbb55d9ae740b8787cfc' {'path': '/jupyterhub_internal/user/xxx/'}
[I 2024-08-05 17:42:55.714 SingleUserNotebookApp log:192] 302 GET /jupyterhub_internal/user/xxx/oauth_callback?code=[secret]&state=[secret] -> /jupyterhub_internal/user/xxx/notebooks/experiments/dibromoethane.molden (@::ffff:10.0.0.30) 32.40ms

If it helps, here are the jupyter packages installed in the notebook image:

(base) jovyan@8fbe2a8cd394:/tmp$ mamba list | grep jupyter
jupyter                   1.0.0                    pypi_0    pypi
jupyter-console           6.6.3                    pypi_0    pypi
jupyter-jsmol             2022.1.0                 pypi_0    pypi
jupyter-lsp               2.2.5              pyhd8ed1ab_0    conda-forge
jupyter-resource-usage    0.7.1              pyhd8ed1ab_0    conda-forge
jupyter-server-mathjax    0.2.6              pyh5bfe37b_1    conda-forge
jupyter_client            7.4.9              pyhd8ed1ab_0    conda-forge
jupyter_contrib_core      0.4.0              pyhd8ed1ab_0    conda-forge
jupyter_contrib_nbextensions 0.7.0              pyhd8ed1ab_0    conda-forge
jupyter_core              5.7.2           py311h38be061_0    conda-forge
jupyter_events            0.10.0             pyhd8ed1ab_0    conda-forge
jupyter_highlight_selected_word 0.2.0           pyhd8ed1ab_1006    conda-forge
jupyter_latex_envs        1.4.6           pyhd8ed1ab_1002    conda-forge
jupyter_nbextensions_configurator 0.6.1              pyhd8ed1ab_0    conda-forge
jupyter_server            2.14.2             pyhd8ed1ab_0    conda-forge
jupyter_server_terminals  0.5.3              pyhd8ed1ab_0    conda-forge
jupyteranalytics          1.1                      pypi_0    pypi
jupyterhub                5.0.0              pyh31011fe_0    conda-forge
jupyterhub-base           5.0.0              pyh31011fe_0    conda-forge
jupyterlab                4.2.4              pyhd8ed1ab_0    conda-forge
jupyterlab-git            0.50.1             pyhd8ed1ab_1    conda-forge
jupyterlab_pygments       0.3.0              pyhd8ed1ab_1    conda-forge
jupyterlab_server         2.27.3             pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.11             pyhd8ed1ab_0    conda-forge

Thanks,
Mark

My guess is it’s running into XSRF check issues introduced in jupyterhub 4.1. Constantly redirecting and setting new xsrf cookies is odd. I’ll see if I can produce a sample environment to reproduce it.

I downgraded the hub back to 2.1 and got this error: Frequent "XSRF Cookies do not match Post Argument" errors · Issue #15083 · jupyterlab/jupyterlab · GitHub

Deleting all _xsrf cookies and reloading the browser fixed them. Interestingly, I went back to hub 5.0 and loaded an old notebook that had jupyterhub 2.1 installed, and everything worked fine. So new hub and old notebook works:

Hub 5.0 and Notebook with Hub 2.1 installed works:

(base) jovyan@925d8fc772af:/tmp$ mamba list | grep jupyter
jupyter                   1.0.0                    pypi_0    pypi
jupyter-console           6.6.3                    pypi_0    pypi
jupyter-jsmol             2022.1.0                 pypi_0    pypi
jupyter-lsp               2.2.5              pyhd8ed1ab_0    conda-forge
jupyter-server-mathjax    0.2.6              pyh5bfe37b_1    conda-forge
jupyter_client            7.4.9              pyhd8ed1ab_0    conda-forge
jupyter_contrib_core      0.4.0              pyhd8ed1ab_0    conda-forge
jupyter_contrib_nbextensions 0.7.0              pyhd8ed1ab_0    conda-forge
jupyter_core              5.7.2           py311h38be061_0    conda-forge
jupyter_events            0.10.0             pyhd8ed1ab_0    conda-forge
jupyter_highlight_selected_word 0.2.0           pyhd8ed1ab_1006    conda-forge
jupyter_latex_envs        1.4.6           pyhd8ed1ab_1002    conda-forge
jupyter_nbextensions_configurator 0.6.1              pyhd8ed1ab_0    conda-forge
jupyter_server            2.14.2             pyhd8ed1ab_0    conda-forge
jupyter_server_terminals  0.5.3              pyhd8ed1ab_0    conda-forge
jupyter_telemetry         0.1.0              pyhd8ed1ab_1    conda-forge
jupyteranalytics          1.1                      pypi_0    pypi
jupyterhub                2.1.1                hd8ed1ab_0    conda-forge
jupyterhub-base           2.1.1              pyhd8ed1ab_0    conda-forge
jupyterlab                4.2.4              pyhd8ed1ab_0    conda-forge
jupyterlab-git            0.50.1             pyhd8ed1ab_1    conda-forge
jupyterlab_pygments       0.3.0              pyhd8ed1ab_1    conda-forge
jupyterlab_server         2.27.3             pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.11             pyhd8ed1ab_0    conda-forge

Hub 5.0 and Notebook with Hub 5.0 installed gives the 500 error on metrics and auth redirect when accessing files through xhr request:

jupyter                   1.0.0                    pypi_0    pypi
jupyter-console           6.6.3                    pypi_0    pypi
jupyter-jsmol             2022.1.0                 pypi_0    pypi
jupyter-lsp               2.2.5              pyhd8ed1ab_0    conda-forge
jupyter-resource-usage    0.7.1              pyhd8ed1ab_0    conda-forge
jupyter-server-mathjax    0.2.6              pyh5bfe37b_1    conda-forge
jupyter_client            7.4.9              pyhd8ed1ab_0    conda-forge
jupyter_contrib_core      0.4.0              pyhd8ed1ab_0    conda-forge
jupyter_contrib_nbextensions 0.7.0              pyhd8ed1ab_0    conda-forge
jupyter_core              5.7.2           py311h38be061_0    conda-forge
jupyter_events            0.10.0             pyhd8ed1ab_0    conda-forge
jupyter_highlight_selected_word 0.2.0           pyhd8ed1ab_1006    conda-forge
jupyter_latex_envs        1.4.6           pyhd8ed1ab_1002    conda-forge
jupyter_nbextensions_configurator 0.6.1              pyhd8ed1ab_0    conda-forge
jupyter_server            2.14.2             pyhd8ed1ab_0    conda-forge
jupyter_server_terminals  0.5.3              pyhd8ed1ab_0    conda-forge
jupyteranalytics          1.1                      pypi_0    pypi
jupyterhub                5.0.0              pyh31011fe_0    conda-forge
jupyterhub-base           5.0.0              pyh31011fe_0    conda-forge
jupyterlab                4.2.4              pyhd8ed1ab_0    conda-forge
jupyterlab-git            0.50.1             pyhd8ed1ab_1    conda-forge
jupyterlab_pygments       0.3.0              pyhd8ed1ab_1    conda-forge
jupyterlab_server         2.27.3             pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.11             pyhd8ed1ab_0    conda-forge

Thanks,
Mark

I’m having trouble setting up a test environment, but I suspect the issue is that requests from jupyter-jsmol are insufficiently authenticated - they must either include xsrf tokens, or provide the access token in the Authorization header. JupyterHub 4.1 / 5.0 fixed a security issue by increasing the number of requests this applies to (basically all XHRs). Some of these requests could usually work before, by assuming cookies are sufficient auth (this wouldn’t work in cross-site situations where tokens are required, etc.), but that’s no longer a safe assumption. Several extensions have been caught up by this, but the fix is usually not too bad, and fully backward compatible (the information required has always been available).

In the context of a JupyterLab extension (e.g. notebook 7), this is best simplified by using JupyterLab’s ServerConnection.makeRequest when making requests to the server, which will add all appropriate auth, depending on the context. I can’t find where jupyter-jsmol makes its requests, though.

Thanks, that makes sense. I think it is making the requests from JSmol, which is its own JS package (jupyter_jsmol is just a wrapper). It’s something that I can work around by using python to read the file and passing it as a string.

Do you have an idea how I can fix the metrics 500 server error?

Thanks,
Mark

can jsmol be instructed to specify an Authorization header when fetching data?

Here is a patch for another extension that uses its own library (axios) to make requests, so it gets the token from jupyterlab settings into the axios configuration. Hopefully jsmol has something equivalent if it has functions for fetching data.