JupyterLab uses Binder for PR review installing the JupyterLab version with modifications from given PR on Binder. Recently the builds stopped spawning with:
Spawn failed: Server at [...] didn't respond in 30 seconds.
How can I debug this? I’ve opened an issue on our side: Binder dev builds are broken slowing down review process · Issue #14033 · jupyterlab/jupyterlab · GitHub
We do have a number of custom options:
An example log (while it lasts): blob:https://mybinder.org/d3351412-56fe-4402-a225-c6ecd8a5a43e - there are no obvious errors, the relevant part is:
Pushing image
Successfully pushed 2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759jasonweill-2djupyterlab-81beb2:f4e8164e1e8a77fc9b57bbe26fac46e4a6f893dbBuilt image, launching...
Launching server...
Server requested
2023-02-18T10:33:52.382390Z [Normal] Successfully assigned ovh2/jupyter-jasonweill-2djupyterlab-2d20l6lqls to user-202211a-node-34f309
2023-02-18T10:33:53Z [Normal] Container image "jupyterhub/mybinder.org-tc-init:2020.12.4-0.dev.git.4289.h140cef5" already present on machine
2023-02-18T10:33:53Z [Normal] Created container tc-init
2023-02-18T10:33:53Z [Normal] Started container tc-init
2023-02-18T10:33:53Z [Normal] Pulling image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759jasonweill-2djupyterlab-81beb2:f4e8164e1e8a77fc9b57bbe26fac46e4a6f893db"
2023-02-18T10:35:33Z [Normal] Successfully pulled image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759jasonweill-2djupyterlab-81beb2:f4e8164e1e8a77fc9b57bbe26fac46e4a6f893db" in 1m39.720531713s
2023-02-18T10:35:33Z [Normal] Created container notebook
2023-02-18T10:35:33Z [Normal] Started container notebook
Spawn failed: Server at http://10.2.27.207:8888/user/jasonweill-jupyterlab-20l6lqls/ didn't respond in 30 seconds
Launch attempt 1 failed, retrying...
Server requested
2023-02-18T10:36:23.812187Z [Normal] Successfully assigned ovh2/jupyter-jasonweill-2djupyterlab-2dgxkt2p39 to user-202211a-node-34f309
2023-02-18T10:36:24Z [Normal] Container image "jupyterhub/mybinder.org-tc-init:2020.12.4-0.dev.git.4289.h140cef5" already present on machine
2023-02-18T10:36:24Z [Normal] Created container tc-init
2023-02-18T10:36:24Z [Normal] Started container tc-init
2023-02-18T10:36:25Z [Normal] Container image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759jasonweill-2djupyterlab-81beb2:f4e8164e1e8a77fc9b57bbe26fac46e4a6f893db" already present on machine
2023-02-18T10:36:25Z [Normal] Created container notebook
2023-02-18T10:36:25Z [Normal] Started container notebook
Spawn failed: Server at http://10.2.27.210:8888/user/jasonweill-jupyterlab-gxkt2p39/ didn't respond in 30 seconds
Launch attempt 2 failed, retrying...
Server requested
2023-02-18T10:37:16.166928Z [Normal] Successfully assigned ovh2/jupyter-jasonweill-2djupyterlab-2dst9pcb34 to user-202211a-node-34f309
2023-02-18T10:37:16Z [Normal] Container image "jupyterhub/mybinder.org-tc-init:2020.12.4-0.dev.git.4289.h140cef5" already present on machine
2023-02-18T10:37:16Z [Normal] Created container tc-init
2023-02-18T10:37:16Z [Normal] Started container tc-init
2023-02-18T10:37:17Z [Normal] Container image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759jasonweill-2djupyterlab-81beb2:f4e8164e1e8a77fc9b57bbe26fac46e4a6f893db" already present on machine
2023-02-18T10:37:17Z [Normal] Created container notebook
2023-02-18T10:37:17Z [Normal] Started container notebook
Spawn failed: Server at http://10.2.27.214:8888/user/jasonweill-jupyterlab-st9pcb34/ didn't respond in 30 seconds
Launch attempt 3 failed, retrying...
Server requested
2023-02-18T10:38:16.127401Z [Normal] Successfully assigned ovh2/jupyter-jasonweill-2djupyterlab-2dkoupg04s to user-202211a-node-34f309
2023-02-18T10:38:17Z [Normal] Container image "jupyterhub/mybinder.org-tc-init:2020.12.4-0.dev.git.4289.h140cef5" already present on machine
2023-02-18T10:38:17Z [Normal] Created container tc-init
2023-02-18T10:38:17Z [Normal] Started container tc-init
2023-02-18T10:38:18Z [Normal] Container image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759jasonweill-2djupyterlab-81beb2:f4e8164e1e8a77fc9b57bbe26fac46e4a6f893db" already present on machine
2023-02-18T10:38:18Z [Normal] Created container notebook
2023-02-18T10:38:18Z [Normal] Started container notebook
Spawn failed: Server at http://10.2.27.218:8888/user/jasonweill-jupyterlab-koupg04s/ didn't respond in 30 seconds
Spawn failed: Server at http://10.2.27.218:8888/user/jasonweill-jupyterlab-koupg04s/ didn't respond in 30 seconds
manics
February 18, 2023, 7:36pm
2
Is it related to jupyter-server 2?
opened 10:20AM - 26 Oct 22 UTC
bug
<!-- Before creating a new issue, please search for relevant issues.
-->
## … Description
If you install jupyter-server v2 as a dependency on Binder, binder will not work.
## Reproduce
You could reproduce the error locally by
1. installing jupyter-server v2, jupyterlab and notebook
2. start JupyterLab with: `jupyter notebook --NotebookApp.default_url="/lab"`
3. You are stuck on the login page
I try setting password and token to blank or to something. But it does not work. There is something about the cookie that gets cleaned I guess.
## Expected behavior
You can access the server.
## Context
- Operating System and version: debian 11
- Browser and version: firefox 98
- Jupyter Server version: v2rc3
<details><summary>Troubleshoot Output</summary>
<pre>
Paste the output from running `jupyter troubleshoot` from the command line here.
You may want to sanitize the paths in the output.
</pre>
</details>
<details><summary>Command Line Output</summary>
<pre>
Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.
</pre>
</details>
<details><summary>Browser Output</summary>
<pre>
Paste the output from your browser Javascript console here, if applicable.
</pre>
</details>
opened 02:46PM - 14 Feb 23 UTC
enhancement
### Proposed change
Recent JupyterLab, and a growing number of extensions req… uire the newer Jupyter Server, and will no longer run on the legacy `jupyter-notebook` server. But BinderHub (without auth) hardcodes the launch command as `jupyter-notebook`, and there's no way for repos to opt-in to the current standard Jupyter server (except by [redefining `jupyter-notebook`](https://github.com/jorgensd/dolfinx-tutorial/pull/113)).
At this point, I think we should switch to launching jupyter-server by default, which I think will be more compatible more often, but is a backward-incompatible change for _some_ repos, especially those that may have old server extensions. In a _binder_ context, I think there are far more lab extensions than classic nb extensions, so I think at this point it will result in _fewer_ broken repos, not more. But it's still a breaking change since which repos are broken will _change_.
xref https://github.com/jupyter-server/jupyter_server/issues/1038
### Alternative options
Make this an option somehow, and possibly opt-in (eventually, it has to be opt-out since we're currently defaulting to a deprecated package that already doesn't work with the current version of our default UI). I don't know where to express that., though, in a way that's going to work.
### Who would use this feature?
Everyone who wants to use recent JupyterLab or any other Jupyter frontends other than `notebook<7`.
### (Optional): Suggest a solution
Add to our launch command `jupyter-server` _by default_ with a fallback on `jupyter-notebook`. We could gate this on the presence of Jupyter Server 2.0, because Jupyter Server 1.0 _tends_ to be compatible with the legacy notebook server. This decision isn't really made by the user though, it's made for most repos by the repo2docker base env. And I think it would be pretty weird to say 'if you want to launch the old jupyter-notebook, install jupyter-server<2.'
jupyterhub:main ← minrk:jupyter-server
opened 02:48PM - 15 Feb 23 UTC
This will launch jupyter-server, and be far more likely to be compatible with cu… rrent repos than launching the old jupyter-notebook server.
It also makes the behavior of unauthenticated binderhub more similar to authenticated binderhub, which already does this same thing via `jupyterhub-singleuser`.
more detail in #1634
This is a breaking change, in that anything that _relied_ on running the old server (mostly legacy extensions, which _often_, but do not _always_ work) may break. But it's also a big fix, because now anything that relies on the _current_ server is not broken (i.e. JupyterLab itself). I think this will result in a _reduction_ of broken repos, but since it's a _change_, not a pure improvement, it's still a backward-incompatibility issue. It's also not great that repos cannot pick this, really, other than uninstalling jupyterlab, given how this works.
I can't seem to find it now, but I feel like we had a discussion and decided on switching the UI first (which we did a long time ago), and then switching the server at some later point, which is what's happening here. That was a long time ago, and I can't find it.
closes #1634
xref https://github.com/jupyter-server/jupyter_server/issues/1038 - this should fix issues with jupyter-server 2.
2 Likes
yerp… i’ve been sheepishly doing this in postBuild:
cp \
${NB_PYTHON_PREFIX}/bin/jupyter-lab \
${NB_PYTHON_PREFIX}/bin/jupyter-notebook
1 Like
Whoa, didn’t mean to delete that, and now running afoul of the bot. Here’s the PR:
jupyterlab:master ← bollwyvl:gh-14033-binder-hack
opened 09:37PM - 19 Feb 23 UTC
[](https://mybinder.org/v2/gh/boll… wyvl/jupyterlab/gh-14033-binder-hack?urlpath=lab/tree/_webpack_stats_.html)
[binder]: https://mybinder.org/v2/gh/bollwyvl/jupyterlab/gh-14033-binder-hack?urlpath=lab/tree/_webpack_stats_.html
## References
- fixes #14033 (sorta)
## Code changes
- `binder/start`
- [x] just remove it, rely on well-known config file locations
- `jupyter_notebook_config.py`
- [x] rename to `jupyter_config.py`
- [x] drop in in `~/.jupyter`
- `binder/environment.yml
- [x] add most of the dependencies from `pyproject.toml` to get better docker caching
- `binder/postBuild`
- [x] copy `jupyter-lab` on top of `jupyter-notebook` :vomiting_face:
- [x] explicitly call `yarn` up-front to reduce chances of deep-in-hatch failures during 200mb solve/download
- [x] invoke `pip install -e` with `-vv` to reveal potential issues
- [x] do a full minimized dev build
- [x] emit the `webpack-bundle-analyzer` report in the top level labeled with the current commit
- [x] run a ton of diagnostic commands up-front
- > there's a lot of :red_envelope: for things like `jupyterlab_pygments`, widgets, etc.
## User-facing changes
- badges displayed to PR authors and reviewers should work... more... than they do now
- PRs reviewers will see an accurate, portable ~1MB `_webpack_stats_.html` report generated by `webpack-bundle-analyzer` which can be downloaded and compared with vs e.g. `master`
## Backwards-incompatible changes
- n/a
- unless you were using `postBuild` and `start` locally, which fairly clearly nobody was
## Future Work
- collaboration features still doesn't appear to work
- presumably someone that understands what's missing will be able to debug from a running binder, so needn't block this PR
- while I usually like to point at the `gzip` size for things, this is partially a lie, as it requires rather a bit of engineering to actually realize those benefits for end users
- we should probably consider
- generating production assets that are pre-compressed
- this takes `dev_mode/static` down to 8.4mb vs 34mb
- determining the right way in backends (e.g. `jupyter_server`) to serve these with the correct headers
- without blanket turning on the built-in `gzip` feature, which has security implications
- providing tools for downstreams to do the same
Here’s that again:
@krassowski I’ve got a PR nearly up (was saturating CI with stuff, so trying to be a little cautious) that gets stuff back up and running.
The main culprit was missing jupyter-server-fileid and jupyterlab-rtc, which don’t have pretty failure modes. I was slightly off on my version locally, and some sqlite stuff failed. Ick. Of course, the collaborative stuff is still not working for some reason, but at least there’s a running server to debug against.
Will post a link when I have something, but you could try it while I’m still flailing:
Presently, as the aforementioned PRs are trying to tighten up time-to-moon-animation against a cold cache, i’m going to make this pr do a full minimized dev build, and put the webpack-bundle-analyzer report in the root of the launched binder.
1 Like
