Installing extensions extremely slow

JupyterLab
1

Hi everyone,

I recently updated JupyterLab to 1.2.1. As I was adding some extensions, I noticed that the building of them was really slow, like 20 minutes on my T495 Ubuntu machine.

I installed JupyterLab through conda

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
appdirs                   1.4.3                      py_1    conda-forge
attrs                     19.3.0                     py_0    conda-forge
backcall                  0.1.0                      py_0    conda-forge
black                     19.10b0                  py37_0    conda-forge
bleach                    3.1.0                      py_0    conda-forge
bzip2                     1.0.8                h516909a_1    conda-forge
ca-certificates           2019.9.11            hecc5488_0    conda-forge
certifi                   2019.9.11                py37_0    conda-forge
click                     7.0                        py_0    conda-forge
colorama                  0.4.1                      py_0    conda-forge
decorator                 4.4.1                      py_0    conda-forge
defusedxml                0.6.0                      py_0    conda-forge
entrypoints               0.3                   py37_1000    conda-forge
icu                       64.2                 he1b5a44_1    conda-forge
importlib_metadata        0.23                     py37_0    conda-forge
ipykernel                 5.1.3            py37h5ca1d4c_0    conda-forge
ipython                   7.9.0            py37h5ca1d4c_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.15.1                   py37_0    conda-forge
jinja2                    2.10.3                     py_0    conda-forge
json5                     0.8.5                      py_0    conda-forge
jsonschema                3.1.1                    py37_0    conda-forge
jupyter_client            5.3.3                    py37_1    conda-forge
jupyter_core              4.5.0                      py_0    conda-forge
jupyterlab                1.2.1                      py_0    conda-forge
jupyterlab-black          0.2.1                    pypi_0    pypi
jupyterlab_code_formatter 0.7.0                      py_0    conda-forge
jupyterlab_server         1.0.6                      py_0    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0  
libsodium                 1.0.17               h516909a_0    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0  
libuv                     1.33.1               h516909a_0    conda-forge
markupsafe                1.1.1            py37h516909a_0    conda-forge
mistune                   0.8.4           py37h516909a_1000    conda-forge
more-itertools            7.2.0                      py_0    conda-forge
mypy_extensions           0.4.3                    py37_0    conda-forge
nbcommands                0.3.2                      py_0    conda-forge
nbconvert                 5.6.1                    py37_0    conda-forge
nbformat                  4.4.0                      py_1    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
nodejs                    12.13.0              h10a4023_0    conda-forge
notebook                  6.0.1                    py37_0    conda-forge
openssl                   1.1.1c               h516909a_0    conda-forge
packaging                 19.2                       py_0    conda-forge
pandoc                    2.7.3                         0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
parso                     0.5.1                      py_0    conda-forge
pathspec                  0.6.0                      py_0    conda-forge
pexpect                   4.7.0                    py37_0    conda-forge
pickleshare               0.7.5                 py37_1000    conda-forge
pip                       19.3.1                   py37_0    conda-forge
prometheus_client         0.7.1                      py_0    conda-forge
prompt_toolkit            2.0.10                     py_0    conda-forge
ptyprocess                0.6.0                   py_1001    conda-forge
pygments                  2.4.2                      py_0    conda-forge
pyparsing                 2.4.4                      py_0    conda-forge
pyrsistent                0.15.5           py37h516909a_0    conda-forge
python                    3.7.3                h33d41f4_1    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
pyzmq                     18.1.0           py37h1768529_0    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
regex                     2019.11.1        py37h516909a_0    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                41.6.0                   py37_1    conda-forge
six                       1.13.0                   py37_0    conda-forge
sqlite                    3.30.1               hcee41ef_0    conda-forge
terminado                 0.8.2                    py37_0    conda-forge
testpath                  0.4.4                      py_0    conda-forge
tk                        8.6.9             hed695b0_1003    conda-forge
toml                      0.10.0                     py_0    conda-forge
tornado                   6.0.3            py37h516909a_0    conda-forge
traitlets                 4.3.3                    py37_0    conda-forge
typed-ast                 1.4.0            py37h516909a_0    conda-forge
typing_extensions         3.7.4.1                  py37_0    conda-forge
wcwidth                   0.1.7                      py_1    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.33.6                   py37_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zeromq                    4.3.2                he1b5a44_2    conda-forge
zipp                      0.6.0                      py_0    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge

and I am installing the extensions through the following bash script

# Choose conda and load the virtual environment
CONDA_BASE=$(conda info --base)
source $CONDA_BASE/etc/profile.d/conda.sh
conda activate jupyter-lab

# Node JS for the extensions
conda install -c conda-forge nodejs
export NODE_OPTIONS=--max-old-space-size=4096

# nbcommands
conda install -c conda-forge nbcommands

# Jupyter Lab code formatter with black
conda install -c conda-forge black jupyterlab_code_formatter
jupyter labextension install @ryantam626/jupyterlab_code_formatter
jupyter serverextension enable --py jupyterlab_code_formatter

# Jupyter widgets extension, FigureWidget and renderer support
jupyter labextension install @jupyter-widgets/jupyterlab-manager@1.0 --no-build
jupyter labextension install jupyterlab-plotly@1.0.0 --no-build
jupyter labextension install plotlywidget@1.0.0 --no-build

# Go To Definition extension
jupyter labextension install @krassowski/jupyterlab_go_to_definition --no-build

# Spell checker extension
jupyter labextension install @ijmbarr/jupyterlab_spellchecker --no-build

# Build extensions
jupyter lab build

unset NODE_OPTIONS

I don’t remember anything particularly slow prior to the update. Right now, the installation lags at jupyter labextension install @ryantam626/jupyterlab_code_formatter or jupyter lab build.

Is the script flawed in some way? Would it be related to this post?

Thanks in advance for your help!

1 Like
2

I’m just guessing, could it be a network issue? If you run jupyter lab build --debug is there a timeout happening somewhere?

3

I’m seeing the same problem. Extremely slow build time for the jupyterlab extensions.
I’m running JupyterLab v1.2.3
I only have three extensions to add, and am running the following (as per the plotly documentation):

# Jupyter widgets extension
jupyter labextension install @jupyter-widgets/jupyterlab-manager@1.1 --no-build

# jupyterlab renderer support
jupyter labextension install jupyterlab-plotly@1.3.0 --no-build

# FigureWidget support
jupyter labextension install plotlywidget@1.3.0 --no-build

# Build extensions (must be done to activate extensions since --no-build is used above)
jupyter lab build

Not only is it slow, but it then crashes with an error 137

4

The hanging seems to be taking place when installing jupyterlab-plotly.
Running wigh --debug shows:

> /usr/bin/npm pack jupyterlab-plotly
\npm notice
npm notice 📦  jupyterlab-plotly@1.3.0
npm notice === Tarball Contents ===
npm notice 1.0kB package.json
npm notice 165B  README.md
npm notice 6.3kB dist/javascript-renderer-extension.js
npm notice 545B  style/index.css
npm notice 1.0kB style/plotly.svg
npm notice === Tarball Details ===
npm notice name:          jupyterlab-plotly
npm notice version:       1.3.0
npm notice filename:      jupyterlab-plotly-1.3.0.tgz
npm notice package size:  3.1 kB
npm notice unpacked size: 9.0 kB
npm notice shasum:        bb648b175b99b6a3ac4784e00372e1056b625f1c
npm notice integrity:     sha512-7goT3ELiFe1yO[...]Tr5fr1fGlOxDg==
npm notice total files:   5
npm notice
jupyterlab-plotly-1.3.0.tgz

Yarn configuration loaded.
Node v12.13.0

Building jupyterlab assets (build:prod:minimize)
> node /usr/local/lib/python3.7/site-packages/jupyterlab/staging/yarn.js install --non-interactive
-yarn install v1.15.2
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
info fsevents@1.2.9: The platform "linux" is incompatible with this module.
info "fsevents@1.2.9" is an optional dependency and failed compatibility check. Excluding it from installation.
[4/5] Linking dependencies...
[5/5] Building fresh packages...
success Saved lockfile.
Done in 7.19s.

> node /usr/local/lib/python3.7/site-packages/jupyterlab/staging/yarn.js yarn-deduplicate -s fewer
/yarn run v1.15.2
$ /usr/local/share/jupyter/lab/staging/node_modules/.bin/yarn-deduplicate -s fewer
Done in 0.63s.

> node /usr/local/lib/python3.7/site-packages/jupyterlab/staging/yarn.js run build:prod:minimize
/

And it hangs there forever before finally timing out

5

I am getting the distinct feeling this is related to plotly. There is some on-going work to resolve some of the likely root cause issues, namely the large size, and potential duplication, of the plotly distribution which gets injected into the main bundle along with lab itself.

This is partially on the Lab side, for making it possible for extensions to break the build so hard, and we’ve been doing what we can to try to alleviate some of the issues that core creates itself. It’s partially up to extension authors to be mindful of the total impact of their dependencies on all of their end users.

In the near term, an effective pattern is, where possible, to use await import for very large packages, even if it means making wrappers and whatnot, and trying to delay the loading of these packages until they are actually asked for by the client, e.g. actually plotting a plot. Aside from helping Lab start faster, it means that the high-water mark of memory used during build is reduced.

There are some additional techniques we could employ, which we have been thus far avoiding, as they would multiply the number of individual files downloaded ten-fold, and HTTP 1.1 is not so great at that kind of request work load. So we’ll have to try to find a happy medium.

In the further future, we’d very much like to get to where we can handle the simultaneous challenges of proper deduplication of assets served to the browser (which is why we went with current webpack approach), and the efficient distribution, installation and serving of extensions without yarn and webpack on end users machines (favoring pip, conda, or system package managers), while still allowing developers to use the millions of packages on npm… but that’s a pretty advanced research topic at this point.

1 Like
6

Given that the root cause may well be plotlywidget, one thing you can try is running the build without minification:

$ jupyter lab build --minimize=False

That will explode your build size, so if you’re running on a remote server then this will cause browser load times to increase. However, if you’re running JupyterLab locally (either same-machine or same-network), then you shouldn’t feel very much impact at all.

7

Thank you all for your help and insights. It seems to me that the lagging was not only from plot.ly. I tried the --minimize=false option and, apparently, it reduces the setup time.

8

Perhaps unrelated, but the CLI parsing here is case-sensitive which means --minimize=False works but --minimize=false does not. At least it fails with a descriptive error, but I would have expected that either should work.