These are just some notes, mainly for myself, on where the JupyterLab build system is today, what problems it is trying to solve, and alternative ways we could solve those problems. Very much in progress!
To be clear this isn’t a roadmap for where JupyterLab is heading, just some directions I am personally interested in exploring.
What I would like to try is articulating the different ways people need the JupyterLab build system, to see if maybe we could serve the different needs with different tools.
The first use case is a user installing a JupyterLab extension. They maybe have seen a github repo advertising some fancy new feature of JupyterLab, and they wanna try it out. I would guess their first goal is try out the extension and they don’t care as much about bundle size.
We also have someone working on a feature in JupyterLab core. They want to be able to edit code and quickly have the build rerun so they can see the results.
Then there are the third party extension developers who are like those working on core, but they are just working on an extension.
Finally, we have the people who are trying to serve a custom build of JupyterLab for their colleagues. They care about optimizing the build so they serve the smallest bundle possible.
Issues and Solutions
- IMHO this is the most important use case, because it could be someone’s first experience with JuptyerLab. If it feels confusing or doesn’t work for them, we risk pushing them away. We keep getting issues opened around build oddities (https://github.com/jupyterlab/jupyterlab/issues/6102, https://github.com/jupyterlab/jupyterlab/issues/5177), and they can be pretty hard to diagnose. One reason I think they are particularly confusing is that users often just want to install an lab extension, and they are thrust (unknowingly) into the world of Webpack, NPM, Yarn, Node, etc. We try to wrap these things for them, with our own
jupyter labcommands to install extensions and trigger the webpack build. One way to make this simpler is if we didn’t require a Webpack build after installing an extension: https://github.com/jupyterlab/jupyterlab/issues/5672
- From my perspective, this works OK at the moment. Changes are relatively quick to be loaded in the browser. However, if the JS build process was seperating from the python server, then we could play with things like Webpack’s Hot Module Replacement more easily to speed up the dev process.
- For the basic case, we do have a workflow here, but if you wanna do more complicated things, like link a local dependency of your extension (https://gitter.im/jupyterlab/jupyterlab?at=5c911975fcaf7b5f73de5a98) then it gets a bit tricky. The key issue is that you not only have to understand Yarn’s linking system but also how it integrates with JuptyterLab’s build system. If we could separate those things, so that you could look at the JS builds in isolation from the Python server process, then this could help conceptually clear up the matter. For example, if the Python process exposed an API, and the JS builds created a static web app, then you could build it however you liked as long as it could connect to the Python server.
- There has been some work to do this already, with the
jupyterlab_deluxcookiecutter: https://github.com/jupyterlab/jupyterlab/issues/6090. Basically the idea is to be able to create a conda package that contains JL plus some preinstalled extensions. So that users who install it don’t have to build jupyterlab to get the packages. There are some continued issues with this approach however (https://github.com/jupyterlab/jupyterlab/issues/6132)
Possible next steps
Separate Python serving from JS building.
- Have JupyterLab JS build as static web app.
- Be able to launch this with a simple file server, and point it to the server URL, with a token.
- Launch the server separately as a Python process.
This would allow you to conceptually separate the two things and understand better how they interact.
Installing extensions without webpack
The basic idea here is we prebuild extensions to bundle all of their logic in one JS files. Then, we use web modules to import that JS file from the user’s browser, instead of bundling with Webpack.
- Prebuild extensions with webpack so that they are already minified and include all their dependencies, besides those JupyterLab and Phosphor packages, and any other that need to be singletons and we know will exist.
- Specify, in some config option, a list of extension URLs. All of these are dynamically imported and should return a default export of a list of extensions (like a normal jupyterlab extension JS package should).
- On startup, instead of building the installed extensions into the JS bundle, we load this list dynamically, and import each extension. We enable every extension exported by these files, except those that are disabled.
This was before my time in JupyterLab, but this has already all be tried (https://github.com/jupyterlab/jupyterlab/issues/728). One difference was that before web modules weren’t a browser standard and they were relying on require JS. The idea here isn’t to make this THE way of serving plugins to JupyterLab, just to allow this to be A way of doing it, if you so choose (yes there are tradeoffs in ensuring the write global versions and shipping many requests to the browser).
What would we need for this?
- We can already use webpack to build a library that includes all it’s dependencies besides some common packages.
- I need to investigate how to do this, then package the libraries you left out of the package into a separate module, and have webpack use that module, when it is imported…
Maybe webpack already can do this (emphasis mine):
The runtime, along with the manifest data, is basically all the code webpack needs to connect your modularized application while it’s running in the browser. It contains the loading and resolving logic needed to connect your modules as they interact. This includes connecting modules that have already been loaded into the browser as well as logic to lazy-load the ones that haven’t.
So it already has a runtime it uses to match up imports to what JS libraries they match with. Ideally, we want one file for core phosphor/jupyterlab packages, and then one for each other extension. Maybe this “webpack runtime” can track "Oh this extension needs to import
phosphor/algorithms and we have already loaded this in our core module, so let’s provide it.