Hello Jupyter community. I’d like to share some of my thoughts and experiences from using Jupyter in a strictly-regulated Enterprise environment for a few years now (note: I’m not talking about Enterprise Gateway, I just work in a large corporate space). I’ve talked with some of you before at Jupytercon, the Paris Dashboarding workshop, and on phone calls. @choldgraf @fperez @lheagy I wish I could have made the trip out to the west coast with Dave Stuart and company a few weeks ago to talk in person. If anyone is not a fan of really long posts, then you have my sincere apology – skip to the bottom for TLDR.
There are a myriad of topics around Enterprise Jupyter use that deserve their own threads: compute infrastructure, user education, Notebook discoverability, quality review processes, maintenance tails, etc. If I ever had the chance to give a Jupytercon talk though, it would be about the “spirit” of Jupyter use in large organizations – are we using Jupyter in the right way? What does the right way even mean? Why use Jupyter over another tool?
Background
In order to frame this discussion the right way, I need to share some context about our workplace first. We have around 10,000 business analysts/domain experts who use Notebooks right now, and that will grow. The user base falls along a spectrum of programming comfort that ranges from proficient coders to so-scared-of-seeing-code-they-immediately-close-the-tab. Most users are on the latter part of the spectrum; around 15% of the user base creates and shares Notebooks publicly (well, public-on-internal-network), the majority just run Notebooks that someone else has created. Ideally we’ll see users progress along that spectrum over time as they become familiar with Notebooks: first just running them, then editing a value here or there, then copy/pasting bits from different Notebooks to come up with something new, and finally writing whole Notebooks from scratch.
I mentioned “strictly-regulated”, which I’ll define as a workplace where every analyst uses the same tools but receive different data depending on their access. Dave went into more detail about our regulations in his 2018 Jupytercon talk. Finance and healthcare use-cases are probably identical to ours. Imagine two doctors who use the same API to pull up patient records but can only retrieve information about their patients. A Jupyter-savvy doctor could write a Notebook that does some nifty task but every other doctor would have to run that Notebook on their own to make it work with their patient set.
Our primary development/compute environment consists of per-user isolated virtual machines. We have one-click deployment of a Jupyter container built off the docker-stacks image. That container is integrated with a Notebook Gallery so users can push and pull Notebooks without having to deal with git workflows or email Notebooks around. NBGallery handles Notebook version control and sanitizing/stripping cell output (another regulation requirement).
Workflow and Widgets
With all that out of the way, let’s move on to singing praises about Jupyter. It is wildly popular at our work because it offers many things to many users, there is not just one killer feature. The low barrier of entry is appealing to newcomers and experienced programmers alike. Documenting workflows/tradecraft is extremely important to us, so the combination of markdown and code in addition to amazing nbextensions like table of contents is great. We can also build Notebooks that rival “corporate web tools” when it comes to user interface and data visualization thanks to the ecosystem around ipywidgets, pyviz, and similar efforts.
In fact, Notebooks that are de-facto webapps is one of the most significant trends I see in our workplace. For many Pythonistas in our Enterprise, it is actually faster and easier to develop a web application in a Notebook than to set up a Flask-based site. Some of that has to do with the way our infrastructure and data hosting policy is set up, but there are also really compelling advantages when it comes to debug, introspection, iterating over new features, etc. From the Notebook users perspective, user interactions such as putting in query terms or selecting date ranges feels more natural with ipywidgets than input prompts or editing code cells. Being able to programmatically change the inputs, such as populating a dropdown with user-specific profile information pulled from an API, is also super useful.
For those of you who know me, you know where this is going. A big part of our user base wants widget-focused webapp-style Notebooks for good reasons. However, writing widget-focused Notebooks tends to draw people into a coding style that undermines many benefits of working in the Notebook environment. Using widgets means putting application logic in callback functions. Besides making cells longer and making code often harder to follow, callback functions take a lot of workflow-relevant variables out of the global scope which really makes introspection and debug more difficult.
UI code can also be a distraction from the tradecraft depending on what the Notebook is trying to do. If we’re talking about a Notebook where data visualization/widget integration is the core concept, like displaying an interactive plot with a slider bar, then widget code should be front and center. More often than not in our “production” Notebooks, widgets are just being used to gather user input but are quite confusing for novice programmers to look at.
What would my ideal Notebook look like? The application logic would be linear and synchronous, with small amounts of code in each cell. I would have a table of contents or other extensions that make it easy to navigate around the Notebook and offer an at-a-glance overview of what’s happening. User input would be handled by beautiful widgets. Depending on what the Notebook is doing, it would also be straightforward to parameterize and execute programmatically as a cron, REST endpoint, papermill setup, or something similar.
I have tried to create hacks to bridge the gap between new-user-friendly widget-based callback Notebooks and linear/synchronous workflows but they only address some use-cases. I don’t have a silver bullet to this problem. If you’re interested, check them out -
-
ipython_blocking - (note: we are really in to Voila but
ipython_blockingdoesn’t work in Voila) - notebook_restified - (much more aspirational than operational here)
The Refactor Cycle
I see many Notebook authors go through a broad three-step process in their “production” Notebook development, detailed in the notebook_restified README. First, they explore their problem with a scratchpad style Notebook. Second, they clean up the Notebook by adding a narrative with markdown/comments and share it with other users for review. Third, they take the application logic from the linear/synchronous/introspective/explanatory Notebook and drop it into a callback function, or use it to build a web-app with Flask, or just pass the tradecraft/workflow/code on to another team to rewrite in some corporate analytic (java, mapreduce, etc).
That cycle would be fine if Notebook authors never had to debug user problems, add extra features, or look more deeply into the data they’re working with. When an author does need to dig deep into some “productized” refactor, what I see them do is revert back to the first steps where they’re in a linear/synchronous/minimal-code-per-cell mode. There is a lot of copy/pasting going on.
MVC
I think the heart of what I’m trying to get at is that I wish I could separate application logic and user input in Jupyter in a sort of Model-View-Controller pattern. For me, Notebooks distinguish themselves as being the absolute best tool to represent a workflow. When a workflow is packaged up as a pip installable library, or in a callback function, or as server-side web code, then it loses the things that a Notebook gives me like narrative, and introspection, and easy extensibility, and everything else.
Side note here, going back to our diverse user base – bite-sized readable code is crucial to luring code-shy domain experts down the path of learning some programming. The oft-touted benefit of using Jupyter to document and execute tradecraft is only really valid if the code is readable AND the end user can actually understand code.
Follow-on discussion topics?
Making some widget actions synchronous, such as stopping further cell execution until a button is clicked, would mitigate much of the problematic patterns I see. I understand that is a very complicated and nuanced change to make. In past discussions and reading through github issues (particularly @jasongrout comments in ipywidgets issue 1349), I get the impression that the most likely way to make this real is to move widget communications to another comms channel separate from cell execution. Is that an accurate impression?
Another mitigating idea is just changing how we distribute our end products. Maybe instead of a single .ipynb file that we share on NBGallery, we should deploy Binder-style repos which have one widget-based Notebook and one application-logic-narrative Notebook? Moving away from a single file has its own problems not least of which would be redundancy and chances of editing changes in one Notebook and forgetting to do so in the other.
TLDR
I see a pattern in our Notebook authoring community showing up over and over again where Notebook authors refine their work from a scratchpad to a mature workflow that is introspective, easy to debug, easy to extend, and includes a narrative about their work. Then they take that application logic from that Notebook and repackage it for use with a more robust user input interface like a Flask app or ipywidgets/callbacks. The repackaging ends up taking away from the benefits of the narrative style document (or forces constant refactors).
I hope this post was thought-provoking and resonates with experiences that you’ve seen or heard about in Enterprise Jupyter use. Thank you for your time, and thank you to all the core developers and contributors that built this wonderful ecosystem. I’m looking forward to any conversations that come out of this.
