Breakout session brainstorming

JupyterHub HPC meeting 2019
1

This thread is for brainstorming topics for breakout sessions.

In the end, we aim to coalesce around 4-5 topics. Let us consider suggestions here and then form separate Discourse Topics for each.

I’ll start with one topic on my mind:

Notebook sharing and distribution models

Including:

  • ways for users to pass notebooks around that’s better than download/email/upload
  • ways for scientists at user facilities to distribute examples and templates to their users
  • ways for scientists at user facilities to “jump in” to a user’s server and provide support
1 Like
3

Security

  • JupyterHub has the ability to run with end-to-end SSL if your spawners support it. What spawners currently in use at HPC and user facilities are configured this way and can we see examples of how to handle it?
  • How can we enable more detailed audit? There seem to be places in notebook and terminado that could be used to log commands to syslog; there’s at least one potential PR about this in notebook I think. Is it feasible to identify all these points where code is going to kernels (in whatever language) and log those?
4

Continuous Operations

  • How can we keep JupyterHub available to users all the time (or as much as possible)?
  • How do we notify and alerts users to changes?
  • How do we do rolling upgrades?
  • How can we do A/B testing?
  • What else?
5

Jupyterlab + Jupyterhub integration

There are some rough edges when trying to run Jupyterlab in a shared Jupyterhub environment. Start a discussion about solving some of these issues, including

  • extensions need to be installed globally. how to support local user installed extensions
  • jupyterlabhub UI quirks
  • “home” dir access
6

Jupyterlab extensions

Ideas for JL extensions that could solve some of our common problems

1 Like
7

Some extension ideas:

  • An extension to track HPC system activity - job state, run time, queues, etc.
  • Tools to interact with jobs - our (DoE KBase) users have a range of backgrounds and needs, and just giving terminal access doesn’t always serve them best.
  • Maybe an activity dashboard?
8

Improvements & Latest on SLURM spawner

9

I think there may be a “Binder for HPC resources” i.e. how do you implement Binder-like functionality with a more traditional HPC backend without some of the support for kubernetes etc.

10

Related: Would a "The Littlest Binder" be useful?

11

Containers and Jupyter

  • Using containers to provide more customized environments
  • Deployment models
  • Integration with JupyterHub and Spawners
12

Expanding on this, as I said in my talk, I would very much like to have a round-up on:

Status and future direction of batchspawner and friends

  • evaluate needs and pain-points for current and prospective users
  • strategies for sustainable development
1 Like
13

Jupyter and remote execution models

What are the patterns, solutions or prototypes for spawning remote tasks from a notebook

  • Existing models (e.g. Dask)
  • Lamda functions
  • code cell to remote execution
  • Can containers help here?
  • Interaction with batch/resource managers
14

Reproducibly and Jupyter

  • Where does Jupyter help?
  • Where are there gaps (e.g. code cell order)
  • What is needed outside of Jupyter
  • What is the state-of-the-art?
1 Like
15

(from Shane Canon, who hit the post limit)

  • How should extensions interact with each other and the notebook canvas?
  • Are there interface gaps or missing standards?