Gateways and Kernel Provisioners

Kernels
1

Hello! I’ve been using Enterprise-Gateway for a bit now and have some questions about it’s future position with the inclusion of Kernel Provisioners in jupyter_client 7.0. As I’ve been operating Enterprise Gateway a desire to have multiple infrastructures, i.e. some kernels local, some in K8s, some in Yarn, all within one JupyterLab, has arisen. I see multi-gateway support on client-side is on the wish list of features for Enterprise Gateway, but as I’m familiarizing myself with Kernel Provisioners I’m wondering if that wish list item is out of date?

Please correct me if I’m wrong, but my sense is that Kernel Provisioners solves this problem. See my attached diagram for my mental model of JupyterLab + Kernel Provisioners + Enterprise Gateway

provisioners-model
provisioners-model1328×961 111 KB

I know that Enterprise Gateway doesn’t currently support Provisioners, but otherwise is this idea roughly correct? And where would Kernel Gateway fit into this picture? My sense is that it’s a 1->1 mapping between kernel and gateway, whereas Enterprise Gateway is a 1->n mapping. Enterprise Gateway seems like the easier setup networking-wise, for interacting w/ kernels from outside a given cluster.

Thanks!

2

Hey Sam did you ever make any sort of headway here in what you were trying to achieve? I’ve been looking to see if there are any solutions for what you’ve described as far as having both local and remote environments to run against at the same time but I haven’t seem to have had any luck

3

I haven’t made any headway here yet unfortunately, other things came up. I do know of this project from Google that does something similar though: notebook-kernel-mixer.

I’ve used their Vertex AI Workbenches (skinned jupyterlab) a tiny bit and they seem to use that mixer to enable both local kernels and kernels running in Dataproc (Hadoop) Clusters through Enterprise Gateway. Conceptually, what they’re doing seems to be very similar to the promise of Kernel Provisioners too.

This repo also has some code related to kernel provisioners and gateways but I haven’t had time to dig into it yet: gateway_provisioners