Julia community is creating a new notebook format

+1 to reaching out to David to hear his take on it. TBH I am a bit surprised that the Julia community decided to go all-in on a platform that is controlled by one company, but ¯_(ツ)_/¯

@sgibson91 I think it’s worth disentangling a few different “aspects” of open source (as I understand it) and how each relates to VSCode.

Open components - in this respect, VSCode does decently well. Many of the underlying components (Monaco being the most well-known example) are openly-licensed. However, as the JupyterLab team learned when they tried to integrate Monaco with JupyterLab, many of these components make assumptions about VSCode infrastructure being present, so aren’t designed to be super easily/readily reusable elsewhere. That said, I think when people say “VSCode is open source” this is really the only thing that it appliesto.

Open governance - in this respect, VSCode does poorly. There is no open governance of VSCode development, nor of the open source components underneath. Those are all Microsoft products and driven by Microsoft roadmaps/PMs/etc. As @matthew.brett mentions - right now Microsoft has decided to align itself with open source (and making a strong marketing push to brand VSCode as “open source” for various reasons). What will that look like in 1/2/5/10 years? History says that relying on a gigantic company with monopolistic tendencies is an anti-pattern if you care about public goods and collective power.

Open products - in this respect, VSCode also does OK but not great. VSCode is a proprietary distribution that contains a collection of open source and proprietary components. Sure you can fork VSCode and keep only the open source parts (that’s what VSCodium is). But then you realize that there are actually a lot of proprietary things interwoven in VSCode in subtle ways. E.g., as others have mentioned the “real time collaboration” API is proprietary, and many of the extentions that ship with VSCode are proprietary. As time goes on, I suspect VSCode will change the ratio of “open source to proprietary” extensions within VSCode such that the “open fork” versions will become relatively more underpowered.

A good example of the tensions that can arise with this are actually related to Julia notebooks. In the PR where they propose creating jlnb files, David actually mentions that one reason they’re considering .jlnb is because the .ipynb format in vscode is managed by the vscode-python team. In other words, Jupyter’s vision of a truly multi-language and multi-stakeholder layer of infrastructure is potentially going to be fractured because a product team inside of Microsoft is building “Jupyter” tooling that is too Python-specific, and on a platform that is becoming so popular that it is influencing decisions about Jupyter tooling in a way that Jupyter has no say into.

Or to put it shortly if VSCode were translated into the Jupyter ecosystem, it’d be like if:

  • JupyterLab was a product of a single large company called The Jupyter Corporation
  • This company also ran a totally proprietary version of JupyterHub on its own cloud
  • This company also controlled the most popular platform for doing open source development, and could use that platform to encourage the use of these products above other options.
  • All development of core JupyterLab components, APIs, and key extensions were done by this company
  • JupyterLab shipped the pip install jupyterlab version with a bunch of proprietary extensions pre-installed
  • JupyterLab encouraged communities to fork it and create their own JupyterLabs if they didn’t want the proprietary stuff
  • JupyterLab encouraged a lot of open source development and experimentation within jupyterlab’s ecosystem, but where the ecosystem’s platform was controlled by The Jupyter Corporation.

Don’t get me wrong, I love VSCode and use it all the time. From an engineering perspective it is super impressive, and I also appreciate all of the effort they’ve put into making it very extensible (and easy to extend). And I should also be clear that all of this is totally Microsoft’s prerogative, there’s no reason that they can’t operate in this way, it is IMO in their interests as a company. I just think we should be realistic about it’s role in the arc of “the open community” and its relationship to research and education. I think it’s not realistic to think of Microsoft as “a friend of open source” but more like “temporarily aligned with the interests of open source” - it may be true now but the last 3 years are a tiny blip in the history of Microsoft’s culture. Whether it is Microsoft/Google/Amazon/etc we want the infrastructure and platforms in our community to be governed by the commons, not by any single entity that doesn’t have a properly open and multi-stakeholder governance model.


note: Theia is I think an interesting project here, because it basically re-uses a lot of the VSCode components but packages them in a distribution that is also open source and designed for composability and reusability. That distribution is managed by the Eclipse Foundation, which is a multi-stakeholder group. I’d feel a lot more comfortable if we found ways to empower the Theia/Eclipse community, rather than to work within the sandbox that Microsoft gives us.

2 Likes

Thanks for the info everyone. To be clear, I answered the question by saying “I think there’s a reason we don’t do this, but I can’t remember what it is right now”. This is obviously it! (I actually think a VSCode extension that automatically gives you a Binder kernel would be more interesting than us serving the VSCode UI.) Given VSCode’s popularity, it’s good that we have a party line to tow and now I know what it is!

And now I’ll stop hijacking the thread about Julia’s new notebook format :slight_smile:

2 Likes

FWIW - I am not trying to give the party line here, just giving my perspective. As I mentioned, I also love VSCode and use it all the time, so I am conflicted here. Binder should be facilitating the workflows that make research and education most-impactful in the ways that align with our values. Maybe VSCode is a part of that, but I just want us to have maximal information as we make those choices, thus my wall-of-text above :slight_smile:

As another example: we also offer RStudio, which is similar to VSCode in its hybrid “open source but produced entirely by one company” nature. We have to get permission from RStudio Inc to use the trademarks etc, and yet we still offer it to users. I think VSCode is in a similar category.

1 Like

Hi everyone! @sgibson91’s excellent session at Juliacon last week made me think that it would be good to have some more contact between the Julia VS Code team and the Jupyter effort, so I browsed around and the first thing I see is this thread :slight_smile: So here I am, and I think it is a great idea to connect!

I should clarify probably one thing right away: we have not made a decision to create our own notebook format. We have a PR that implements a native notebook editing and run experience for Julia in VS Code and it uses a custom file format, but the whole thing is WIP at the moment and whether we will go with a custom format or not is up in the air right now. Certainly a possibility, but a the moment we are much more focused on getting the actual functionality of the notebook UI up and running, rather than focus on the the file load and save story.

But one thing is becoming pretty clear in this process: the notebook UI we will end up with is going to be much richer than what Julia users will have in JupyterLab. We’ll have full language server, debugger, workspace viewer, plot pane, documentation browser etc. support from day one because we built these things pre-notebook days and they can now be integrated with almost no extra work into the notebook UI. I’m a huge JupyterLab fan, but when I recently realized that we will probably have the premier notebook UI for Julia in VS Code and not JupyterLab, it did give me pause and made me wonder about the larger strategic implications for JuptyerLab… I say “recently realized” because there was very little strategy behind any of this on our end, it was more that all of this fell into place when MS started to work on the notebook API (which btw. is entirely distinct and unrelated from the existing Jupyter notebook support in the Python extension for VS Code).

In any case, I think some generic meeting would be great. I could report a bit on what our plans are on the Julia extension side of things. I also saw that there were various discussions ongoing about Theia, and how Jupyter might in general be able to integrate more with that or VS Code directly. I could probably say a few things about that as well. In particular I’m just very, very familiar with the VS Code extension points, which might be useful for that discussion. Could one of you take the lead in setting up such a meeting? I really only know @sgibson91 and @betatim from this thread, so wouldn’t really know who should be invited. My email is anthoff@berkeley.edu.

Oh, and we just had two talks at Juliacon about the state of the Julia VS Code integration, that might be a quick way to get a sense of the functionality we have. Video 1 is https://www.youtube.com/watch?v=rQ7D1lXt3GM (there is a very short demo of the notebook PR in there) and video 2 is https://www.youtube.com/watch?v=IdhnP00Y1Ks.

4 Likes

A good starting point may be our monthly team meeting, or do folks want something separate?

1 Like

I’d prefer something separate, and invite JupyterLab and Jupyter Notebook people, as well as post on the ongoing notebook format thread elsewhere.

2 Likes

Thanks @davidanthoff for reaching out! I’m excited to hear about how the Julia community has been navigating these issues, and where they see this space moving forward.

I agree with @jasongrout that it’d be good to have this meeting in a place that the JupyterLab/Notebook communities can interact, ask questions, etc. That’s probably the part of Jupyter that has thought the most about these kinds of issues (and also probably the most likely intersection point?)

I would be very interested to join such a meeting, though can’t be the one to organize (and may not be able to attend) because I’m expecting a baby girl literally any day now. @davidanthoff and others if you want to hop on a short call this week I’d be happy to do so just to touch base, but if you’ve only got one meeting in you, I’d recommend it be one w/ the broader community and I can always check the notes.

I’ll try to think about my main questions around VSCode/Theia/Julia/Microsoft and will post them here in case it sparks ideas or discussion.

@choldgraf Good luck with that project, and get some sleep while you still can :slight_smile:

I think having a meeting this week would be good, it would be great if @choldgraf could be part of that. I certainly would be up for another one later on.

If someone sends me some emails of folks that should be part of that, I’ll send a scheduling poll around.

2 Likes

Thanks folks for the details! I’d be very happy to join such a discussion as well - @davidanthoff if you’re looping people in for scheduling, feel free to use my campus email…

I would also like to be a part of this discussion, if you schedule it! s.shanabrook@gmail.com

thanks :grimacing:

I’m free most afternoons this week. Shall we pick a time and invite whoever is available to make it then, with a plan to have a broader lab/notebook community meeting later on? (I assume it’ll be complex / take time to organize between all of these folks for this week). I don’t want to limit the conversation to just my schedule, I’m just worried that if it’s not this week, I’ll probably be unavailable until like October :baby:

Here is a link for anyone to sign up for a meeting this week and indicate your availability:

https://zvite.co/i/Wwmm5v7l

I’ll pick a time either tonight (if enough folks have responded) or early tomorrow (Mon) morning.

2 Likes

Thanks for the scheduling link! I signed up: co-maintaining jupyter[lab]-lsp, and interested in some of the other gotta-haves mentioned, like deeper docs integration.

2 Likes

Please loop me in on the meeting too. I’m a long time Julia lover (in addition to Python) after I saw some high school students use it in biotech projects.

It would be great to have a meeting now and after for @choldgraf who I hope will enjoy every moment of his baby daughter coming into this world.

Please add me as willingc@gmail.com.

1 Like

@MSeal @captainsafia Passing along as an FYI.

Thanks everyone for responding to the poll! I scheduled the meeting for Wed 10:30 pacific time, which miraculously works for everyone who responded. Zoom meeting details are in the calendar invite that you should all have received.

If anyone wants to join who didn’t respond, just send me a quick email at anthoff@berkeley.edu, and I’ll add you.

Here are two more links that have some background info that might be of interest for tomorrow’s call:

The official notebook API for the upcoming notebook support in VS Code.

The discussion about Jupyter notebooks in particular on the Python extension repository.

Thanks for those links - I will take a look.

I am curious if you have any helpful links to discussions around VSCode governance and ownership. To me the biggest questions are not whether VSCode provides the right APIs for building UIs, but around the fact that it’s a product from one company rather than a multi-stakeholder or community-led project. I am interested to learn more about how the Julia community feels about this and how it pertains to their mission and values.

I’m also curious what is the relationship between VSCode and Theia, as I see the latter as (potentially?) a project more aligned with the values of Jupyter than VSCode, and potentially an intersection point. But, I must admit I haven’t been able to find out much about Theia’s governance model.

I think that (for me) understanding these more will help me figure out how it is possible and/or how it is most-productive to engage on the VSCode side.

I think we never had any discussion about the stakeholder question when it comes to VS Code in the Julia community. There have always been efforts under way to add Julia support for pretty much all editors out there, and it just happens that the Julia extension for VS Code at the moment seems to appeal to the largest number of users and is probably the feature richest (at least of the actively developed ones). There really isn’t much of a strategy going on here, other than “lets provide a good Julia experience in every editor and environment”, and people then flocking to the solution that provides them the best experience :slight_smile:

I can report a bit back tomorrow how the relationship with MS has played in practice for us.

1 Like

I think this is a good way of approaching things. I feel like Jupyter should adopt a similar mentality, and unfortunately thus far we haven’t had any connections with the MS developer teams working on notebooks (or even just ipynb specifically). I’d be curious if you think there are productive pathways for us to engage with those folks.