Governance Questions 2: Teams and Projects

Meta Governance
1

This is the second community input post covering the group of issues surrounding teams and projects within Jupyter and the governance questions that they raise. Here is the first post, covering community questions.

Please take a look at the following questions and answer them however you see fit. You can answer all or part of them, and either as a response to this thread or as a separate post to link to.

Community questions: Teams and projects

  • How do we create project unity while supporting autonomous, smaller teams?

  • What is the official list of Jupyter projects and how do new projects become part of Jupyter? How are projects conducted and governed?

  • What are the “shared resources” of Project Jupyter and how are they allocated among the projects?

Note - this post was edited by @choldgraf for structure and clarity

4 Likes
3

For me Project Jupyter is a group of people who share a common vision or otherwise like working together to create things they couldn’t create by themselves.

The people define what Project Jupyter is and what it does. That some of this activity involves software and github repositories is an “implementation detail”. In particular because there are things that Project Jupyter does (or wants to do) that don’t involve any code (conferences, sprints, standards, protocols, teaching, etc).

How do we create project unity while supporting autonomous, smaller teams?

Create more opportunities for people to interact as people with each other. The goal being for them to become closer friends that trust each other more.

Right now a large part of the mission/vision is unknown to a part of the people in the project, yet collectively the project is making progress on the big goal. Maybe minimally tweaking the current process and sharing ideas more is all that is needed.

re: independent teams: each of the large areas of Jupyter are taken care of by separate teams and because each of those areas could be its own (or several) projects by themselves it is nearly impossible for someone to be involved in several areas. This is because of limited time but also skills and interests. For me personally that is a-ok because those who take care of the other areas are doing a good job. All I could offer is the opinion of an interested passer by. Maybe at the afore mentioned events we could organise retros where different teams ask for feedback on what they are doing, things that other teams like/dislike about getting involved, etc.

What is the official list of Jupyter projects and how do new projects become part of Jupyter? How are projects conducted and governed?

Easier one first: a set of cookie cutter LICENSE, CoC, CI, documentation and copyright statements as the minimum standard for new projects. This would help unify things (easier to move around), help developers with tedious tasks and set standards. Something like https://github.com/scikit-learn-contrib/project-template maybe? Can GitHub repos also be used to represent projects like organising a conference or other projects that aren’t about code? If not how are they represented and what tools do they use that let people contribute?

A slightly radical proposal: there are no official jupyter projects. There are projects that a significant number of jupyter team members work on. This is because Jupyter is about people and standards/protocols not code.

Maybe projects could be “Jupyter certified” if they are standards compliant or provide implementations of the protocols that Jupyter defines. By having a badge like that the project could exercise something it has: influence.

Making a project a “official jupyter project” means that the project is now maintained by Project Jupyter. This means that Project Jupyter promises to invest effort into the project. However Project Jupyter has no resources (effort that can be directed by Project Jupyter) of its own with which to full fill this promise. The only effort available to Project Jupyter is that of (self-directed) volunteers. The people in Project Jupyter can try to influence each other, create a share understanding and motivate each other to help maintain projects. There are three main sources of effort: grants, companies and volunteers. Neither of them is primarily interested in chopping wood and carrying water. They will however do a little bit on the side to help maintain the basic infrastructure while working on a project they actually care about/can obtain funding for.

What are the “shared resources” of Project Jupyter and how are they allocated among the projects?

(this is written about a future project jupyter not the current one)

Project Jupyter has assets that include influence, branding and cash.

Influence can be used by creating a “jupyter compliant” badge or similar. This gives some of the shine of Jupyter to other projects. The people who make Project Jupyter wield (sometimes considerable) influence in the world, by being a member of the Jupyter team they are responsible to not abuse this. For example when are they speaking for the project and when as private individuals/employees? I think it is hard to put in writing/create precise rules for this but we all “know it when we see it”. Maybe the best way forward is to collect positive and negative examples to help people learn where the line is. Maybe also a way for team members to give each other feedback on how they perceived other’s behaviour. If in doubt don’t use your Jupyter association.

An aside on standard practices in large scientific collaborations:

In large scientific collaborations there are strict rules about giving talks/public statements/publishing papers that are related to your membership in a collaboration. They require significant effort though so might be too much for a loose project like Jupyter. The standard practice is that the collaboration submits abstracts to conferences (not individuals) and then distributes accepted talks to members of the collaboration. The precise formula/method used varies but generally it takes into account seniority, job market needs, topic fit, number of talks given in the preceding X months, etc). The goal is fairness amongst members of the collaboration.

Only talks distributed via this mechanism are “on behalf of the X collaboration”.

Talks (including slides) are approved by the collaboration before the talk is given via a rehearsal that is open to everyone in the collaboration, slides are posted for everyone in the collaboration to see. Content is mostly dictated by the abstract that was submitted.

External seminar or conference organisers should not contact individuals when looking for (keynote) speakers but instead contact the collaboration. There are exceptions for local/low key seminars.

In all of the talks only material that is already public can be discussed. On contentious results there is a official position from the collaboration that has to be represented by speakers.

Individuals can give talks outside of this framework but only if they do not (at all) refer to work or knowledge related to their membership in the collaboration. Some individuals try and stretch these rules (too far) which is considered a social faux-pas and repeated offences lead to serious consequences. However this is rare.

Branding is controlled by the branding guidelines and trademark rules.

Cash is controlled by the steering council. Donations that arrive via Numfocus are spent by the SC as they see fit. Larger donations like those that fund the community workshops would be controlled by a committee that the SC delegated its power to.

1 Like
4

Some thoughts from the governance team

These are a few extra pieces of information, contextual details, and brainstorming around the
questions above.

How do we create project unity while supporting autonomous, smaller teams?

  • We have (approximately annual) steering council in-person meetings, where a lot of the activities center on overall project philosophy and building trust across participants. Given the size of the steering council, it is hard to find a time that everyone can attend, and often there are a number of people that aren’t there. We also have a number of non-steering council community members at these in-person meetings who fully participate in the discussions and decisions.

  • The project’s vision is one of building open tools for interactive computation that benefit society at large, with an emphasis on research and educational questions but an open model that explicitly includes and welcomes close industry partnerships as well as a large, distributed volunteer community. Given such diverse stakeholders, each with different short- and long-term goals and constraints, as we’ve grown, our governance has encountered complex challenges. We try to maintain, mostly through communication, a sense of shared mission and vision. However, when specific decisions regarding direction or resource allocation need to be made, it can be challenging to express those more abstract ideas as concrete decisions that rally agreement from the various teams in the project (see next point).

  • We do not have an overall unified project roadmap. Being a volunteer-based organization, we often punt tactical roadmap decisions to smaller subprojects. However, we should have trust among each other and a way to reconcile differences amicably. At times we have struggled with this, and at times members of the community or leadership have disengaged because project roadmap disagreements were not resolved.

  • Traditionally, the project has had a very strong mission and vision, but it wasn’t shared publicly or visible to everyone. The current language on the Jupyter website, “Project Jupyter exists to develop open-source software, open-standards, and services for interactive computing across dozens of programming languages,” comes close to capturing this, but is still probably incomplete.

    • Work began in 2019 to codify this, that will play an important role in creating project unity through a shared set of principles. The initial work done on that was very encouraging in that there was a relatively strong consensus about our shared values.

  • It is difficult for users and organizations to follow the various activities of the project, and we historically have not done a good job of signalling an overall roadmap. Even for those of us within the organization that follow multiple GitHub repositories, it is easy to miss the forest for the trees. What might help with this is a single place to look that is updated on a regular basis that distills the current activities and overall direction of the project.

What is the official list of Jupyter projects and how do new projects become part of Jupyter? How are projects conducted and governed?

  • Official Jupyter projects are reflected by their location in Jupyter Orgs on GitHub: Jupyter, JupyterHub, JupyterLab, IPython, Jupyter-Resources, Jupyter-widgets, Jupyter-enterprise-gateway, and recently voila-dashboards.

  • We have a ‘standard’ for how to bring new projects into the repo, with an associated checklist, but it’s not clear that list has been disseminated widely.

    • We don’t have good practices that are consistently followed around how to handle Jupyter ‘explorations’.
    • We should keep our badges up to date to communicate the status of a project, “Exploratory”, “Archived”, “Active”, and “Beta”.

  • Sometimes it is not clear to the public when a project is a community project and when a project is an official project under the Jupyter governance model. For example, nteract is a strong well-regarded community project that is adopted widely and promoted, including by members of Project Jupyter leadership, but is not under Jupyter governance, and thus is not an official Project Jupyter subproject. In situations like this, we can do better at both drawing the line for clarity and at the same time promoting community projects to strengthen the ecosystem.

  • The definitive official projects could/should be listed on jupyter.org.

What are the “shared resources” of Project Jupyter and how are they allocated among the projects?

  • One of the strongest ‘resources’ in the Jupyter is influence.

  • To first order, Jupyter doesn’t have shared financial resources. Given that we have a multistakeholder model, each stakeholder is responsible for funding their Jupyter related work in however manner they can or choose. At the same time, traditionally, Fernando and Brian have raised money for the project at Berkeley and Cal Poly respectively and that funding has played a key role in the evolution of the project. Over time, as other entities (mostly companies, but also other universities) have begun to participate in the project, they have both funded their employees to work on Jupyter, but also given financial resources to the project as donations to NumFOCUS. Over the years, these donations to NumFOCUS have provided a modest amount of centrally managed funds for the project that have been used for things such as Outreachy, JupyterDays, JupyterCon scholarships, Jupyter Community Workshops, and paying contributors for short periods of time.

  • While it is difficult to quantify or describe, the project also essentially has a shared human capital that is distributed across the different universities, research labs, companies and unaffiliated people. It is sort (maybe?) of like a labor union that crosses organizational boundaries, but we don’t really have a formal membership abstraction.

  • As a volunteer-based open-source project, people and resources flow between projects as needs and desires change.

5

A few thoughts from me:

How do we create project unity while supporting autonomous, smaller teams?

A few thoughts:

  • Have channels for high-bandwidth communication between sub-teams. Oftentimes it’s enough to just know what other teams are working on to find ways to connect with them. Online conversation spaces that cross teams (like this community forum) help people speak across team boundaries.
  • Periodic updates from each community to communicated recent activity, goals, next steps, challenges that are currently faced. Maybe something like a monthly community forum post, or a centralized location for notes from team meetings.
  • Standardize expectations for team culture and discourse. By this I mean that each team should adhere to the same values, tone, and standards around openness/friendliness that the broader Jupyter community has defined.
  • Give teams autonomy over a specific domain of code / practices. For example, “JupyterHub can decide how it wants to change the code under /jupyterhub/, but anything significant that changes under /jupyter/ is a decision that must be made by made by a team made up of representatives from each sub-team”.
  • Try to standardize resources and attention across teams. Another challenge we have is that some of the sub-communities in Jupyter are “less-interesting” or novel than others. JupyterLab has oodles more resources than ipython, even though ipython has a much larger user-base than JupyterLab. We should find ways to ensure that the “cool fancy new growing projects” lend their resources to the more stable-but-important projects (not picking on jlab here, just one example of many that comes to mind).

What is the official list of Jupyter projects and how do new projects become part of Jupyter? How are projects conducted and governed?

Again I think this can differ between sub-teams. One idea is to:

  • Use /jupyter/ for the “core” infrastructure that underlies the whole project. These are projects that tend to touch many sub-teams (e.g. jupyter-server or the notebook specification). These projects should be well-resourced, stable, and probably have a higher bar for major changes.
  • Use sub-teams such as /jupyterhub/ for topic-specific repositories that have more wiggle-room for experimentation and development. The sub-teams can define their own definition of this.
  • Create new GitHub orgs for some of the major themes of Jupyter that aren’t codified in a GitHub organization yet (e.g., I’d argue that nbconvert could exist in a GitHub org like jupyter-notebook, which would also handle the notebook UI etc. (I understand this may not be possible since the notebook has been in its current home for a long time now).
  • Have a clearly-defined JEP process for moving a repository from a sub-team into the /jupyter/ organization
  • Each of the major sub-teams in Jupyter could also correspond to a top-level section of the community forum

More broadly, we should define a minimum set of constraints for any official Jupyter organization. Probably something like:

  • Adheres to the Jupyter code of conduct
  • Has representation on a “meta” team that crosses sub-teams (e.g., a new, smaller steering council?)
  • Has a license that Jupyter is comfortable with

What are the “shared resources” of Project Jupyter and how are they allocated among the projects?

Currently the most precious resource that Jupyter has is its brand. This is useful both for boosting the profile of technical projects, as well as boosting the profile of people and organizations that work with Jupyter. I think we should create an official group of people that is something like the “Jupyter core team” that recognizes the contributions people have made without giving them “steering council” status. This group can grow more quickly without creating governance strains on the system.

As far as financial resources, this is trickier. Jupyter is in the strange situation of having a large disparity in resources between major stakeholders in the project. If Jupyter had an organizational structure that could raise funds in a stakeholder-neutral way, and then deploy those funds according to the needs of the project and the needs of stakeholders with fewer resources, it could be a way to normalize these imbalances a bit.

One other thing I want to note after writing all of this out - a lot of the structure / process / practices described above will take a lot of work over an extended period of time to implement. I think we should try and prioritize the pieces that are most-doable in a very short amount of time and have those as the goals for “governance refactor”, and then define a longer-term organizational roadmap for other things we’d like to pilot.