I appreciate this coming up here, because we’re also working on this at Gigantum. At this point we have a few models for how this can work.
At the core of our approach is a desire to be more accessible than GitHub and more financially sustainable by ensuring broad portability of both data and compute (as opposed to a single cloud or national / institutional infrastructure). I will point out that Binder has a similar decentralized model, but I think the decentralization is more for administrators (or at least developers) than for end-users. That’s not bad - just different (and I’d be happy to discuss finer points - but for now I’m focused on an overview of my perspective from my experience at Gigantum).
There is the clear and vocal contingent that wants to put stuff on GitHub. There are lots of people who are intimidated or simply find GitHub and the related requirements burdensome. Support for inter-op with external git repositories is a medium term goal for us because of this demand, and I guess a GitHub option is important for any review tool.
I still believe that folks underestimate the impact of cognitive burden on “open science best practices” (and there is reasonable empirical evidence to this underestimation effect in general) - so it’s better for the actual science itself if review systems provide a scaffolded or even automated process. For example, the workflow of the PLOS or Frontiers review system is far more universal than anything I can imagine achieving directly with GitHub. (Presumably open journal also - I’ve not used it! And if I’m behind the times on GitHub based review, I would appreciate pointers!)
Relatedly, if things are easier to set up, the author and/or reviewer can use any extra bandwidth to improve the quality of the work and communication itself.
But I think it’s especially important to make things accessible to hand (not just to look at, but to use). I think Randal Burns did a pretty good job with this project:
This involves benchmarking first on a local machine, then on a standardized AWS reference. Anyone can poke around with these benchmarking results by clicking the “launch jupyterlab button” but they can also paste that URL into an application running locally and get a “launch jupyterlab” button there. This makes it far easier to reproduce a benchmark than it would be on Kubernetes (which is what we and at least Binder are using). Or, you could just look through our complete record of every command sent to the Jupyter kernel and see what the person did and trust their benchmarks. The reviewer can move directly towards subjecting the author to whatever level of scrutiny is desired.
The large data inputs are in an attached
forestpacking dataset, so if you just want to pull the project onto your laptop to review results on a plane or in the woods, you can conserve space and leave the datasets behind, or just grab one file, etc.
Anyway, in terms of tools for review per se, I wonder if a review system could be de-coupled from the publishing side of the Open Journal system?
Foundationally, my hope is that we get a variety of projects that have different focus (e.g., empowering end-user, making administration by institutions easier, hard-core developer mode, etc.). This translates into a desire for a review system that’s not tightly coupled to the systems for actually running or inspecting code and data!