Thanks for the replies and thoughts!
I see it exactly the other way around: travis is a robot, it doesn’t care if the setup instructions are 10, 20, 50 or 100 lines. It just executes them all. The hard part about our CI is that it sometimes (rarely) breaks or stops doing what it is meant to be doing. This is when someone has to go and understand what it should be doing, what it is doing and why the two are different. Usually breakage happens because something external to the repository under test changed. Someone released a new version of some tool, a service got deprecated, etc Every layer of abstraction or automation that we add makes this “once every few months” debugging of a complex testing setup on a remote machine I can’t even ssh to more intimidating.
It is the combination of rare breakage, external factors, a solid&debugged
travis.yml and complex test scenarios that make me very conservative when it comes to more automation or changing the CI setup “for fun”. I’d much rather see those hours used to develop new features or used next time when the CI is broken and we can’t merge PRs because no one has time/spare brain power to debug what has changed somewhere on the internet that causes our CI to break.
I use conda environments to achieve this isolation on my macbook and you use tox (I think) and a third person uses vagrant to manage separate VMs for separate projects
This diversity of setups is why I think having the instructions for humans in the docs be “lowest common denominator” instead of picking one tool and giving instructions for that one. The unsuspecting newcomer (to Python development) can copy&paste the commands to get something up and running. The grizzled veteran developers will all be equally annoyed that the instructions don’t use “the correct tool” for automating this but they can read the commands and translate them to their way of doing things. With explicit commands you have to translate them but with automation you’d first have to learn enough about the tool we use for automation to then translate it to your way of doing things.
As you noticed we already struggle to keep things updated, coherent and nice with the simple instructions we have. For them to be friendly to newcomers it is probably more about how discoverable they are and the words around the commands than the commands themselves. No amount of automation will help us with the words for humans.
As an example: in Zero2JupyterHub we recently switched to having two tracks for installing things: automated and manual. One thing that has happened is that the automated setup is very hard for newcomers to the project to adapt and the number of regular contributors who understand it well enough to help is also small. The contributing guide is hard to read because it switches between the two modes of installing things. While I don’t know if the switch was a success or not, I think right now the majority feels like “it is different and also has problems, but hey they are different from the ones we had before”.
So overall my feeling is that unless there is a very concrete problem that is painful for several people making changes to the CI setup or dev setup instructions from one type of setup to another different type is unlikely to reduce the total pain, it just moves it. My money is on small incremental changes to the existing setup/instructions to make it more robust, easier to follow, well understood by more people, etc as the way to reduce the total pain (and time spent maintaining this support infrastructure).