Does anyone have any suggestions for an online environment where we can run jupyter notebook IPYNB document files that connect to our Mysql database (on our servers) to pull data, run python computational scripts, and then output csv files to be written back to a folder on our server?
The IPYNB files will be crunching rather large datasets (approx 15 million) daily and the scripts need to run automatically (like a cron).
We are visualizing the csv files for a dashboard system that is already on our servers and simply need the daily update/overwrite of the data while also allowing our python developer an online environment to modify/update the scripts.
We want to keep this separate from the server environment as we have other sensitive material on the server.
We also want the jupyter notebook platform to be private and NOT in a collaborative environment.
Im looking for a reliable yet cost-effective solution since we already have a robust VPS server.
You are correct, Mark. The developer uses Jupyter Notebook and is running those files to “execute” the python code. I guess I just need to run these as crons. Just need to figure out how to do that. Im assuming it is similar to the way I run PHP crons from my cpanel.
Naively, you can convert your users notebook to a script using jupytext, see the first command listed here. Then in you cpanel you just put the following as the command in place of ./my-script.sh in the example here:
python <path>/<script_name>.py
Jupytext can be installed with pip or conda.
Adding conda in the mix would allow you to handle the environment issue on whichever machine you pick to run modern Python since you say you want it separate from the servers where your Mysql database is. There’s a number of posts around on how you can use conda with cron jobs, such as here.
I think papermill might be close to what you are looking for. You can combine it with a job runner like cron or airflow. See this discussion from some of the excellent developers of papermill at Netflix.