Using Magic commands with Python Multiprocessing Libraries

To accomplish my tasks
I need to run one notebook for different set of input(python list)
I decided to use IPython Magic command %run notebook.ipynb
This is working but that is taking too much time to execute
So I’ve opted to implement python’s multiprocessing library to execute faster

Here I’m facing a problem like when i try to run the code of multiprocessing the input is not passed to %run command
so I’m getting a error like name ‘input’ not found
Thanks in advance for the solutions

CROSSPOST: here     (see UPDATE at bottom)

Please @ Nitheeswaran_B, if you are going to crosspost, note in all places where you crosspost. Otherwise you can possibly take up time of multiple persons offering the same guidance. Additionally, it helps people in a similar situation find the best solution.

You aren’t sharing with us how you are trying to execute %run in the context of multiprocessing and the input and so advising you is difficult.

You say it works; however, it takes too long. Are you sure it works then? Do you expect it to take too long? Do smaller runs with less notebooks complete?

I wonder if you’d have better luck using jupyter nbconvert or jupytext or papermill to run the other notebook or notebooks in conjunction with multiprocessing from within a notebook or on the command line? This post uses multiprocessing and papermill.
Here suggests subprocess and papermill being a good alternative.


Saw you crosspost on stackoverflow had more information, and so I answered there. Some of my advice and points above hold, and so I’m adding this part just at bottom as an update.