Running Jupyter notebook with Papermill within Pyinstaller

Notebook nteract
,
1

Hello everyone, first time posting here.

I’m part of a finance related open-source project called OpenBB (GitHub - OpenBB-finance/OpenBBTerminal: Investment Research for Everyone, Anywhere.).

Currently, we are trying to integrate Papermill to parametrize Jupyter notebooks in a Pyinstaller application. Everything works fine at editor level or command line, but when the installer is built it always crash before running the actual notebook.

Anybody here has experience launching notebooks within Pyinstaller applications, ideally from papermill?

Apparently there is some issue with ipykernel_launcher trying to run inside the application itself that causes the crash later on.

Executing:   0%|                                                       | 0/36 [00:00<?, ?cell/s]0.00s - Debugger warning: The os.path.realpath.__code__.co_filename (posixpath.py)
0.00s - is not absolute, which may make the debugger miss breakpoints.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
Warning: Input is not a terminal (fd=0).
usage: terminal [-h] [-d] [--file PATH [PATH ...]] [-t] [--filter FILTERT] [-v]
                [-i ROUTINE_ARGS]
terminal: error: unrecognized arguments: -B -S -E -s -c from multiprocessing.resource_tracker import main;main(30)
usage: terminal [-h] [-d] [--file PATH [PATH ...]] [-t] [--filter FILTERT] [-v]
                [-i ROUTINE_ARGS]
terminal: error: unrecognized arguments: -m ipykernel_launcher -f /private/var/folders/65/56k7h3zj24b56y8nv5mr2m400000gn/T/tmp4tioi2bx.json --HistoryManager.hist_file=:memory:
Executing:   0%|                                                       | 0/36 [00:03<?, ?cell/s]

Link to PR: Refactor reports menu and make it work on installer by montezdesousa · Pull Request #2878 · OpenBB-finance/OpenBBTerminal · GitHub

Thanks!

2 
unrecognized arguments: -m

So… do you have an actual python executable? Can you get it to run jupyter --paths inside somehow, or otherwise list where it’s looking for files?

You might need to install custom kernels that have your appropriate magically-munged paths.

I don’t know much about pyinstaller, but have nothing but good things to say about constructor which powers JupyterLab Desktop, among other things. It also opens the door for shipping things other than python.

2 Likes
3

Thanks for your reply! We managed to make it work.

Apparently, under frozen applications, sys.executable points to the application bootloader instead of Python interpreter. Source: Run-time Information — PyInstaller 5.5 documentation

So, the fix was to parse the unrecognized arguments into our application and route the module -m ipykernel_launcher execution to IPKernelApp, which then launches the kernel. From then on papermill works as usual.

        from ipykernel.kernelapp import IPKernelApp

        IPKernelApp.launch_instance(
            argv=[
                "-f",
                "/private/var/folders/65/56k7h3zj24b56y8nv5mr2m400000gn/T/tmp4tioi2bx.json",
                "--HistoryManager.hist_file",
                ":memory:",
            ]
        )
1 Like