How to show the result of shell instantly

For eg. I have a python file:
— foo.py —
import time
for i in range(1, 6):
print(i)
time.sleep(1)


Then I run this in a jupyter notebook:
! python foo.py
Theoretically, it will show numbers 1,2,3,4,5 one by one with time gap 1 second, which is also the result in the google colab.
But in jupyter notebook, it show all numbers together when the command is totally run finished.

2 Likes

Can you share more information about your environment (Python version, OS, Jupyter Lab or Notebook, pip list or conda list, other relevant info like docker, etc.)? I do get the output you expect (one number printed each second) when I run !python foo.py myself (Python 3.8, latest notebook, ipykernel, etc.).

One thing that may or may not affect results: Try setting

os.environ["PYTHONUNBUFFERED"] = "1"

which can sometimes improve responsiveness of subprocess output.

Thank you for reply.
My OS is win10, and I use Jupyter Notebook (6.0.3) installed by Anaconda3.
ipython 7.12.0
ipykernel 5.1.4
Python 3.7.6
No docker.

Ah, interaction with subprocesses is very different on Windows than posix. That’s going to be the difference from colab - colab runs on linux. I’m not sure what the current expected level of subprocess output streaming support on Windows is.

Have you tried the following?

%run -i foo.py

Read about %run here.

I stumbled over the same issue recently. All output of subprocess only showed up once the command finished executing so thatI wrote this small fix for Windows for me:

import subprocess
import shlex
import os
from IPython.core.magic import register_line_magic

codepage_output = !chcp
print("Command output: >>" + codepage_output.s + "<<")
codepage_digits = "".join([d for d in codepage_output.s.split(":")[1] if d.isnumeric()])
print("Parsed Codepage: >>" + codepage_digits + "<<")

def run_windows_command(command, output_function=print):
    print(f"=== Executing command: {command}")
    process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=os.environ.copy())
    while True:
        stdout_line = process.stdout.readline().decode(f"CP{codepage_digits}")
        if not stdout_line:
            break
        else:
            print(stdout_line.replace("\r\n", "\n")[:-1])  # Remove '\r\n' and '\n'
    print(f"\n=== Command has finished")

@register_line_magic
def cmd_unbuffered(line):
    line_with_resolved_vars = line.format(**globals(), **locals())
    return run_windows_command(line_with_resolved_vars)

Now you can write something like this in a cell:

%cmd_unbuffered PING 127.0.0.1 -n 1

I have only scratched on the surface of the documenation, I am sure that there is a more elegant way of doing in. Additionally, this does not check for Linux and assumes that everything works like under Windows, e.g. the command chcp.

3 Likes

Thank you for reply.
This indeed can help, but is also limited. As I mentioned in the title, I want to show the result from shell, which can run other languages, besides Python. But the magic command %run can only run python code with build-in IPython.

Yes, it indeed help. Thanks lot. Sometimes need co-work with -u and flush=True in python.

Thank you very much. This is very helpful for printing subprocess output of yolov5 in jupyter-notebook in windows.

1 Like