For eg. I have a python file:
— foo.py —
import time
for i in range(1, 6):
print(i)
time.sleep(1)
Then I run this in a jupyter notebook: ! python foo.py
Theoretically, it will show numbers 1,2,3,4,5 one by one with time gap 1 second, which is also the result in the google colab.
But in jupyter notebook, it show all numbers together when the command is totally run finished.
Can you share more information about your environment (Python version, OS, Jupyter Lab or Notebook, pip list or conda list, other relevant info like docker, etc.)? I do get the output you expect (one number printed each second) when I run !python foo.py myself (Python 3.8, latest notebook, ipykernel, etc.).
One thing that may or may not affect results: Try setting
os.environ["PYTHONUNBUFFERED"] = "1"
which can sometimes improve responsiveness of subprocess output.
Ah, interaction with subprocesses is very different on Windows than posix. That’s going to be the difference from colab - colab runs on linux. I’m not sure what the current expected level of subprocess output streaming support on Windows is.
I stumbled over the same issue recently. All output of subprocess only showed up once the command finished executing so thatI wrote this small fix for Windows for me:
import subprocess
import shlex
import os
from IPython.core.magic import register_line_magic
codepage_output = !chcp
print("Command output: >>" + codepage_output.s + "<<")
codepage_digits = "".join([d for d in codepage_output.s.split(":")[1] if d.isnumeric()])
print("Parsed Codepage: >>" + codepage_digits + "<<")
def run_windows_command(command, output_function=print):
print(f"=== Executing command: {command}")
process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=os.environ.copy())
while True:
stdout_line = process.stdout.readline().decode(f"CP{codepage_digits}")
if not stdout_line:
break
else:
print(stdout_line.replace("\r\n", "\n")[:-1]) # Remove '\r\n' and '\n'
print(f"\n=== Command has finished")
@register_line_magic
def cmd_unbuffered(line):
line_with_resolved_vars = line.format(**globals(), **locals())
return run_windows_command(line_with_resolved_vars)
Now you can write something like this in a cell:
%cmd_unbuffered PING 127.0.0.1 -n 1
I have only scratched on the surface of the documenation, I am sure that there is a more elegant way of doing in. Additionally, this does not check for Linux and assumes that everything works like under Windows, e.g. the command chcp.
Thank you for reply.
This indeed can help, but is also limited. As I mentioned in the title, I want to show the result from shell, which can run other languages, besides Python. But the magic command %run can only run python code with build-in IPython.