This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: SubProcess stdin.flush freezes when running python interpreter
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: djp1012878, eryksun
Priority: normal Keywords:

Created on 2021-11-05 00:42 by djp1012878, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
example.py djp1012878, 2021-11-05 00:42 Simplified Example of what I mean
Messages (2)
msg405761 - (view) Author: DJ PJs (djp1012878) Date: 2021-11-05 00:42
Keeping writing to a subprocess, and then flushing it to get the stdout works for other interpreters, but not python's.

Simplified Example of what I mean:

from subprocess import PIPE, Popen, CREATE_NEW_CONSOLE, run

subProcess = Popen("Python", stdin=PIPE, stdout=PIPE, text=True, universal_newlines=True)

subProcess.stdin.write('Print("HelloWorld")')

subProcess.stdin.flush()

for line in subProcess.stdout:
    print(">>> " + str(line.rstrip()))
    subProcess.stdout.flush()
msg405856 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-11-06 11:34
Interactive mode isn't automatically enabled when stdin isn't a tty, so the interpreter tries to read all of stdin, until the result is empty (i.e. EOF). You can force interactive mode with the -i command-line option. The example would also benefit from passing bufsize=0 to Popen, since full buffering on either the child side or parent side of the pipe prevents interactive I/O. Finally, input statements have to end with a newline character since the interpreter reads input in lines.

Here's a toy example that interacts with a REPL in a child process. Note that spawn_repl() sets the sys.ps1 prompt in the child to a string that ends with a newline. This allows iterating over the lines in p.stdout up to the REPL prompt. Also, spawn_repl() forces UTF-8 mode in order to reliably support Unicode. Otherwise text support in Windows is limited to the ANSI code page.

    import subprocess

    def _write_input(p, input):
        if not input.endswith('\n'):
            input += '\n'
        p.stdin.write(input)

    def _read_output(p):
        lines = []
        for line in p.stdout:
            if line.endswith(p._ps1):
                break
            lines.append(line)
        return lines

    def eval_remote(p, input):
        if not hasattr(p, '_ps1'):
            raise ValueError('p._ps1 prompt not defined; use p = spawn_repl()')
        _write_input(p, input)
        return _read_output(p)

    def spawn_repl():
        p = subprocess.Popen(
                ['python', '-i', '-q', '-X utf8'],
                stdin=subprocess.PIPE,
                stdout=subprocess.PIPE,
                stderr=subprocess.STDOUT,
                bufsize=0, encoding='utf-8')
        p._ps1 = f'<PY:{p.pid}>\n'
        eval_remote(p, f'import sys; sys.ps1="<PY:{p.pid}>\\n"')
        return p

For example:

    >>> p = spawn_repl()
    >>> eval_remote(p, 'double = lambda x: 2 * x')
    []
    >>> eval_remote(p, 'double(2)')
    ['4\n']
    >>> eval_remote(p, 'print("spam\\neggs")')
    ['spam\n', 'eggs\n']
History
Date User Action Args
2022-04-11 14:59:52adminsetgithub: 89882
2021-11-06 11:34:10eryksunsetnosy: + eryksun
messages: + msg405856
2021-11-05 00:42:46djp1012878create