classification
Title: py launcher stderr is not piped to subprocess.Popen.stderr
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, paul.moore, python-dev, steve.dower, tim.golden, vinay.sajip, wolma, zach.ware
Priority: normal Keywords: patch

Created on 2015-12-03 17:25 by wolma, last changed 2015-12-26 12:36 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
issue25789_1.patch eryksun, 2015-12-03 18:50 review
issue25789_2.patch eryksun, 2015-12-04 15:30 review
Messages (6)
msg255822 - (view) Author: Wolfgang Maier (wolma) * Date: 2015-12-03 17:25
from the console:

> py -3.7

or any other not installed Python version gives:
Requested Python version (3.7) not installed


However, when the launcher is executed from python via subprocess.Popen:

>>>import subprocess
>>>p=subprocess.Popen(['py', '-3.7'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>>p.communicate()
(b'', b'')

the error message is not accessible. (Error messages from any successfully launched Python interpreter are available through p.stderr though.)
msg255833 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2015-12-03 18:50
The error() function in PC/launcher.c should call exit(rc) instead of ExitProcess(rc). This allows the CRT to terminate properly and flush the stderr FILE stream. With this change it works as expected:

    >>> import subprocess
    >>> p = subprocess.Popen(r'amd64\py_d -3.7', stderr=subprocess.PIPE)
    >>> p.stderr.read()
    b'Requested Python version (3.7) not installed\r\n'
msg255854 - (view) Author: Wolfgang Maier (wolma) * Date: 2015-12-04 10:07
> The error() function in PC/launcher.c should call exit(rc) instead of ExitProcess(rc). This allows the CRT to terminate properly and flush the stderr FILE stream.

Interesting. Why do you need to flush stderr? I would have expected it to be unbuffered in the first place. What's the usecase/advantage of calling ExitProcess then?
Sorry, if that does not really belong here and just shows my limited knowledge of C.
msg255865 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-12-04 14:12
ExitProcess is a system API and exit is the C runtime API. The C runtime is doing the buffering, so the system doesn't know about it and can't flush if you terminate through that API. The exit function should, or we could explicitly flush the buffer after writing to it.
msg255866 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2015-12-04 15:30
Patch 2 additionally modifies run_child to call exit() instead of ExitProcess. For example:

    >>> import os, subprocess
    >>> os.environ['PYLAUNCH_DEBUG'] = '1'
    >>> p = subprocess.Popen(r'py -3 -c ""', stderr=subprocess.PIPE, stdout=subprocess.PIPE)
    >>> p.stderr.read()
    b''

Patched:

    >>> p = subprocess.Popen(r'amd64\py_d -3 -c ""', stderr=subprocess.PIPE, stdout=subprocess.PIPE)
    >>> p.stderr.readlines()[-1]
    b'child process exit code: 0\r\n'

For good measure I also added a call to setvbuf to disable buffering stderr.
msg257029 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-12-26 12:36
New changeset d0a84d0c5ceb by Vinay Sajip in branch 'default':
Closes #25789: Improved buffering behaviour in launcher.
https://hg.python.org/cpython/rev/d0a84d0c5ceb
History
Date User Action Args
2015-12-26 12:36:12python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg257029

resolution: fixed
stage: patch review -> resolved
2015-12-04 15:30:22eryksunsetfiles: + issue25789_2.patch

messages: + msg255866
2015-12-04 14:12:36steve.dowersetmessages: + msg255865
2015-12-04 10:29:01vinay.sajipsetnosy: + vinay.sajip
2015-12-04 10:07:57wolmasetmessages: + msg255854
2015-12-03 18:50:47eryksunsetfiles: + issue25789_1.patch

nosy: + eryksun
messages: + msg255833

keywords: + patch
stage: patch review
2015-12-03 17:25:52wolmacreate