classification
Title: multiprocessing calls flush on sys.stdout at exit even if it is None (pythonw)
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Pox TheGreat, davin, pitrou, terry.reedy
Priority: normal Keywords: patch

Created on 2017-10-17 14:59 by Pox TheGreat, last changed 2018-03-11 19:10 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
process.py.patch Pox TheGreat, 2017-10-17 14:59 patch file with trivial bugfix
Pull Requests
URL Status Linked Edit
PR 5492 closed python-dev, 2018-02-02 12:46
PR 6079 merged pitrou, 2018-03-11 18:06
PR 6080 merged miss-islington, 2018-03-11 18:24
PR 6081 merged pitrou, 2018-03-11 18:28
Messages (12)
msg304512 - (view) Author: Pox TheGreat (Pox TheGreat) * Date: 2017-10-17 14:59
If you start Python by pythonw then sys.stdout and sys.stderr are set to None. If you also use multiprocessing then when the child process finishes BaseProcess._bootstrap calls sys.stdout.flush() and sys.stderr.flush() finally. This causes the process return code to be not zero (it is 1).
msg304678 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-10-20 20:39
This looks to be a duplicate of https://bugs.python.org/issue28326
msg309708 - (view) Author: Pox TheGreat (Pox TheGreat) * Date: 2018-01-09 16:11
Unfortunately this is NOT a duplicate of https://bugs.python.org/issue28326. That issue is about a closed output stream. In that case sys.stdout and sys.stderr are file like objects which have been closed.

This issue is about sys.stdout and sys.stderr being None! This is because pythonw was used not python.
msg309716 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-01-09 16:51
@Pox, nevertheless, the fix committed in https://github.com/python/cpython/pull/4073 should also fix this issue. Do you disagree?
msg309721 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-01-09 17:38
Pox, please retest with either 3.6.4 or 3.7.0a3 (or .0a4 when released, soon).  Both were released after the merge that should fix this.
msg309759 - (view) Author: Pox TheGreat (Pox TheGreat) * Date: 2018-01-10 09:33
Retested it with a freshly installed 3.6.4 version. Used the following code to test:

import sys
import multiprocessing


def foo():
    return 'bar'


if __name__ == '__main__':
    proc = multiprocessing.Process(target=foo)
    proc.start()
    proc.join()
    with open('process_exit_code.txt', 'w') as f:
        f.write(sys.version)
        f.write('\nprocess exit code: ')
        f.write(str(proc.exitcode))

It is very important to run the script with pythonw, not just with python. This is the content of the resulting process_exit_code.txt file on my machine:
3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)]
process exit code: 1

As it can be seen the problem was not fixed. The process exit code should be 0. By default the new multiprocessing process created uses the same interpreter as the creator process, so it uses pythonw too.
msg309762 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-01-10 10:59
I'm not using Windows, so I'm unable to test using pythonw.  You'll have to provide a fix, or someone else will have to.
msg309853 - (view) Author: Pox TheGreat (Pox TheGreat) * Date: 2018-01-12 12:55
I have already uploaded a patch file but it is not in the required format. Also I realize that most of the confusion was because I forgot to provide the OS version. Perhaps it would be good to have a separate field for that.

I will upload a patch as it is described in the developer guide.
msg313608 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-03-11 18:21
New changeset e756f66c83786ee82f5f7d45931ae50a6931dd7f by Antoine Pitrou in branch 'master':
bpo-31804: Fix multiprocessing.Process with broken standard streams (#6079)
https://github.com/python/cpython/commit/e756f66c83786ee82f5f7d45931ae50a6931dd7f
msg313613 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-03-11 18:42
New changeset ff5d21331ec6cefec6ba5b78d256d8dbcd67a069 by Antoine Pitrou (Miss Islington (bot)) in branch '3.7':
bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) (GH-6080)
https://github.com/python/cpython/commit/ff5d21331ec6cefec6ba5b78d256d8dbcd67a069
msg313616 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-03-11 19:09
New changeset 069b8d20be8018fbd49ed5aaf64c4caba311e48f by Antoine Pitrou in branch '3.6':
[3.6] bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) (GH-6081)
https://github.com/python/cpython/commit/069b8d20be8018fbd49ed5aaf64c4caba311e48f
msg313617 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-03-11 19:10
This is all fixed in the Python 3 branches now.  I won't bother with Python 2 as it has quite a different code structure and backporting would take too much of my time.

Pox TheGreat, thanks for reporting and the initial patch!
History
Date User Action Args
2018-03-11 19:10:52pitrousetstatus: open -> closed
resolution: fixed
messages: + msg313617

stage: patch review -> resolved
2018-03-11 19:09:22pitrousetmessages: + msg313616
2018-03-11 18:42:40pitrousetmessages: + msg313613
2018-03-11 18:35:42pitrousetversions: + Python 3.8, - Python 2.7
2018-03-11 18:28:17pitrousetpull_requests: + pull_request5842
2018-03-11 18:24:19miss-islingtonsetpull_requests: + pull_request5841
2018-03-11 18:21:40pitrousetmessages: + msg313608
2018-03-11 18:15:55pitrousetsuperseder: multiprocessing.Process depends on sys.stdout being open ->
2018-03-11 18:06:33pitrousetpull_requests: + pull_request5840
2018-02-02 12:46:44python-devsetstage: resolved -> patch review
pull_requests: + pull_request5325
2018-01-12 12:55:47Pox TheGreatsettype: crash -> behavior
messages: + msg309853
2018-01-10 10:59:53pitrousetmessages: + msg309762
2018-01-10 09:33:29Pox TheGreatsetmessages: + msg309759
2018-01-09 17:38:16terry.reedysetmessages: + msg309721
2018-01-09 16:51:06pitrousetmessages: + msg309716
2018-01-09 16:11:04Pox TheGreatsetstatus: closed -> open
resolution: duplicate -> (no value)
messages: + msg309708
2017-10-20 20:39:55pitrousetstatus: open -> closed
superseder: multiprocessing.Process depends on sys.stdout being open
messages: + msg304678

resolution: duplicate
stage: resolved
2017-10-20 18:47:31terry.reedysetversions: + Python 2.7, Python 3.6, Python 3.7, - Python 3.5
2017-10-20 18:47:06terry.reedysetnosy: + terry.reedy, pitrou, davin
2017-10-17 14:59:34Pox TheGreatcreate