Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multiprocessing calls flush on sys.stdout at exit even if it is None (pythonw) #75985

Closed
poxthegreat mannequin opened this issue Oct 17, 2017 · 12 comments
Closed

multiprocessing calls flush on sys.stdout at exit even if it is None (pythonw) #75985

poxthegreat mannequin opened this issue Oct 17, 2017 · 12 comments
Labels
3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@poxthegreat
Copy link
Mannequin

poxthegreat mannequin commented Oct 17, 2017

BPO 31804
Nosy @terryjreedy, @pitrou, @applio, @poxthegreat
PRs
  • bpo-31804: multiprocessing calls flush on sys.stdout at exit even if … #5492
  • bpo-31804: Fix multiprocessing.Process with broken standard streams #6079
  • [3.7] bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) #6080
  • [3.6] bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) #6081
  • Files
  • process.py.patch: patch file with trivial bugfix
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2018-03-11.19:10:52.978>
    created_at = <Date 2017-10-17.14:59:34.932>
    labels = ['3.7', '3.8', 'type-bug', 'library']
    title = 'multiprocessing calls flush on sys.stdout at exit even if it is None (pythonw)'
    updated_at = <Date 2018-03-11.19:10:52.977>
    user = 'https://github.com/poxthegreat'

    bugs.python.org fields:

    activity = <Date 2018-03-11.19:10:52.977>
    actor = 'pitrou'
    assignee = 'none'
    closed = True
    closed_date = <Date 2018-03-11.19:10:52.978>
    closer = 'pitrou'
    components = ['Library (Lib)']
    creation = <Date 2017-10-17.14:59:34.932>
    creator = 'Pox TheGreat'
    dependencies = []
    files = ['47224']
    hgrepos = []
    issue_num = 31804
    keywords = ['patch']
    message_count = 12.0
    messages = ['304512', '304678', '309708', '309716', '309721', '309759', '309762', '309853', '313608', '313613', '313616', '313617']
    nosy_count = 4.0
    nosy_names = ['terry.reedy', 'pitrou', 'davin', 'Pox TheGreat']
    pr_nums = ['5492', '6079', '6080', '6081']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue31804'
    versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

    @poxthegreat
    Copy link
    Mannequin Author

    poxthegreat mannequin commented Oct 17, 2017

    If you start Python by pythonw then sys.stdout and sys.stderr are set to None. If you also use multiprocessing then when the child process finishes BaseProcess._bootstrap calls sys.stdout.flush() and sys.stderr.flush() finally. This causes the process return code to be not zero (it is 1).

    @poxthegreat poxthegreat mannequin added type-crash A hard crash of the interpreter, possibly with a core dump stdlib Python modules in the Lib dir labels Oct 17, 2017
    @terryjreedy terryjreedy added the 3.7 (EOL) end of life label Oct 20, 2017
    @pitrou
    Copy link
    Member

    pitrou commented Oct 20, 2017

    This looks to be a duplicate of https://bugs.python.org/issue28326

    @pitrou pitrou closed this as completed Oct 20, 2017
    @poxthegreat
    Copy link
    Mannequin Author

    poxthegreat mannequin commented Jan 9, 2018

    Unfortunately this is NOT a duplicate of https://bugs.python.org/issue28326. That issue is about a closed output stream. In that case sys.stdout and sys.stderr are file like objects which have been closed.

    This issue is about sys.stdout and sys.stderr being None! This is because pythonw was used not python.

    @poxthegreat poxthegreat mannequin reopened this Jan 9, 2018
    @pitrou
    Copy link
    Member

    pitrou commented Jan 9, 2018

    @pox, nevertheless, the fix committed in #4073 should also fix this issue. Do you disagree?

    @terryjreedy
    Copy link
    Member

    Pox, please retest with either 3.6.4 or 3.7.0a3 (or .0a4 when released, soon). Both were released after the merge that should fix this.

    @poxthegreat
    Copy link
    Mannequin Author

    poxthegreat mannequin commented Jan 10, 2018

    Retested it with a freshly installed 3.6.4 version. Used the following code to test:

    import sys
    import multiprocessing
    
    
    def foo():
        return 'bar'
    
    
    if __name__ == '__main__':
        proc = multiprocessing.Process(target=foo)
        proc.start()
        proc.join()
        with open('process_exit_code.txt', 'w') as f:
            f.write(sys.version)
            f.write('\nprocess exit code: ')
            f.write(str(proc.exitcode))

    It is very important to run the script with pythonw, not just with python. This is the content of the resulting process_exit_code.txt file on my machine:
    3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)]
    process exit code: 1

    As it can be seen the problem was not fixed. The process exit code should be 0. By default the new multiprocessing process created uses the same interpreter as the creator process, so it uses pythonw too.

    @pitrou
    Copy link
    Member

    pitrou commented Jan 10, 2018

    I'm not using Windows, so I'm unable to test using pythonw. You'll have to provide a fix, or someone else will have to.

    @poxthegreat
    Copy link
    Mannequin Author

    poxthegreat mannequin commented Jan 12, 2018

    I have already uploaded a patch file but it is not in the required format. Also I realize that most of the confusion was because I forgot to provide the OS version. Perhaps it would be good to have a separate field for that.

    I will upload a patch as it is described in the developer guide.

    @poxthegreat poxthegreat mannequin added type-bug An unexpected behavior, bug, or error and removed type-crash A hard crash of the interpreter, possibly with a core dump labels Jan 12, 2018
    @pitrou
    Copy link
    Member

    pitrou commented Mar 11, 2018

    New changeset e756f66 by Antoine Pitrou in branch 'master':
    bpo-31804: Fix multiprocessing.Process with broken standard streams (bpo-6079)
    e756f66

    @pitrou pitrou added the 3.8 only security fixes label Mar 11, 2018
    @pitrou
    Copy link
    Member

    pitrou commented Mar 11, 2018

    New changeset ff5d213 by Antoine Pitrou (Miss Islington (bot)) in branch '3.7':
    bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) (GH-6080)
    ff5d213

    @pitrou
    Copy link
    Member

    pitrou commented Mar 11, 2018

    New changeset 069b8d2 by Antoine Pitrou in branch '3.6':
    [3.6] bpo-31804: Fix multiprocessing.Process with broken standard streams (GH-6079) (GH-6081)
    069b8d2

    @pitrou
    Copy link
    Member

    pitrou commented Mar 11, 2018

    This is all fixed in the Python 3 branches now. I won't bother with Python 2 as it has quite a different code structure and backporting would take too much of my time.

    Pox TheGreat, thanks for reporting and the initial patch!

    @pitrou pitrou closed this as completed Mar 11, 2018
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants