Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sys.exit() in a multiprocessing.Process does not align with Python behavior #79908

Closed
chrahunt mannequin opened this issue Jan 12, 2019 · 7 comments
Closed

sys.exit() in a multiprocessing.Process does not align with Python behavior #79908

chrahunt mannequin opened this issue Jan 12, 2019 · 7 comments
Labels
3.9 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@chrahunt
Copy link
Mannequin

chrahunt mannequin commented Jan 12, 2019

BPO 35727
Nosy @pitrou, @applio, @eamanu, @chrahunt
PRs
  • bpo-35727: Use exit code 0 on sys.exit() in multiprocessing.Process. #11538
  • bpo-35727: Use exit code 0 on sys.exit() in multiprocessing.Process. #11538
  • bpo-35727: Use exit code 0 on sys.exit() in multiprocessing.Process. #11538
  • Files
  • multiprocessing-exitcode-3.7.1.patch: patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2020-02-21.09:34:23.641>
    created_at = <Date 2019-01-12.19:33:37.108>
    labels = ['type-bug', 'library', '3.9']
    title = 'sys.exit() in a multiprocessing.Process does not align with Python behavior'
    updated_at = <Date 2020-02-21.09:34:23.638>
    user = 'https://github.com/chrahunt'

    bugs.python.org fields:

    activity = <Date 2020-02-21.09:34:23.638>
    actor = 'pitrou'
    assignee = 'none'
    closed = True
    closed_date = <Date 2020-02-21.09:34:23.641>
    closer = 'pitrou'
    components = ['Library (Lib)']
    creation = <Date 2019-01-12.19:33:37.108>
    creator = 'chrahunt'
    dependencies = []
    files = ['48045']
    hgrepos = []
    issue_num = 35727
    keywords = ['patch']
    message_count = 7.0
    messages = ['333531', '333553', '352362', '352396', '356841', '362396', '362397']
    nosy_count = 4.0
    nosy_names = ['pitrou', 'davin', 'eamanu', 'chrahunt']
    pr_nums = ['11538', '11538', '11538']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue35727'
    versions = ['Python 3.9']

    @chrahunt
    Copy link
    Mannequin Author

    chrahunt mannequin commented Jan 12, 2019

    When a function is executed by a multiprocessing.Process and uses sys.exit,
    the actual exit code reported by multiprocessing is different than would be
    expected given the Python interpreter behavior and documentation. For example,
    given:

        from functools import partial
        from multiprocessing import get_context
        import sys
        
        
        def run(ctx, fn):
            p = ctx.Process(target=fn)
            p.start()
            p.join()
            return p.exitcode
        
        
        if __name__ == '__main__':
            ctx = get_context('fork')
            print(run(ctx, partial(sys.exit, 2)))
            print(run(ctx, partial(sys.exit, None)))
            print(run(ctx, sys.exit))
        
            ctx = get_context('spawn')
            print(run(ctx, partial(sys.exit, 2)))
            print(run(ctx, partial(sys.exit, None)))
            print(run(ctx, sys.exit))
        
            ctx = get_context('forkserver')
            print(run(ctx, partial(sys.exit, 2)))
            print(run(ctx, partial(sys.exit, None)))
            print(run(ctx, sys.exit))

    when executed results in

        $ python exit.py
        2
        1
        1
        2
        1
        1
        2
        1
        1

    but when Python itself is executed we see different behavior

        $ for arg in 2 None ''; do python -c "import sys; sys.exit($arg)"; echo $?; done
        2
        0
        0

    The documentation states

    sys.exit([arg])
    ...
    The optional argument arg can be an integer giving the exit status
    (defaulting to zero), or another type of object.

    The relevant line in multiprocessing (https://github.com/python/cpython/blame/1cffd0eed313011c0c2bb071c8affeb4a7ed05c7/Lib/multiprocessing/process.py#L307)
    seems to be from the original pyprocessing module itself, and I could
    not locate an active site that maintains the repository to see if there
    was any justification for the behavior.

    @chrahunt chrahunt mannequin added 3.7 (EOL) end of life stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Jan 12, 2019
    @eamanu
    Copy link
    Mannequin

    eamanu mannequin commented Jan 13, 2019

    The same behavior on 3.8 and 3.5

    @eamanu eamanu mannequin added the 3.8 only security fixes label Jan 13, 2019
    @applio
    Copy link
    Member

    applio commented Sep 13, 2019

    I believe the mentality behind multiprocessing.Process triggering an exit code of 1 when sys.exit() is invoked inside its process is to indicate a non-standard exit out of its execution. There may yet be other side effects that could be triggered by having a sys.exit(0) translate into an exit code of 0 from the Process's process -- and we might not notice them with the current tests.

    Was there a particular use case that motivates this suggested change?

    @chrahunt
    Copy link
    Mannequin Author

    chrahunt mannequin commented Sep 13, 2019

    I believe the mentality behind multiprocessing.Process triggering an exit code of 1 when sys.exit() is invoked inside its process is to indicate a non-standard exit out of its execution.

    Can I ask what this is based on? I did a pretty good amount of digging but didn't find any justification for it. It just seems like a simple oversight to me.

    There may yet be other side effects that could be triggered by having a sys.exit(0) translate into an exit code of 0 from the Process's process -- and we might not notice them with the current tests.

    This is definitely a behavior change and will break any code that currently relies on sys.exit(None) or sys.exit() exiting with a non-zero exit code from a multiprocessing.Process. The fact that all documentation indicates that sys.exit(None) or sys.exit() results in a 0 exit code in normal Python (with no documentation on it related to multiprocessing) makes me think that any code relying on this behavior is subtly broken, however. Any impacted user can update their code and explicitly pass 1 to sys.exit, which should be forward and backwards compatible.

    Was there a particular use case that motivates this suggested change?

    I have a wrapper library that invokes arbitrary user code and attempts to behave as if that code was executed in a vanilla Python process, to include propagating the correct exit code.

    Currently I have a workaround here: https://github.com/chrahunt/quicken/blob/2dd00a5f024d7b114b211aad8a2618ec8f101956/quicken/_internal/server.py#L344-L353, but it would be nice to get rid of it in 5-6 years if this fix gets in and the non-conformant Python versions fall out of support. :)

    @chrahunt
    Copy link
    Mannequin Author

    chrahunt mannequin commented Nov 18, 2019

    Any other concerns here?

    @pitrou
    Copy link
    Member

    pitrou commented Feb 21, 2020

    New changeset c2ac4cf by Christopher Hunt in branch 'master':
    bpo-35727: Use exit code 0 on sys.exit() in multiprocessing.Process. (GH-11538)
    c2ac4cf

    @pitrou
    Copy link
    Member

    pitrou commented Feb 21, 2020

    Sorry for the delay. I've now merged the PR for 3.9. Since this is a slight behaviour change, and the original issue was easy to workaround, I won't backport it.

    @pitrou pitrou added 3.9 only security fixes and removed 3.7 (EOL) end of life 3.8 only security fixes labels Feb 21, 2020
    @pitrou pitrou closed this as completed Feb 21, 2020
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants