classification
Title: sys.exit() in a multiprocessing.Process does not align with Python behavior
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: chrahunt, davin, eamanu, pitrou
Priority: normal Keywords: patch

Created on 2019-01-12 19:33 by chrahunt, last changed 2020-02-21 09:34 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
multiprocessing-exitcode-3.7.1.patch chrahunt, 2019-01-12 19:33 patch
Pull Requests
URL Status Linked Edit
PR 11538 merged chrahunt, 2019-01-12 20:23
PR 11538 merged chrahunt, 2019-01-12 20:23
PR 11538 merged chrahunt, 2019-01-12 20:23
Messages (7)
msg333531 - (view) Author: Christopher Hunt (chrahunt) * Date: 2019-01-12 19:33
When a function is executed by a multiprocessing.Process and uses sys.exit,
the actual exit code reported by multiprocessing is different than would be
expected given the Python interpreter behavior and documentation. For example,
given:

    from functools import partial
    from multiprocessing import get_context
    import sys
    
    
    def run(ctx, fn):
        p = ctx.Process(target=fn)
        p.start()
        p.join()
        return p.exitcode
    
    
    if __name__ == '__main__':
        ctx = get_context('fork')
        print(run(ctx, partial(sys.exit, 2)))
        print(run(ctx, partial(sys.exit, None)))
        print(run(ctx, sys.exit))
    
        ctx = get_context('spawn')
        print(run(ctx, partial(sys.exit, 2)))
        print(run(ctx, partial(sys.exit, None)))
        print(run(ctx, sys.exit))
    
        ctx = get_context('forkserver')
        print(run(ctx, partial(sys.exit, 2)))
        print(run(ctx, partial(sys.exit, None)))
        print(run(ctx, sys.exit))

when executed results in

    $ python exit.py
    2
    1
    1
    2
    1
    1
    2
    1
    1


but when Python itself is executed we see different behavior

    $ for arg in 2 None ''; do python -c "import sys; sys.exit($arg)"; echo $?; done
    2
    0
    0

The documentation states

> sys.exit([arg])
> ...
> The optional argument arg can be an integer giving the exit status
> (defaulting to zero), or another type of object.

The relevant line in multiprocessing (https://github.com/python/cpython/blame/1cffd0eed313011c0c2bb071c8affeb4a7ed05c7/Lib/multiprocessing/process.py#L307)
seems to be from the original pyprocessing module itself, and I could
not locate an active site that maintains the repository to see if there
was any justification for the behavior.
msg333553 - (view) Author: Emmanuel Arias (eamanu) * Date: 2019-01-13 11:49
The same behavior on 3.8 and 3.5
msg352362 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2019-09-13 16:41
I believe the mentality behind multiprocessing.Process triggering an exit code of 1 when sys.exit() is invoked inside its process is to indicate a non-standard exit out of its execution.  There may yet be other side effects that could be triggered by having a sys.exit(0) translate into an exit code of 0 from the Process's process -- and we might not notice them with the current tests.

Was there a particular use case that motivates this suggested change?
msg352396 - (view) Author: Christopher Hunt (chrahunt) * Date: 2019-09-13 23:17
> I believe the mentality behind multiprocessing.Process triggering an exit code of 1 when sys.exit() is invoked inside its process is to indicate a non-standard exit out of its execution.

Can I ask what this is based on? I did a pretty good amount of digging but didn't find any justification for it. It just seems like a simple oversight to me.

> There may yet be other side effects that could be triggered by having a sys.exit(0) translate into an exit code of 0 from the Process's process -- and we might not notice them with the current tests.

This is definitely a behavior change and will break any code that currently relies on `sys.exit(None)` or `sys.exit()` exiting with a non-zero exit code from a multiprocessing.Process. The fact that all documentation indicates that `sys.exit(None)` or `sys.exit()` results in a 0 exit code in normal Python (with no documentation on it related to multiprocessing) makes me think that any code relying on this behavior is subtly broken, however. Any impacted user can update their code and explicitly pass 1 to `sys.exit`, which should be forward and backwards compatible.

> Was there a particular use case that motivates this suggested change?

I have a wrapper library that invokes arbitrary user code and attempts to behave as if that code was executed in a vanilla Python process, to include propagating the correct exit code.

Currently I have a workaround here: https://github.com/chrahunt/quicken/blob/2dd00a5f024d7b114b211aad8a2618ec8f101956/quicken/_internal/server.py#L344-L353, but it would be nice to get rid of it in 5-6 years if this fix gets in and the non-conformant Python versions fall out of support. :)
msg356841 - (view) Author: Christopher Hunt (chrahunt) * Date: 2019-11-18 05:17
Any other concerns here?
msg362396 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2020-02-21 09:33
New changeset c2ac4cf040ea950bf552d1e77bea613a1a5474fe by Christopher Hunt in branch 'master':
bpo-35727: Use exit code 0 on sys.exit() in multiprocessing.Process. (GH-11538)
https://github.com/python/cpython/commit/c2ac4cf040ea950bf552d1e77bea613a1a5474fe
msg362397 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2020-02-21 09:34
Sorry for the delay.  I've now merged the PR for 3.9.  Since this is a slight behaviour change, and the original issue was easy to workaround, I won't backport it.
History
Date User Action Args
2020-02-21 09:34:23pitrousetstatus: open -> closed
versions: + Python 3.9, - Python 3.7, Python 3.8
messages: + msg362397

resolution: fixed
stage: patch review -> resolved
2020-02-21 09:33:08pitrousetnosy: + pitrou
messages: + msg362396
2019-11-18 05:17:47chrahuntsetmessages: + msg356841
2019-09-13 23:17:26chrahuntsetmessages: + msg352396
2019-09-13 16:41:20davinsetmessages: + msg352362
2019-05-13 13:00:12cheryl.sabellasetnosy: + davin

versions: - Python 3.5
2019-01-13 11:49:14eamanusetnosy: + eamanu

messages: + msg333553
versions: + Python 3.5, Python 3.8
2019-01-12 20:23:42chrahuntsetstage: patch review
pull_requests: + pull_request11145
2019-01-12 20:23:40chrahuntsetstage: (no value)
pull_requests: + pull_request11144
2019-01-12 20:23:36chrahuntsetstage: (no value)
pull_requests: + pull_request11143
2019-01-12 20:08:41chrahuntsetversions: - Python 2.7, Python 3.4, Python 3.5, Python 3.6
2019-01-12 19:33:37chrahuntcreate