classification
Title: Multiprocessing not timely flushing stack trace to stderr
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: brandjon, davin, jnoller, memeplex, sbt
Priority: normal Keywords:

Created on 2015-08-27 21:20 by memeplex, last changed 2021-06-21 16:08 by iritkatriel. This issue is now closed.

Messages (3)
msg249262 - (view) Author: Memeplex (memeplex) Date: 2015-08-27 21:20
Related to but not the same than https://bugs.python.org/issue13812.

Try this:

```
import multiprocessing as mp
import time


def g():
    time.sleep(100)

def f():
    mp.Process(target=g).start()
    1/0

mp.Process(target=f).start()
```

It won't show the ZeroDivisionError until you keyboard interrupt the g() process or wait for it to end. This is because _exit_function will join every active non-daemon child, which happens before printing and flushing the error. IMO the exception should be shown before joining children, since keeping the error silent is asking for trouble.
msg249264 - (view) Author: Memeplex (memeplex) Date: 2015-08-27 21:41
One possible fix to multiprocessing/process.py:

<             try:
<                 self.run()
<                 exitcode = 0
<             finally:
<                 util._exit_function()
---
>             self.run()
>             exitcode = 0
274a272
>             util._exit_function()

This removes the try/finally pair around self.run() and calls _exit_function in the last finally clause. It doesn't honour the original control flow, as this last clause is even executed when the flow is aborted before reaching self.run(). That said, I can't see any particular reason to prefer the original flow.
msg250534 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-09-12 21:50
If I understand your motivations correctly, I'd restate them as:  you want a mechanism for being immediately notified of an exception in a parent process without waiting on any child processes of that parent to finish and furthermore propose this should be the default behavior.

Quick side note:  As the docs explain and as the code you propose modifying in Lib/multiprocessing/process.py shows, a Process is designed to track its live children.  Interrupting or otherwise destroying a parent process's ability to track its children even after experiencing an exception in its target function would not be desirable.

I think we agree that the current behavior you describe is not catastrophic in any sense -- the exception occurs, the process finishes waiting on its children, and that exception bubbles up at the end -- nothing is lost or suppressed.

This becomes a question of when should a calling routine be notified of such an exception (happening in what I was just calling a parent process):  immediately, if we ask for it, or when it's otherwise done?  Different use cases motivate different answers to this question.  The current behavior satisfies many common use cases and  thankfully it is not difficult to implement the other behaviors in your own code.  It can start with a try-except inside the example function 'f' where the except clause's code takes some appropriate action right away rather than waiting on the children, if that is what is desired.

    def f():
        try:
            p = mp.Process(target=g)
            p.start()
            1/0
            p.join()
        except Exception as e:
            # Take action, sound the alarm, call out the guards, notify the BDFL!
            raise e


I'm not sure I see the relation to issue13812 which is concerned with traceback text that was sent to stderr but surprisingly wasn't getting flushed to stderr.  This issue has no such problem with flushes on stderr -- we are not waiting on stderr here, we are waiting on the parent to finish waiting on its children before any traceback gets written to stderr.
History
Date User Action Args
2021-06-21 16:08:14iritkatrielsetstatus: open -> closed
resolution: not a bug
stage: resolved
2015-09-12 21:53:00davinsetnosy: + jnoller, sbt
2015-09-12 21:52:42davinsettype: behavior
2015-09-12 21:50:09davinsetnosy: + davin
messages: + msg250534
2015-08-27 23:14:25brandjonsetnosy: + brandjon
2015-08-27 21:41:59memeplexsetmessages: + msg249264
2015-08-27 21:20:29memeplexcreate