classification
Title: Merely importing tkinter breaks parallel code (multiprocessing, sharedmem)
Type: crash Stage:
Components: Documentation, macOS, Tkinter Versions: Python 3.8, Python 3.7, Python 3.6, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, ezwelty, ned.deily, pitrou, ronaldoussoren, terry.reedy
Priority: normal Keywords:

Created on 2018-03-20 19:20 by ezwelty, last changed 2018-07-28 13:27 by ronaldoussoren.

Messages (18)
msg314160 - (view) Author: Ethan Welty (ezwelty) Date: 2018-03-20 19:20
Merely importing tkinter breaks the use of parallel code on my system (Mac OSX 10.11.6, tested on Python 2.7.13 / 2.7.14 / 3.5.0 / 3.6.4, all barebones distributions installed with pyenv). I've tested this with both multiprocessing and sharedmem (see minimal scripts below).

The issue seems to apply only to functions that evoke multithreading within their respective package (e.g. `numpy.matmul()`, `cv2.SIFT.detectAndCompute()`). If I make the matrix in the scripts below much smaller (e.g. change `5000` to `5`), avoiding internal multithreading, the scripts work.

## with `multiprocessing`

```python
import numpy as np
import multiprocessing
import _tkinter

def parallel_matmul(x):
    R = np.random.randn(3, 3)
    return np.matmul(R, x)

pool = multiprocessing.Pool(4)
results = pool.map(parallel_matmul, [np.random.randn(3, 5000) for i in range(2)])
```

> *Code never exits and Python has to be force quit*

## with `sharedmem`

```python
import numpy as np
import sharedmem
import _tkinter

def parallel_matmul(x):
    R = np.random.randn(3, 3)
    return np.matmul(R, x)

with sharedmem.MapReduce() as pool:
    results = pool.map(parallel_matmul,
        [np.random.randn(3, 5000) for i in range(2)])
```

> sharedmem.sharedmem.SlaveException: slave process 1 killed by signal 11
msg314169 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-03-20 22:05
Are you by any chance trying to run this under IDLE?
msg314191 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-03-21 11:44
I'm fairly sure this is expected behavior on macOS: _tkinter loads the Tk library, which loads Apple GUI frameworks. Those frameworks are not save w.r.t. fork(2) and that can lead to all kinds of unwanted behavior (although I would have expected a more explicit crash than this)
msg314212 - (view) Author: Ethan Welty (ezwelty) Date: 2018-03-21 17:14
I have tried running the script with:

- command line (python <file>): Works without (breaks with) `import _tkinter`.
- basic python console (python): Same as with command line.
- ipython: Fails with or without `import _tkinter`. However, if for example I make the matrix in the scripts smaller or use only functions that don't resort to multithreading, code runs successfully in parallel in ipython.
- IDLE: Perhaps unsurprisingly, the code fails with or without `import _tkinter` (never terminates, or prompts endless "Your program is still running! Do you want to kill it?")

Suspecting a problem with Tcl/Tk (Apple's original 8.5.9), I rebuilt python pointing to tcl-tk installed with brew (8.6.8), which I checked with IDLE and _tkinter.TCL_VERSION() / _tkinter.TK_VERSION(). However, this did not fix the problem.
msg314217 - (view) Author: Ethan Welty (ezwelty) Date: 2018-03-21 18:11
I've tried with additional backends: WX, WXAgg, WXCairo, Qt5Agg (in matplotlib speak).

With these, I can at least import matplotlib.pyplot, but as soon as say matplotlib.pyplot.plot is called and the backend is loaded, the code breaks (error message for WXAgg below). Is multiprocessing in an interactive shell simply not meant to be supported on MacOS?

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/Users/Admin/.pyenv/versions/3.6.4/Python.framework/Versions/3.6/lib/python3.6/site-packages/matplotlib/_pylab_helpers.py", line 78, in destroy_all
    manager.destroy()
  File "/Users/Admin/.pyenv/versions/3.6.4/Python.framework/Versions/3.6/lib/python3.6/site-packages/matplotlib/backends/backend_wx.py", line 1303, in destroy
    self.frame.Destroy()
  File "/Users/Admin/.pyenv/versions/3.6.4/Python.framework/Versions/3.6/lib/python3.6/site-packages/matplotlib/backends/backend_wx.py", line 1256, in Destroy
    if not self.IsBeingDeleted():
RuntimeError: wrapped C/C++ object of type FigureFrameWxAgg has been deleted
msg314304 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-03-23 12:19
I cannot reproduce the problem on macOS 10.13.3 with python 3.6 (but haven't spent much time on this).

The combination of GUI code with multiprocessing is a known source of problems on macOS, the system libraries are hostile to that.

Changing the start method from "fork" to "spawn" or "forkserver" should help with this, using multiprocessing.set_start_method(). This should be changed before doing anything that might Apple system frameworks related to GUIs.
msg314339 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-03-23 23:05
The examples in the multiprocessing doc all put all multiprocessing calls within a "if __name__ == '__main__':" statement.  I know that this is necessary on Windows, but don't know if or when it helps on other OSes.
msg314499 - (view) Author: Ethan Welty (ezwelty) Date: 2018-03-27 04:16
Terry Reedy: I just tried your suggestion of a main clause (code below) and it made no difference.

```
import _tkinter
import numpy as np
# multiprocessing.set_start_method("spawn")
import multiprocessing

def parallel_matmul(x):
    R = np.random.randn(3, 3)
    return np.matmul(R, x)

if __name__ == '__main__':
    pool = multiprocessing.Pool(4)
    results = pool.map(parallel_matmul, [np.random.randn(3, 5000) for i in range(2)])
```

Ronald Oussoren: Good to know that perhaps upgrading to the latest OSX would resolve this issue. I tried 'spawn' and 'forkserver' with multiprocessing.set_start_method() (see above) but both result in an endless stream of errors (with or without importing a graphics backend), although that may be my not knowing how to properly use them.
msg314515 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-03-27 11:12
@ezwelty: The import of multiprocessing and the call to set_start_method should be at the very start of the code, before other imports, to avoid the annoying Apple behavior I mentioned.
msg314528 - (view) Author: Ethan Welty (ezwelty) Date: 2018-03-27 14:53
@ronaldoussoren: The order of the imports made no difference. Even with the call at the top, I got endless errors with both 'spawn' and 'forkserver', with or without importing a graphics backend. Only 'fork' works, and only if a graphics package is not imported.

** HOWEVER **

Setting method='spawn' or 'forkserver' and force=True at top, or calling multiprocessing.set_start_method in __main__ does work (when run as scripts from command line). It's a shame to have to give up the convenience of 'fork', but this is a start.

```
import multiprocessing
multiprocessing.set_start_method("spawn", force=True)
import numpy as np
# import _tkinter

def parallel_matmul(x):
    R = np.random.randn(3, 3)
    return np.matmul(R, x)

if __name__ == '__main__':
    # multiprocessing.set_start_method("spawn", force=False)
    pool = multiprocessing.Pool(4)
    results = pool.map(parallel_matmul, [np.random.randn(3, 5000) for i in range(2)])
```
msg321656 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-07-14 18:30
On MacOS, 3.7.0 is compiled for and the installer loads tcl/tk 8.6.8.  The same is true for the 3.6.6 64-bit installer.  Do tkinter and multiprocessing work together better with these installations?

I want to consider using multiprocessing and pipes instead subprocess and socket for IDLE's user-code execution process, started from and communicating with the initial tkinter gui process.  idlelib.run, which runs in the execution process to communicae with the gui process and supervise running user code, imports tkinter.  Besides which, users have to be able to import tkinter in their programs.
msg321693 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-07-15 18:54
As I mentioned in msg314304 this is almost certainly expected behavior on macOS because Apple’s GUI frameworks are not fork()-safe.  There is nothing we can do about this other than changing the default spawn mechanism for multiprocessing on macOS. 

--
On the road, hence brief. 

Op 14 jul. 2018 om 19:30 heeft Terry J. Reedy <report@bugs.python.org> het volgende geschreven:

> 
> Terry J. Reedy <tjreedy@udel.edu> added the comment:
> 
> On MacOS, 3.7.0 is compiled for and the installer loads tcl/tk 8.6.8.  The same is true for the 3.6.6 64-bit installer.  Do tkinter and multiprocessing work together better with these installations?
> 
> I want to consider using multiprocessing and pipes instead subprocess and socket for IDLE's user-code execution process, started from and communicating with the initial tkinter gui process.  idlelib.run, which runs in the execution process to communicae with the gui process and supervise running user code, imports tkinter.  Besides which, users have to be able to import tkinter in their programs.
> 
> ----------
> nosy: +pitrou
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue33111>
> _______________________________________
msg321695 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-07-15 19:11
I was uncertain from Ethan's post what does work best. "endless errors with both 'spawn", "Only 'fork' works...but without graphics", "method='spawn'and ... does work".  Do the 1st and 3rd sentences mean that spawn + graphics fails without the proper additional code but does work with it?  Should the 2nd sentence be "fork only works without graphics" rather than starting with 'only fork'?

If so, this would suggest that the default should depend on whether there is a graphic import ;-).  In any case, how to use successfully use multiprocessing with graphics on Mac should be documented.
msg321723 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-07-16 09:24
macOS users, feel free to propose a doc PR for multiprocessing.
msg321803 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-07-17 07:47
I can provide a patch, but that will likely be during EuroPython as I’m currently taking a roundabout route towards Edinburgh. 

Btw. I’m not sure yet how impactful changing the default spawning mechanism would be for 3.8.  How likely is that to break user code?   

Op 16 jul. 2018 om 10:24 heeft Antoine Pitrou <report@bugs.python.org> het volgende geschreven:

> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
> macOS users, feel free to propose a doc PR for multiprocessing.
> 
> ----------
> assignee:  -> docs@python
> components: +Documentation
> nosy: +docs@python
> versions: +Python 3.7, Python 3.8 -Python 3.5
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue33111>
> _______________________________________
msg321804 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-07-17 07:48
It will definitely break *some* user code. Also, everyone not using tkinter (which I think is the majority of users) isn't affected by this.
msg321845 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-07-17 17:42
Ronald Oussoren> I can provide a patch, but that will likely be during 
EuroPython as I’m currently taking a roundabout route towards Edinburgh.

Can you add a test of the procedure you document for using _tkinter with 
multiprocessing?
msg322560 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-07-28 13:27
That's annoying, I cannot reproduce the issue on macOS 10.13. I don't have access to my test VM running 10.11 at the moment, I'll work on documenting this when I get back home (and do have access to 10.11 again).

BTW. I don't know if providing a reliable test is possible other than testing that the documented procedure works in general.
History
Date User Action Args
2018-07-28 13:27:43ronaldoussorensetmessages: + msg322560
2018-07-17 17:42:21terry.reedysetmessages: + msg321845
2018-07-17 07:48:36pitrousetmessages: + msg321804
2018-07-17 07:47:16ronaldoussorensetmessages: + msg321803
2018-07-16 09:24:25pitrousetversions: + Python 3.7, Python 3.8, - Python 3.5
nosy: + docs@python

messages: + msg321723

assignee: docs@python
components: + Documentation
2018-07-15 19:11:06terry.reedysetmessages: + msg321695
2018-07-15 18:54:30ronaldoussorensetmessages: + msg321693
2018-07-14 18:30:49terry.reedysetnosy: + pitrou
messages: + msg321656
2018-03-27 14:53:47ezweltysetmessages: + msg314528
2018-03-27 11:12:36ronaldoussorensetmessages: + msg314515
2018-03-27 04:16:30ezweltysetmessages: + msg314499
2018-03-23 23:05:42terry.reedysetnosy: + terry.reedy
messages: + msg314339
2018-03-23 12:19:37ronaldoussorensetmessages: + msg314304
2018-03-21 19:30:53ezweltysetcomponents: + macOS
2018-03-21 18:11:42ezweltysetmessages: + msg314217
2018-03-21 17:14:36ezweltysetmessages: + msg314212
2018-03-21 11:44:05ronaldoussorensetnosy: + ronaldoussoren
messages: + msg314191
2018-03-20 22:05:27ned.deilysetnosy: + ned.deily
messages: + msg314169
2018-03-20 19:23:50ezweltysetversions: - Python 3.8
2018-03-20 19:20:56ezweltycreate