classification
Title: Conflict between tqdm and multiprocessing on windows
Type: compile error Stage:
Components: Windows Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: paul.moore, ronaldoussoren, schwemro, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2018-04-16 10:21 by schwemro, last changed 2018-04-16 15:33 by schwemro.

Files
File name Uploaded Description Edit
tqdm_multi.py schwemro, 2018-04-16 10:21
Messages (5)
msg315353 - (view) Author: (schwemro) Date: 2018-04-16 10:21
Apparently, there occurs a conflict between tqdm and multiprocessing. There is an AttributeError displayed. I provide here an minimal working example. It works on UNIX but unfortunately not on windows.
msg315355 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-04-16 13:32
What's the specific exception you are getting?

BTW. This involves third-party code and this may be a bug in that code. An important difference between linux and windows is how multiprocessing launches additional processes.
msg315356 - (view) Author: (schwemro) Date: 2018-04-16 13:55
Here is the traceback:

Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 
16:13:55) [MSC v.1900 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 6.3.1 -- An enhanced Interactive Python.


runfile('C:/Users/Downloads/temp.py', 
wdir='C:/Users/Downloads')
Traceback (most recent call last):

File "<ipython-input-1-8046ea5eaef4>", line 1, in <module>
runfile('C:/Users/Downloads/temp.py', 
wdir='C:/Users/Downloads')

File "C:\ProgramData\Anaconda3\lib\site- 
packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
execfile(filename, namespace)

File "C:\ProgramData\Anaconda3\lib\site- 
packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "C:/Users/Downloads/temp.py", line 22, in <module>
p.map(progresser, L)

File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 
266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()

File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 
644, in get
raise self._value

AttributeError: 'NoneType' object has no attribute 'write'

Where can I find more details about differences between the two OS? By the way, I'm working on mac.
msg315360 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2018-04-16 15:27
Multiprocessing by default uses the fork system call to start new processes on Linux. This system call is not available on Windows, and there multiprocessing starts a fresh interpreter (see <https://docs.python.org/3.6/library/multiprocessing.html#multiprocessing.get_start_method>). 

I'm also on macOS, and cannot reproduce the problem there even when using the 'spawn' method there by adding some lines to the start of your script (before the other import statements):

import multiprocessing
if __name__ == "__main__":
    multiprocessing.set_start_method('spawn')

But: I have a fairly old version of 3.6 on my machine.
msg315361 - (view) Author: (schwemro) Date: 2018-04-16 15:33
On macOS it works perfectly for me as well. The issue is about running it on windows 10...
History
Date User Action Args
2018-04-16 15:33:42schwemrosetmessages: + msg315361
2018-04-16 15:27:37ronaldoussorensetmessages: + msg315360
2018-04-16 13:55:58schwemrosetmessages: + msg315356
2018-04-16 13:32:37ronaldoussorensetnosy: + ronaldoussoren
messages: + msg315355
2018-04-16 10:21:46schwemrocreate