classification
Title: multiprocessing on Windows does not properly manage class attributes
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, icordasc, paul.moore, r.david.murray, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2016-07-29 10:54 by icordasc, last changed 2016-07-29 14:23 by r.david.murray. This issue is now closed.

Messages (4)
msg271618 - (view) Author: Ian Cordasco (icordasc) * Date: 2016-07-29 10:54
In trying to add support for multiprocessing on Windows in Flake8, I found that the behaviour of the module is significantly different than on Unix. If you have a class that has class-level attributes on Unix, and you change them, put the class into a Queue (or have it attached to an object that moves through a Queue) then the values you set will be preserved [1]. On Windows [2], the attributes are reset.


As a minimal reproduction, I've thrown together this project: https://github.com/sigmavirus24/windows-multiprocessing-bug which is running tox on AppVeyor (https://ci.appveyor.com/project/sigmavirus24/windows-multiprocessing-bug) and Travis CI (https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug) so the behaviours can be compared.

For application developers, it would be wonderful if we could rely on the standard library behaving the same way in this case across both operating systems.

[1] https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug/jobs/148276925#L158 https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug/jobs/148276926#L158 https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug/jobs/148276927#L157

[2] https://ci.appveyor.com/project/sigmavirus24/windows-multiprocessing-bug/build/1.0.2#L24
msg271622 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-07-29 13:51
Queuing the class doesn't do anything here. It gets pickled as a module attribute; the queue isn't marshaling the class dict. For example:

    >>> pickletools.dis(pickle.dumps(mccabe.McCabeChecker))
        0: \x80 PROTO      3
        2: c    GLOBAL     'mccabe McCabeChecker'
       24: q    BINPUT     0
       26: .    STOP
    highest protocol among opcodes = 2

Unpickling this is basically getattr(sys.modules['mccabe'], 'McCabeChecker'). So you'll see the same result if you only put ('max_complexity', 10) in the queue and hard code the class in the _target function. 

Of course for a system that uses fork() the class attribute is the same in the child process. In 3.4+ you can force spawning by initially calling multiprocessing.set_start_method('spawn'). Then you'll see the same result as in Windows.
msg271627 - (view) Author: Ian Cordasco (icordasc) * Date: 2016-07-29 14:20
Why did you remove Python 3.3? It's still affected by this behaviour.
msg271628 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-07-29 14:23
Right this is a fork vs spawn issue, not a bug.

I don't know why Eryk changed the versions the way he did, but FYI we use that field to indicate which version we will apply a patch to, so it is pretty much irrelevant when there isn't actually a bug :)
History
Date User Action Args
2016-07-29 14:23:49r.david.murraysetstatus: open -> closed

nosy: + r.david.murray
messages: + msg271628

resolution: not a bug
stage: resolved
2016-07-29 14:20:47icordascsetmessages: + msg271627
2016-07-29 13:51:42eryksunsetnosy: + eryksun

messages: + msg271622
versions: - Python 3.3
2016-07-29 10:54:15icordasccreate