In trying to add support for multiprocessing on Windows in Flake8, I found that the behaviour of the module is significantly different than on Unix. If you have a class that has class-level attributes on Unix, and you change them, put the class into a Queue (or have it attached to an object that moves through a Queue) then the values you set will be preserved [1]. On Windows [2], the attributes are reset.


As a minimal reproduction, I've thrown together this project: https://github.com/sigmavirus24/windows-multiprocessing-bug which is running tox on AppVeyor (https://ci.appveyor.com/project/sigmavirus24/windows-multiprocessing-bug) and Travis CI (https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug) so the behaviours can be compared.

For application developers, it would be wonderful if we could rely on the standard library behaving the same way in this case across both operating systems.

[1] https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug/jobs/148276925#L158 https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug/jobs/148276926#L158 https://travis-ci.org/sigmavirus24/windows-multiprocessing-bug/jobs/148276927#L157

[2] https://ci.appveyor.com/project/sigmavirus24/windows-multiprocessing-bug/build/1.0.2#L24

Queuing the class doesn't do anything here. It gets pickled as a module attribute; the queue isn't marshaling the class dict. For example:

    >>> pickletools.dis(pickle.dumps(mccabe.McCabeChecker))
        0: \x80 PROTO      3
        2: c    GLOBAL     'mccabe McCabeChecker'
       24: q    BINPUT     0
       26: .    STOP
    highest protocol among opcodes = 2

Unpickling this is basically getattr(sys.modules['mccabe'], 'McCabeChecker'). So you'll see the same result if you only put ('max_complexity', 10) in the queue and hard code the class in the _target function. 

Of course for a system that uses fork() the class attribute is the same in the child process. In 3.4+ you can force spawning by initially calling multiprocessing.set_start_method('spawn'). Then you'll see the same result as in Windows.

Why did you remove Python 3.3? It's still affected by this behaviour.

Right this is a fork vs spawn issue, not a bug.

I don't know why Eryk changed the versions the way he did, but FYI we use that field to indicate which version we will apply a patch to, so it is pretty much irrelevant when there isn't actually a bug :)