classification
Title: More lost updates with multiprocessing.Value and .Array
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.6
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Just a Person, davin, eryksun, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2017-01-22 17:02 by Just a Person, last changed 2017-01-22 17:58 by eryksun. This issue is now closed.

Files
File name Uploaded Description Edit
2017-01-22-1140-25.mp4 Just a Person, 2017-01-22 17:02 Video of problem
Messages (4)
msg286015 - (view) Author: Just a Person (Just a Person) Date: 2017-01-22 17:02
Lately, I have been having trouble using the multiprocessing library's shared memory on Windows. Often, updating the .value property seems to fail for no reason. As shown in the attached video, changing ```if __name__ == '__main__':``` in the sample code from the documentation to  ```if True:``` causes the program to not work.

This issue does not arise under Linux as far as I can tell (testing the same code).

Any explanation/fix would be helpful.
msg286017 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2017-01-22 17:28
I'm having difficulty watching your video attachment.  Would it be possible to instead describe, preferably with example code that others can similarly try to reproduce the behavior, what you're experiencing?

Please keep in mind what the documentation repeatedly advises about the need for capturing your process-creating multiprocessing calls inside a "if __name__ == '__main__'" clause, especially on Windows platforms.
msg286019 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-01-22 17:43
Change the last print to test the exit code, as follows:

    if p.exitcode != 0:
        print('Child failed with exit code:', p.exitcode)
    else:
        print(num.value)
        print(arr[:])

Note that when you fail to limit creating a new process to just the "__main__" script, the child process ends up failing. Run it in a command prompt to see the following RuntimeError from the child:

    RuntimeError:
            An attempt has been made to start a new process before the
            current process has finished its bootstrapping phase.

            This probably means that you are not using fork to start your
            child processes and you have forgotten to use the proper idiom
            in the main module:

                if __name__ == '__main__':
                    freeze_support()
                    ...

            The "freeze_support()" line can be omitted if the program
            is not going to be frozen to produce an executable.

Since there's no fork() in the Windows API (the NT kernel actually implements forking for other subsystems such as the old POSIX system and the new Linux system, but the Windows API wasn't designed for it), multiprocessing has to import the main script in a new process, in which case the __name__ is not "__main__". But since you're not checking the __name__, it ends up trying to start another Process instance. You're lucky that this API has been made smart enough to detect this. It used to blow up recursively creating processes.
msg286022 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-01-22 17:58
>    if p.exitcode != 0:
>        print('Child failed with exit code:', p.exitcode)

Interestingly the exit code is always 1 when running under pythonw.exe. I just inspected what's going on by setting sys.stderr to a file and discovered the following issue:

      File "C:\Program Files\Python35\lib\multiprocessing\process.py", line 268, in _bootstrap
        sys.stdout.flush()
    AttributeError: 'NoneType' object has no attribute 'flush'

Calling flush() on stderr and stdout should be gated by a check that they're not None and have a flush method. Or simply ignore the AttributeError.
History
Date User Action Args
2017-01-22 17:58:23eryksunsetmessages: + msg286022
2017-01-22 17:43:32eryksunsetstatus: open -> closed

nosy: + eryksun
messages: + msg286019

resolution: not a bug
stage: resolved
2017-01-22 17:28:23davinsetnosy: + davin
messages: + msg286017
2017-01-22 17:02:49Just a Personcreate