classification
Title: multiprocessing freeze_support needed outside win32
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: bbayles, dancol, davin, miss-islington, paul.moore, pitrou, ronaldoussoren, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2017-11-27 10:33 by dancol, last changed 2019-01-22 09:36 by pitrou.

Pull Requests
URL Status Linked Edit
PR 5195 open bbayles, 2018-01-15 19:44
PR 5850 merged bbayles, 2018-02-24 14:34
PR 11511 merged miss-islington, 2019-01-10 17:51
PR 11511 merged miss-islington, 2019-01-10 17:51
PR 11511 merged miss-islington, 2019-01-10 17:51
Messages (11)
msg307043 - (view) Author: Daniel Colascione (dancol) Date: 2017-11-27 10:33
multiprocessing's freeze_support makes freshly-launched subprocesses integrate correctly until the multiprocessing ecosystem even when the main executable is some application binary instead of a Python interpreter. The documentation and code assume that this support is needed only on win32, but it's equally applicable to POSIX systems using the spawn strategy in combination with systems like cx_freeze.

In particular:

1) The special case in context.py's version of freeze_support() for win32 should be removed

2) The semaphore tracker should correctly tickle the freeze support

3) The documentation should be updated to be platform-neutral
msg307527 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-12-03 22:51
Thank you for reporting this.  Would you like to submit a PR?
msg309886 - (view) Author: bbayles (bbayles) * Date: 2018-01-13 03:11
I ran into this issue, and found references to it on StackOverflow [1] and GitHub [2] as well. I found that the problem applies to both the 'spawn' and 'forkserver' start methods on Linux.

I made an attempt to implement dancol's fix above. (1) and (3) are straightforward, but if there's an elegant way to do the "tickle" in (2) I wasn't able to figure it out?

My branch [3] has something that seems to work with the 'spawn' method and cx_Freeze. Perhaps someone could look at that and tell me if I'm on a reasonable track? If so, I can attempt to extend the method to cover 'forkserver' as well.

[1] https://stackoverflow.com/q/47325297/353839

[2] https://github.com/anthony-tuininga/cx_Freeze/issues/264

[3] bpo-32146-freeze_support">https://github.com/python/cpython/compare/master...bbayles:bpo-32146-freeze_support
msg311274 - (view) Author: bbayles (bbayles) * Date: 2018-01-30 14:34
I presume it's too late to get my pull request in for 3.7, but perhaps we could get a note in the documentation for this before release?

That could be as simple as: 

    .. warning::
    
    The ``'spawn'`` and ``'forkserver'`` start methods cannot currently be used with "frozen" executables (i.e., binaries produced by packages like **PyInstaller** and **cx_Freeze**) on Unix. The ``'fork'`` start method does work.

I will make that into a PR if it seems like a good option.
msg333411 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-01-10 17:46
I'm sorry about not seeing your PR before.  Would you like to update it against current git master?

(note I'm not thrilled by the use of the "ast" module; I hope we can change the command-line args so that this isn't needed)
msg333412 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-10 17:51
New changeset bab4bbb4c9cd5d25ede21a1b8c99d56e3b8dae9d by Victor Stinner (Bo Bayles) in branch 'master':
bpo-32146: Add documentation about frozen executables on Unix (GH-5850)
https://github.com/python/cpython/commit/bab4bbb4c9cd5d25ede21a1b8c99d56e3b8dae9d
msg333414 - (view) Author: miss-islington (miss-islington) Date: 2019-01-10 18:13
New changeset b9cd38f928f7eb4e18ad4b63e5c49c05c626c33e by Miss Islington (bot) in branch '3.7':
bpo-32146: Add documentation about frozen executables on Unix (GH-5850)
https://github.com/python/cpython/commit/b9cd38f928f7eb4e18ad4b63e5c49c05c626c33e
msg333432 - (view) Author: bbayles (bbayles) * Date: 2019-01-11 01:28
Thanks for the note. I've merged in master and fixed a conflict in the test file.

In an earlier rev I tried to do the argument parsing without ast.literal_eval, but found it awkward to support all the ways [1] can manifest. You might have a better idea?


[1] https://github.com/python/cpython/blob/3.7/Lib/multiprocessing/forkserver.py#L106-L107
msg333460 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-01-11 11:37
The key here would be to replace function execution with module execution and command-line arguments.

So instead of:

  python -c 'from multiprocessing.forkserver import main; main(ARG1, ARG2...)'

you would do:

  python -m multiprocessing.forkserver ARG1 ARG2 ....

Then it should become easy to extract the concatenated command lines.

(and if that works, please do the same for semaphore_tracker ;-))
msg333533 - (view) Author: bbayles (bbayles) * Date: 2019-01-12 22:41
I'm currently having difficulty getting either cx_Freeze or PyInstaller to work with Python 3.8, which is hindering testing. I also no longer have easy access to a Windows environment.

If I change the implementation to use -m instead of -c would you be able to test?
msg334201 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-01-22 09:36
@bbayles Hopefully a Windows user could test if you give a step-by-step guide of what to check for.
History
Date User Action Args
2019-01-22 09:36:32pitrousetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
messages: + msg334201
2019-01-15 09:15:49ronaldoussorensetnosy: + ronaldoussoren
2019-01-12 22:41:21bbaylessetmessages: + msg333533
2019-01-11 11:37:08pitrousetmessages: + msg333460
versions: + Python 3.8, - Python 3.6
2019-01-11 01:28:52bbaylessetmessages: + msg333432
2019-01-10 18:13:25miss-islingtonsetnosy: + miss-islington
messages: + msg333414
2019-01-10 17:52:01miss-islingtonsetpull_requests: + pull_request11066
2019-01-10 17:51:53miss-islingtonsetpull_requests: + pull_request11065
2019-01-10 17:51:44miss-islingtonsetpull_requests: + pull_request11064
2019-01-10 17:51:30vstinnersetnosy: + vstinner
messages: + msg333412
2019-01-10 17:46:57pitrousetmessages: + msg333411
2018-02-24 14:34:43bbaylessetpull_requests: + pull_request5626
2018-01-30 14:34:47bbaylessetmessages: + msg311274
2018-01-15 19:44:39bbaylessetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request5049
2018-01-13 03:11:23bbaylessetnosy: + bbayles
messages: + msg309886
2017-12-03 22:51:59pitrousetversions: - Python 2.7, Python 3.4, Python 3.5, Python 3.8
nosy: + pitrou, davin

messages: + msg307527

stage: needs patch
2017-11-27 10:33:55dancolcreate