classification
Title: multiprocessing page leaves out important part of Pool example
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: BreamoreBoy, Chris.Curvey, berker.peksag, davin, docs@python, jnoller, python-dev, rhettinger, sbt
Priority: normal Keywords: patch

Created on 2013-08-01 20:30 by Chris.Curvey, last changed 2016-01-21 22:08 by berker.peksag. This issue is now closed.

Files
File name Uploaded Description Edit
issue18620_py35.patch davin, 2015-02-06 21:11 (default) changes to apply_async code examples review
issue18620_py34.patch davin, 2015-02-06 21:13 (3.4) changes to apply_async code examples review
issue18620_py27.patch davin, 2015-02-06 21:14 (2.7) changes to apply_sync code examples review
Messages (6)
msg194115 - (view) Author: Chris Curvey (Chris.Curvey) Date: 2013-08-01 20:30
on http://docs.python.org/2/library/multiprocessing.html, there is a bit about how to use a Pool properly, which looks like this

pool = Pool(processes=4)              # start 4 worker processes
result = pool.apply_async(f, [10])

What this neglects to mention is that only one process will get any of the work.  If you really want four processes in the pool to work, you have to call apply_async four times.  For example:

results = []
pool = Pool(processes=4)
for i in xrange(4):
    results.append(pool.apply_async(f, [10]))

hat tip to http://stackoverflow.com/questions/12483512/python-multiprocessing-apply-async-only-uses-one-process
msg225037 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-08-07 20:20
Looks as if the v2 and v3 docs need changing.
msg235497 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-02-06 21:11
Attached are proposed patches for 2.7, 3.4, and default (3.5) branches.

In these patches, the 2 examples in the documentation that showcase the use of pool.apply_async have been modified to specifically highlight that a single invocation of apply_async will only lead to a single process doing any work (not all of the pool's processes).

Specifically, in the first code example:
* In the 3.x branches, the example is expanded slightly to demonstrate that only a single process is used by a single apply_async call and that to accomplish similar to the mapping examples just above it, multiple apply_async calls are necessary.
* Also in the 3.x branches, the code deliberately causes an exception to be raised which will terminate the execution of this example when anyone attempts to run it.  Rather than provide an example which terminates prematurely, the exception is now caught and a message displayed to demonstrate the exception did indeed occur, permitting the example to run through to a successful exit.
* In the 2.7 branch, this first example is much smaller than what is in use in the 3.x branches -- I've expanded it to cover the same information as is being shared in the 3.x branches now.

Specifically, in the second code example:
* In the 3.x branches, the only change is to the comment to the right of the apply_async call which emphasizes that only a single process is being used.  Also, for clarity, the reference to the exception being raised in the last line was changed to highlight that it's an exception defined in the multiprocessing library (multiprocessing.TimeoutError).
* The 2.7 branch is the same -- there were no other differences on this example between branches.

These updated examples have been tested with the latest from 2.7, 3.4, and default/3.5 branches on OSX 10.10 and Windows 7.
msg258771 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-01-21 22:00
New changeset c89f4f59872f by Berker Peksag in branch '3.5':
Issue #18620: Improve Pool examples in multiprocessing documentation
https://hg.python.org/cpython/rev/c89f4f59872f

New changeset 9f201578d8d9 by Berker Peksag in branch 'default':
Issue #18620: Improve Pool examples in multiprocessing documentation
https://hg.python.org/cpython/rev/9f201578d8d9
msg258772 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-01-21 22:06
New changeset 9480e217ade0 by Berker Peksag in branch '2.7':
Issue #18620: Improve Pool examples in multiprocessing documentation
https://hg.python.org/cpython/rev/9480e217ade0
msg258773 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-01-21 22:08
Thanks for the patch, Davin. I tweaked code style a bit and removed trailing whitespaces.
History
Date User Action Args
2016-01-21 22:08:58berker.peksagsetstatus: open -> closed

versions: - Python 3.4
nosy: + berker.peksag

messages: + msg258773
resolution: fixed
stage: patch review -> resolved
2016-01-21 22:06:58python-devsetmessages: + msg258772
2016-01-21 22:00:04python-devsetnosy: + python-dev
messages: + msg258771
2015-11-19 22:24:30rhettingersetassignee: docs@python -> rhettinger
2015-09-20 22:27:32davinsetversions: + Python 3.6
2015-03-26 15:44:18davinsetnosy: + rhettinger
2015-03-10 20:24:05serhiy.storchakasetnosy: + jnoller, sbt
2015-02-06 21:14:09davinsetfiles: + issue18620_py27.patch
2015-02-06 21:13:45davinsetfiles: + issue18620_py34.patch
2015-02-06 21:11:44davinsetfiles: + issue18620_py35.patch
keywords: + patch
messages: + msg235497

stage: patch review
2015-01-29 23:07:57davinsetnosy: + davin
2014-08-07 20:20:44BreamoreBoysetversions: + Python 2.7, Python 3.4, Python 3.5
nosy: + BreamoreBoy

messages: + msg225037

type: enhancement -> behavior
2013-08-01 20:30:44Chris.Curveycreate