classification
Title: Implement PEP 371: multiprocessing module
Type: enhancement Stage:
Components: Extension Modules Versions: Python 3.0, Python 2.6
process
Status: closed Resolution: accepted
Dependencies: 1683 3042 Superseder:
Assigned To: benjamin.peterson Nosy List: alecthomas, benjamin.peterson, georg.brandl, giampaolo.rodola, jnoller, roudkerk, tim.golden
Priority: release blocker Keywords: patch

Created on 2008-06-06 11:57 by jnoller, last changed 2008-06-11 18:14 by jnoller. This issue is now closed.

Files
File name Uploaded Description Edit
multiprocessing_docs.patch benjamin.peterson, 2008-06-10 02:39
mpdocs.diff georg.brandl, 2008-06-10 07:12
multiprocessing_patch_complete.patch.jnoller.v1.bz2 jnoller, 2008-06-10 18:58
py3k_failing.diff benjamin.peterson, 2008-06-11 13:46
py3k_fixed.diff jnoller, 2008-06-11 15:25
pcbuild.patch tim.golden, 2008-06-11 16:31 Patch against r64120 to add build instructions for multiprocessing module under Windows
news.diff jnoller, 2008-06-11 18:14
Messages (25)
msg67763 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-06 11:57
This issue tracks the work pending for the inclusion of the pyprocessing 
module under the multiprocessing name for Python 2.6 and 3.0. If needed, 
additional tickets will be created for issues which will need to be 
addressed after initial inclusion.

Currently open:
- Redo all documentation to match ReST format of stdlib
- Update docs to reflect new PEP8 names
- Update all unit tests to be in Unittest format

Closed:
- Rename module API to be PEP8 compliant (roudkerk)
- Link csource libs into python-trunk build (jnoller)
- redo all local "." style imports to reflect the new namespace 
(jnoller)
msg67884 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-10 01:55
Quick status:

Completed:
- Redo all documentation to match ReST format of stdlib
    - thanks to Benjamin Peterson for the massive conversion he did.
- All unit tests are converted, I'm just chasing down a few lingering 
bugs  that came with the conversion. (jnoller)

Richard and I are ironing out a few remaining API cleanups that came 
about with the renaming and I'll update the docs to reflect the new 
method names.
msg67888 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-06-10 02:35
I'm attaching my version of the docs. (Jesse, I did fix the Sphinx
warnings and some other problems I noticed.) Georg, if you could cast
your expert reST eye over them, I'd be much obliged.
msg67899 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-06-10 07:12
Attaching reviewed docs patch.
msg67905 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-10 13:29
I applied Ben's updated patch and then applied Georg's patch - but now I 
am getting an error doing "make html", unfortunately sphinx/ReST doesn't 
include line numbers with the error:

mify library/miniaeframe library/misc library/mm library/mmap 
library/modulefinder library/modules library/msilib library/msvcrt 
library/multifile library/multiprocessing reST markup error:
/Users/jesse/open_source/subversion/python-
trunk/Doc/library/multiprocessing.rst:: (SEVERE/4) Duplicate ID: 
"module-multiprocessing".
make: *** [build] Error 1

Ben/Georg - did I apply them wrong?
msg67906 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-06-10 13:40
Do this:

svn revert -R .;
rm Doc/library/multiprocessing.rst
rm Doc/includes/mp*.py

Then apply just Georg's patch. That should work.
msg67907 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-10 15:15
Thanks Ben - even with that, for some reason I ended up with duplicate 
chunks and data inside of the .rst file - wiping it all out, and then 
making a blank version of the .rst and then applying the patch from Georg 
worked.
msg67911 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-10 18:58
Attached is rev 1 of a complete patch for inclusion in 2.6. This patch 
was developed off of trunk and includes the unit tests, docs, modules, c 
module, etc. 

2 of the unit tests had to be commented out, the first due to the 
difference in the threading versus process API (is_set vs. isSet) and 
the second is an OS/X issue I am working with Richard on.

Note, some of the examples may be out of date - I am working on those.

A full review of this would be appreciated. Pending the application of 
the patch in issue 1638, this initial revision could be applied to trunk 
for the wednesday build.
msg67912 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-10 19:04
Added to the code review system too: http://codereview.appspot.com/2061
msg67917 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-06-10 20:29
How much work needs to be done to port this to Py3k?
msg67919 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-10 20:50
Richard mentioned it should just work after being run though lib2to3
msg67920 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-06-10 21:00
On Tue, Jun 10, 2008 at 3:50 PM, Jesse Noller <report@bugs.python.org> wrote:
>
> Jesse Noller <jnoller@gmail.com> added the comment:
>
> Richard mentioned it should just work after being run though lib2to3

Excellent! That clears up a main worry of mine.
msg67939 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-06-11 03:33
Congrats, you're in! See r64104 in the trunk.
msg67951 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-06-11 12:06
Please see Georg's Python-dev message about importing multiprocessing.
msg67956 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-11 12:29
Specifically:

>>> import multiprocessing
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jesse/open_source/subversion/python-
trunk/Lib/multiprocessing/__init__.py", line 63, in <module>
import _multiprocessing
AttributeError: 'module' object has no attribute 'BufferTooShort'

This occurs due to the fact that __init__.py imports _multiprocessing 
prior to the definition of the exceptions. This should be moved to the 
ccode. From Georg:

"The test suite passes (at least for some buildbots) because it imports
_multiprocessing first, which then in its init function imports 
multiprocessing
to get the BufferTooShort exception.

Since BufferTooShort and other exceptions inheriting from ProcessError 
are
simple derived exceptions, why aren't they created in the C module in 
the
first place?"
msg67959 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-11 12:39
Additional things to clean up post-beta:

- Review all docs fully to ensure the documented API is still in sync with 
the renames.
- Restore the commented out tests in test_event
- Restore the commented out test in test_bounded_semaphore
msg67975 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2008-06-11 14:20
The _multiprocessing module is not building under Windows at the moment.
Attempting to import multiprocessing (from an .exe build from the
current svn) gives "ImportError: No module named _multiprocessing" and
the test suite skips the test for the same reason.
msg67979 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-06-11 14:24
On Wed, Jun 11, 2008 at 9:20 AM, Tim Golden <report@bugs.python.org> wrote:
>
> Tim Golden <mail@timgolden.me.uk> added the comment:
>
> The _multiprocessing module is not building under Windows at the moment.
> Attempting to import multiprocessing (from an .exe build from the
> current svn) gives "ImportError: No module named _multiprocessing" and
> the test suite skips the test for the same reason.

Can you tell why it's not building?
msg67981 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2008-06-11 14:39
Benjamin Peterson wrote:
> Benjamin Peterson <musiccomposition@gmail.com> added the comment:
> 
> On Wed, Jun 11, 2008 at 9:20 AM, Tim Golden <report@bugs.python.org> wrote:
>> Tim Golden <mail@timgolden.me.uk> added the comment:
>>
>> The _multiprocessing module is not building under Windows at the moment.
>> Attempting to import multiprocessing (from an .exe build from the
>> current svn) gives "ImportError: No module named _multiprocessing" and
>> the test suite skips the test for the same reason.
> 
> Can you tell why it's not building?

I was hoping you wouldn't ask :) I can't run VS visually at the
moment so I'm down to inspecting the .sln file by hand which
I have no experience of. I suspect that someone better versed
in VS solution files than I needs to add the module in. I'll try to
get to it unless someone else chips in.
msg67986 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-11 15:25
Here is a diff for the tests under py3k, I applied the svnmerge, then 
ben's patch and made a couple of import changes.
msg67988 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2008-06-11 15:44
I'm sorry; I've had a look and there's no chance of my updating the
solution and project files by hand; and I can't run Visual Studio at the
moment. I'll try emailing Trent or Christian in the hope that one of
them's available to do it.
msg67992 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2008-06-11 16:31
Trent's supplied me with enough info to patch the project files
manually. The attached patch against r64120 results in the
_multiprocessing module building. I'm running the tests now but I'll
upload the patch in any case.
msg67995 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-06-11 16:58
Thanks very much Tim and Trent. I've merged multiprocessing into Py3k
and added the Windows files. I hope that works.
msg68002 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-11 17:46
Debian PPC build crash:
http://python.org/dev/buildbot/stable/ppc%20Debian%20unstable%203.0/buil
ds/1063/step-test/0

...snip
Re-running test 'test_multiprocessing' in verbose mode
test test_multiprocessing crashed -- <class 'OSError'>: [Errno 38] 
Function not implemented
Traceback (most recent call last):
  File "./Lib/test/regrtest.py", line 601, in runtest_inner
    indirect_test()
  File "/home/pybot/buildarea/3.0.klose-debian-
ppc/build/Lib/test/test_multiprocessing.py", line 1764, in test_main
    ProcessesMixin.pool = multiprocessing.Pool(4)
  File "/home/pybot/buildarea/3.0.klose-debian-
ppc/build/Lib/multiprocessing/__init__.py", line 225, in Pool
    return Pool(processes, initializer, initargs)
  File "/home/pybot/buildarea/3.0.klose-debian-
ppc/build/Lib/multiprocessing/pool.py", line 84, in __init__
    self._setup_queues()
  File "/home/pybot/buildarea/3.0.klose-debian-
ppc/build/Lib/multiprocessing/pool.py", line 131, in _setup_queues
    self._inqueue = SimpleQueue()
  File "/home/pybot/buildarea/3.0.klose-debian-
ppc/build/Lib/multiprocessing/queues.py", line 315, in __init__
    self._rlock = Lock()
  File "/home/pybot/buildarea/3.0.klose-debian-
ppc/build/Lib/multiprocessing/synchronize.py", line 106, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/home/pybot/buildarea/3.0.klose-debian-
ppc/build/Lib/multiprocessing/synchronize.py", line 38, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented
[801671 refs]
make: *** [buildbottest] Error 1
program finished with exit code 2
msg68004 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-06-11 18:14
Small diff for the ACKs/NEWS file
History
Date User Action Args
2008-06-11 18:14:35jnollersetfiles: + news.diff
messages: + msg68004
2008-06-11 17:46:20jnollersetmessages: + msg68002
2008-06-11 16:58:40benjamin.petersonsetstatus: open -> closed
messages: + msg67995
2008-06-11 16:31:38tim.goldensetfiles: + pcbuild.patch
messages: + msg67992
2008-06-11 15:44:15tim.goldensetmessages: + msg67988
2008-06-11 15:26:08jnollersetfiles: + py3k_fixed.diff
messages: + msg67986
2008-06-11 14:39:08tim.goldensetmessages: + msg67981
2008-06-11 14:24:03benjamin.petersonsetmessages: + msg67979
2008-06-11 14:20:04tim.goldensetnosy: + tim.golden
messages: + msg67975
2008-06-11 13:46:45benjamin.petersonsetfiles: + py3k_failing.diff
2008-06-11 12:39:46jnollersetmessages: + msg67959
2008-06-11 12:29:51jnollersetmessages: + msg67956
2008-06-11 12:06:31benjamin.petersonsetstatus: closed -> open
messages: + msg67951
2008-06-11 03:33:17benjamin.petersonsetstatus: open -> closed
resolution: accepted
messages: + msg67939
2008-06-10 21:00:32benjamin.petersonsetmessages: + msg67920
2008-06-10 20:50:46jnollersetmessages: + msg67919
2008-06-10 20:29:28benjamin.petersonsetmessages: + msg67917
2008-06-10 19:04:03jnollersetmessages: + msg67912
2008-06-10 18:58:52jnollersetfiles: + multiprocessing_patch_complete.patch.jnoller.v1.bz2
messages: + msg67911
2008-06-10 15:28:35giampaolo.rodolasetnosy: + giampaolo.rodola
2008-06-10 15:15:08jnollersetmessages: + msg67907
2008-06-10 13:40:18benjamin.petersonsetmessages: + msg67906
2008-06-10 13:29:04jnollersetmessages: + msg67905
2008-06-10 07:12:44georg.brandlsetfiles: + mpdocs.diff
messages: + msg67899
2008-06-10 02:39:48benjamin.petersonsetfiles: + multiprocessing_docs.patch
keywords: + patch
2008-06-10 02:35:40benjamin.petersonsetpriority: release blocker
assignee: benjamin.peterson
messages: + msg67888
nosy: + georg.brandl, benjamin.peterson
2008-06-10 01:55:18jnollersetmessages: + msg67884
2008-06-06 13:17:22alecthomassetnosy: + alecthomas
2008-06-06 13:09:41benjamin.petersonsetdependencies: + Thread local storage and PyGILState_* mucked up by os.fork(), Add PEP 8 compliant aliases to threading module
2008-06-06 11:57:54jnollercreate