classification
Title: test_multiprocessing fails on systems with HAVE_SEM_OPEN=0
Type: Stage:
Components: Extension Modules Versions: Python 3.0, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jnoller Nosy List: aimacintyre, bms, djmdjm, jnoller, ncoghlan
Priority: critical Keywords: needs review, patch

Created on 2008-09-04 00:23 by djmdjm, last changed 2009-04-02 03:57 by jnoller. This issue is now closed.

Files
File name Uploaded Description Edit
issue3770.diff jnoller, 2008-09-29 20:07 Disable synchronize module on sem_open lacking systems
Messages (27)
msg72442 - (view) Author: Damien Miller (djmdjm) Date: 2008-09-04 00:23
test_multiprocessing crashes on platforms that lack a working
sem_open(), despite it being turned off at compilation time by setting
HAVE_SEM_OPEN=0 in the Extension macros in setup.py

I think the multiprocessing module should disable the functionality
gracefully when it is missing from _multiprocessing.

Failure message:

test test_multiprocessing crashed -- <type 'exceptions.AttributeError'>:
'module' object has no attribute 'SemLock'
Traceback (most recent call last):
  File ".//Lib/test/regrtest.py", line 556, in runtest_inner
    indirect_test()
  File
"/usr/ports/lang/python/2.6/w-Python-2.6b3/Python-2.6b3/Lib/test/test_multiprocessing.py",
line 1758, in test_main
    ProcessesMixin.pool = multiprocessing.Pool(4)
  File
"/usr/ports/lang/python/2.6/w-Python-2.6b3/Python-2.6b3/Lib/multiprocessing/__init__.py",
line 226, in Pool
    return Pool(processes, initializer, initargs)
  File
"/usr/ports/lang/python/2.6/w-Python-2.6b3/Python-2.6b3/Lib/multiprocessing/pool.py",
line 84, in __init__
    self._setup_queues()
  File
"/usr/ports/lang/python/2.6/w-Python-2.6b3/Python-2.6b3/Lib/multiprocessing/pool.py",
line 130, in _setup_queues
    from .queues import SimpleQueue
  File
"/usr/ports/lang/python/2.6/w-Python-2.6b3/Python-2.6b3/Lib/multiprocessing/queues.py",
line 22, in <module>
    from multiprocessing.synchronize import Lock, BoundedSemaphore,
Semaphore, Condition
  File
"/usr/ports/lang/python/2.6/w-Python-2.6b3/Python-2.6b3/Lib/multiprocessing/synchronize.py",
line 29, in <module>
    SEM_VALUE_MAX = _multiprocessing.SemLock.SEM_VALUE_MAX
AttributeError: 'module' object has no attribute 'SemLock'
1 test failed:
    test_multiprocessing
msg72450 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-09-04 01:30
Which platforms is this appearing on?
msg72455 - (view) Author: Damien Miller (djmdjm) Date: 2008-09-04 01:52
On Thu, 4 Sep 2008, Jesse Noller wrote:

> 
> Jesse Noller <jnoller@gmail.com> added the comment:
> 
> Which platforms is this appearing on?

OpenBSD, with this section added to setup.py:

@@ -1269,6 +1268,14 @@ class PyBuildExt(build_ext):
                 )
             libraries = []

+        elif platform.startswith('openbsd'):
+            macros = dict(                  # OpenBSD
+                HAVE_SEM_OPEN=0,            # Not implemented
+                HAVE_SEM_TIMEDWAIT=0,
+                HAVE_FD_TRANSFER=1,
+                )
+            libraries = []
+
         else:                                   # Linux and other unices
             macros = dict(
                 HAVE_SEM_OPEN=1,
msg73221 - (view) Author: Andrew I MacIntyre (aimacintyre) * Date: 2008-09-14 13:26
2.6rc1 shows the same failure traceback on FreeBSD 6.3, which is covered
by a section of setup.py the same as Damien added for OpenBSD.
msg73440 - (view) Author: Damien Miller (djmdjm) Date: 2008-09-19 20:20
So the bug is actually in the multiprocessing module rather than the
unittest. If HAVE_SEM_OPEN is not defined then SemLock is never
built into _multiprocessing.so, but multiprocessing/syncronize.py
unconditionally depends on its presence.

I guess _multiprocessing could always define a dummy SemLock or
synchronize.py could check before it depends on it.

(it would be great to see this fixed for 2.6)

-d
msg73442 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-09-19 21:12
Bumping up _ I'll need help with a patch
msg73918 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2008-09-27 02:51
Looking at mp.synchronize, the whole module really does depend on a
working _multiprocessing.SemLock instance. If these platforms don't have
a native semaphore implementation, what is their basic inter-process
synchronisation primitive?

Once that is identified, then it should be possible to code either a C
or Python semaphore implementation based on that primitive.
msg73948 - (view) Author: Damien Miller (djmdjm) Date: 2008-09-27 23:58
For 2.6/3.0 it would probably be best to just disable the module
entirely on platforms that lack shareable semaphores (OpenBSD & FreeBSD
at least)
msg73957 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2008-09-28 06:48
Jesse, how much (if any) of the rest of the package will work without
the synchronize module? If it isn't a lot, then it may be a matter of
just making this a cleaner ImportError and an expected test suite skip
on OpenBSD and FreeBSD.

(Unfortunately, our OpenBSD and FreeBSD buildbots are so unreliable that
they don't get much attention when they go red - it looks to me like the
OpenBSD buildbot isn't even managing to build _multiprocessing at the
moment, because HAVE_SEM_OPEN is incorrectly set to 1).
msg73958 - (view) Author: Andrew I MacIntyre (aimacintyre) * Date: 2008-09-28 07:46
I've done some more digging into this for the FreeBSD case.

FreeBSD 6.3 and 7.0 both have sem_open, and the man pages suggest it 
should be fully functional. (see
http://www.freebsd.org/cgi/man.cgi?query=sem_open&apropos=0&sektion=0&manpath=FreeBSD+6.3-RELEASE&format=html)
There is a caveat on the length of the name, which I think could trigger
if the counter variable passed into SEM_CREATE() is >9999.  But as that 
variable seems like it can only ever be 0 (not sure this is intended!)
this shouldn't happen as it stands.

If I change HAVE_SEM_OPEN to 1 in setup.py, the _multiprocessing module
builds, but test_multiprocessing provokes a core dump in both cases.
The backtrace (on 6.3 i386) looks like:
#0  0x2820ef17 in ksem_open () from /lib/libc.so.6
#1  0x2820592c in sem_open () from /lib/libc.so.6
#2  0x284494a0 in semlock_new (type=0x2844b380, args=0x836443c, kwds=0x0)
    at
/home/andymac/build/python-svn/trunk-r66550/Modules/_multiprocessing/semaphore.c:439
#3  0x0809e710 in type_call (type=0x2844b380, args=0x836443c, kwds=0x0)
    at Objects/typeobject.c:731
#4  0x0806042a in PyObject_Call (func=0x2844b380, arg=0x836443c, kw=0x0)
    at Objects/abstract.c:2487
#5  0x080cc367 in PyEval_EvalFrameEx (f=0x84dd00c, throwflag=0)
    at Python/ceval.c:3890
#6  0x080cf9ac in PyEval_EvalCodeEx (co=0x8478800, globals=0x4e,
locals=0xa00,
    args=0x819b02c, argcount=4, kws=0x0, kwcount=0, defs=0x0, defcount=0,
    closure=0x0) at Python/ceval.c:2942
#7  0x08123f84 in function_call (func=0x8486ca4, arg=0x82e107c, kw=0x0)
    at Objects/funcobject.c:524
#8  0x0806042a in PyObject_Call (func=0x8486ca4, arg=0x82e107c, kw=0x0)
    at Objects/abstract.c:2487
#9  0x08069c15 in instancemethod_call (func=0x4e, arg=0x82e107c, kw=0x0)
    at Objects/classobject.c:2579
#10 0x0806042a in PyObject_Call (func=0x84857ac, arg=0x82e107c, kw=0x0)
    at Objects/abstract.c:2487
#11 0x080cc367 in PyEval_EvalFrameEx (f=0x826140c, throwflag=0)
    at Python/ceval.c:3890
---Type <return> to continue, or q <return> to quit---

On 7.0 amd64, the top of the backtrace scrolls off screen so I can't get 
the start of the trace (X not installed...).

To try and remove threads from the equation, due to FreeBSD 6.3 having 
an issue with fork() in a threaded build (see issue3864 for more info), 
I configured without threads (ie ./configure --without-threads) and 
the _multiprocessing module fails to build:

gcc -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3
-Wall-Wstrict-prototypes -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1
-DHAVE_SEM_TIMEDWAIT=0 -IModules/_multiprocessing -I.
-I/home/andymac/build/python-svn/trunk-r66550/./Include -I. -IInclude
-I./Include -I/usr/local/include
-I/home/andymac/build/python-svn/trunk-r66550/Include
-I/home/andymac/build/python-svn/trunk-r66550 -c
/home/andymac/build/python-svn/trunk-r66550/Modules/_multiprocessing/semaphore.c
-o
build/temp.freebsd-6.3-RELEASE-i386-2.6/home/andymac/build/python-svn/trunk-r66550/Modules/_mult
iprocessing/semaphore.o
/home/andymac/build/python-svn/trunk-r66550/Modules/_multiprocessing/semaphore.c:
In function `semlock_acquire':
/home/andymac/build/python-svn/trunk-r66550/Modules/_multiprocessing/semaphore.c:314:
error: `_save' undeclared (first use in this function)
/home/andymac/build/python-svn/trunk-r66550/Modules/_multiprocessing/semaphore.c:314:
error: (Each undeclared identifier is reported only once
/home/andymac/build/python-svn/trunk-r66550/Modules/_multiprocessing/semaphore.c:314:
error: for each function it appears in.)

It appears that some support is there for a single threaded build, but
is incomplete (there's a similar problem in socket_connections.c, but
the module build bails before then).

If its not to be supported on single threaded builds (which would be a
big shame in my opinion!) then the code should make this explicit,
otherwise the single threaded build case needs to be fixed.

I'm still trying to understand the core dump in the multithreaded build
- unfortunately I'm not terribly familiar with gdb or with debugging
from cores (and the actual failure appears to be triggering in the C
library for which I currently don't appear to have symbols...)

Any suggests on how/where to dig further on this?
msg73959 - (view) Author: Andrew I MacIntyre (aimacintyre) * Date: 2008-09-28 07:52
Oops - meant to add that the actual reported cause of the core dump is
"Bad system call".

Also, the OpenBSD man pages make clear that shared semaphores aren't
supported and sem_open() doesn't exist:
http://www.openbsd.org/cgi-bin/man.cgi?query=sem_open&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html
msg73993 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-09-28 21:58
I've been thinking about this - Right now, having a working mp.synchronize 
module, and thread support is key to package currently. For 2.6 - it's 
really too late to try to mock up a working mp.synchronize module, or 
significantly change the package.

My recommendation (I'm going to work on a patch to do this) is to not 
support fbsd/openbsd in this release cycle, which is unfortunate. Any 
other thoughts?
msg74005 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-09-29 04:14
Well, even if 2.6 slipped (which it is looking probably won't happen),
how much time would you need to deal with this? Sounds like you just
won't be able to get to it even with an extra week. So that means
multiprocessing is just not supported on FreeBSD and OpenBSD at this
moment. Sucks, but hopefully it can get fixed in the future. And if
people complain, bug them to get a reliable buildbot going.
msg74015 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2008-09-29 08:16
Agreed - Jesse, can you work up a patch that generates a clean import
error when _multiprocessing.SemLock can't be defined (due to
HAVE_SEM_OPEN=0 or a single-threaded build), adds test_multiprocessing
to the expected skips for FreeBSD and OpenBSD, and updates the
multiprocessing docs to note the restriction to systems with a working
sem_open() implementation?

Improving the *BSD and single-threaded build compatibility of the
multiprocessing package will just have to be high on the to-do list for
2.6.1.

This may also be worth mentioning in the release notes - Barry's call on
that one.
msg74036 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-09-29 18:20
Hows this error look:

>>> import multiprocessing
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jesse/open_source/subversion/python-
trunk/Lib/multiprocessing/__init__.py", line 178, in <module>
    " function, see issue 3770.")
multiprocessing.SemaphoreImportError: This platform lacks a functioning 
sem_open implementation and thereforce, the required synchronization 
primitives needed will not function, see issue 3770.
>>>
msg74037 - (view) Author: Damien Miller (djmdjm) Date: 2008-09-29 18:33
looks good to me
msg74040 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-09-29 19:24
On Mon, Sep 29, 2008 at 11:20 AM, Jesse Noller <report@bugs.python.org> wrote:
>
> Jesse Noller <jnoller@gmail.com> added the comment:
>
> Hows this error look:
>
>>>> import multiprocessing
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
>  File "/Users/jesse/open_source/subversion/python-
> trunk/Lib/multiprocessing/__init__.py", line 178, in <module>
>    " function, see issue 3770.")
> multiprocessing.SemaphoreImportError: This platform lacks a functioning
> sem_open implementation and thereforce, the required synchronization
> primitives needed will not function, see issue 3770.

Is "thereforce" an actual word? Otherwise it looks fine to me.
msg74041 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-09-29 19:29
> Is "thereforce" an actual word? Otherwise it looks fine to me.
>

Yeah, I caught that. Rather than disable the entire package, which
would be frustrating to many - I've changed it to only disable
mp.synchronize for now, patch is pending my final build and test/doc
check locally
msg74043 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-09-29 20:07
Here's a patch, works on my machine. Please review it and make sure it 
satisfies what we've spoken about.
msg74045 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2008-09-29 20:29
Could use confirmation from Damien and Andrew that they now see the
expected skips with the patch applied, but otherwise looks good to me.
msg74054 - (view) Author: Damien Miller (djmdjm) Date: 2008-09-29 22:36
I can confirm that the patch works on OpenBSD -current. Only one nit:

Does this line in Lib/test/test_multiprocessing.py need to be there?

+#import multiprocessing.SemaphoreImportError
msg74055 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-09-29 22:48
On Sep 29, 2008, at 6:36 PM, Damien Miller <report@bugs.python.org>  
wrote:

>
> Damien Miller <djmdjm@users.sourceforge.net> added the comment:
>
> I can confirm that the patch works on OpenBSD -current. Only one nit:
>
> Does this line in Lib/test/test_multiprocessing.py need to be there?
>
> +#import multiprocessing.SemaphoreImportError
>

Oops - my bad. I'll remove it and check it tonight
msg74057 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2008-09-30 00:16
checked in r66688, lowering from rb to crit to address post 2.6 final
msg74077 - (view) Author: Andrew I MacIntyre (aimacintyre) * Date: 2008-09-30 13:43
The checked in change has the planned effect on FreeBSD 6.3 i386. I
can't check on my 7.0 amd64 box as I can't quickly put a current source
tree on it.
msg83495 - (view) Author: (bms) Date: 2009-03-12 15:51
POSIX semaphores should be fixed in 8-CURRENT, pending MFC.

There are rtld + malloc issues in FreeBSD. Python multiprocessing's
use of POSIX threads is not strictly POSIX compliant, as it tries
to do a lot more than just call exec() or async-signal-safe POSIX
APIs after fork()-ing. There is a degree of reluctance in the camp
to fix for these reasons...

In the meantime, you may wish to try building Python 2.6 on FreeBSD
using GNU Pth, here is a patch:

http://people.freebsd.org/~bms/dump/python26-fbsd-pth.patch

thanks!
BMS
msg84610 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-03-30 18:44
Sorry, I hit the wrong bug
msg85155 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-04-02 03:57
Closing; we've removed hard-coded platform variables for a better autoconf 
approach. We currently skip the test suite on platforms which don't 
support or have what we need.
History
Date User Action Args
2009-04-02 03:57:50jnollersetstatus: open -> closed
resolution: fixed
messages: + msg85155
2009-03-30 21:43:03brett.cannonsetnosy: - brett.cannon
2009-03-30 18:44:15jnollersetmessages: + msg84610
2009-03-30 18:43:58jnollersetmessages: - msg84608
2009-03-30 18:42:09jnollersetmessages: + msg84608
2009-03-12 15:51:44bmssetnosy: + bms
messages: + msg83495
2008-09-30 13:43:55aimacintyresetmessages: + msg74077
2008-09-30 00:16:46jnollersetpriority: release blocker -> critical
messages: + msg74057
2008-09-29 22:48:35jnollersetmessages: + msg74055
2008-09-29 22:36:21djmdjmsetmessages: + msg74054
2008-09-29 20:29:31ncoghlansetmessages: + msg74045
2008-09-29 20:07:33jnollersetkeywords: + patch, needs review
files: + issue3770.diff
messages: + msg74043
2008-09-29 19:29:00jnollersetmessages: + msg74041
2008-09-29 19:24:06brett.cannonsetmessages: + msg74040
2008-09-29 18:33:15djmdjmsetmessages: + msg74037
2008-09-29 18:20:39jnollersetmessages: + msg74036
2008-09-29 08:16:02ncoghlansetmessages: + msg74015
2008-09-29 04:14:31brett.cannonsetnosy: + brett.cannon
messages: + msg74005
2008-09-28 21:58:09jnollersetmessages: + msg73993
2008-09-28 07:52:39aimacintyresetmessages: + msg73959
2008-09-28 07:47:01aimacintyresetmessages: + msg73958
2008-09-28 06:48:06ncoghlansetmessages: + msg73957
2008-09-27 23:58:55djmdjmsetmessages: + msg73948
2008-09-27 02:51:29ncoghlansetnosy: + ncoghlan
messages: + msg73918
2008-09-19 21:12:34jnollersetpriority: release blocker
messages: + msg73442
2008-09-19 20:20:09djmdjmsetmessages: + msg73440
2008-09-14 13:26:10aimacintyresetnosy: + aimacintyre
messages: + msg73221
2008-09-04 01:52:41djmdjmsetmessages: + msg72455
2008-09-04 01:30:37jnollersetmessages: + msg72450
2008-09-04 00:58:41benjamin.petersonsetassignee: jnoller
nosy: + jnoller
2008-09-04 00:23:58djmdjmcreate