classification
Title: Cannot import name SemLock on Ubuntu
Type: behavior Stage:
Components: Extension Modules Versions: Python 3.1
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: barry Nosy List: barry, bukzor, coconutrb, doko, jnoller, mark.dickinson, r.david.murray, sandro.tosi, skrah
Priority: normal Keywords:

Created on 2010-04-06 10:29 by coconutrb, last changed 2011-04-26 22:26 by barry. This issue is now closed.

Messages (21)
msg102447 - (view) Author: David Coconut (coconutrb) Date: 2010-04-06 10:29
Operating system: Ubuntu 10.04 Lucid Lynx (Beta)

This worked with Python 3.1 on 9.10 Karmic Koala. The same error appears on two separate installations of Lucid.

Issue 3770 does not seem to be relevant here.

Traceback (most recent call last):
File "/usr/lib/python3.1/multiprocessing/synchronize.py", line 28, in <module>
from _multiprocessing import SemLock
ImportError: cannot import name SemLock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
                                                                        File "main.py", line 16, in <module>
q = JoinableQueue()
File "/usr/lib/python3.1/multiprocessing/__init__.py", line 218, in JoinableQueue
from multiprocessing.queues import JoinableQueue

File "/usr/lib/python3.1/multiprocessing/queues.py", line 22, in <module>
from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition

File "/usr/lib/python3.1/multiprocessing/synchronize.py", line 33, in <module>
" function, see issue 3770.")
                                                                    ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
msg102458 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-04-06 12:00
Why do you think issue 3770 is not relevant?  It looks from the traceback like the built version of multiprocessing decided that sem_open wasn't functional.  So this sounds like a lucid packaging problem to me.
msg102460 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-04-06 12:23
I'm guessing Matthias might have some insight into this on the Ubuntu side. so I'm adding him as nosy.  I'm thinking this isn't a Python problem, but I could be wrong.
msg102953 - (view) Author: David Coconut (coconutrb) Date: 2010-04-12 15:30
Issue 3770 is not relevant because this code works in Python 2.6 on both machines (x86 and x86-64). These machines run plenty of other concurrent processes also, which would imply to me that sem_open() does work.

I could be completely wrong as I'm no expert on the inner workings of either Python or Ubuntu, but that's what I've been able to deduce so far.

Ubuntu bug in package python3.1 is here: https://bugs.launchpad.net/ubuntu/+source/python3.1/+bug/556477
msg102957 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-04-12 16:54
I guess I wasn't clear.  The error you are seeing is exactly the error that issue 3770 talks about.  The *build* process wasn't able to find a functioning sem_open implementation, and therefore you see that error.  Now, clearly your platform does have a functioning sem_open implementation, so something must have gone wrong during the build process.  Since the build works fine on our Ubuntu buildbots (http://www.python.org/dev/buildbot/builders/i386%20Ubuntu%203.1/builds/475), and since you are working from a prepackaged Lucid binary, I conclude that this is most likely an Ubuntu packaging problem.
msg103156 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2010-04-14 22:21
this was caused by r76567, introduction of the sem_open configure checks. The build machines building the binary packages for lucid use kernels from the last stable LTS release:

./a.out; echo $?
sem_open: Function not implemented
1
$ uname -r
2.6.24-25-server

I can work around this, but others may be affected as well. can't the required kernel version be checked at runtime?
msg103188 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-04-15 09:01
As an aside, I don't really understand why the sem_open check is failing on the build machine.  'man sem_overview' says:

"On a system with Linux 2.6 and a glibc that provides the NPTL threading implementation, a complete implementation of POSIX semaphores is provided."

Is some part of this not true for the Ubuntu build machines?

> can't the required kernel version be checked at runtime?

What exact check would that be, and where would you put it?  It seems to me that the check may be more complicated than just looking for a kernel version (e.g., might depend on libc version, on whether POSIX semaphore support is actually compiled into the kernel or not).  Whatever runtime check is used would also need to work for *BSD and other systems.

We could try just doing a sem_open test call somewhere at run time (e.g. when importing the relevant piece of multiprocessing?);  we'd have to find some way to catch the SIGSYS signal that generates on FreeBSD (and on Linux?) though.
msg133711 - (view) Author: Buck Evan (bukzor) * Date: 2011-04-14 05:20
On Ubuntu 10.10 (maverick), python2.6 is functioning correctly, but python2.7 is giving this error again.


$ /usr/bin/python2.7
>>> from multiprocessing.synchronize import Semaphore
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
msg133716 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-04-14 06:22
I cannot reproduce this on Lucid with:

$ uname -r
2.6.32-28-generic


Isn't this an Ubuntu problem if sem_open only works with some
specific kernels?
msg133753 - (view) Author: Buck Evan (bukzor) * Date: 2011-04-14 17:29
> Isn't this an Ubuntu problem if sem_open only works with some specific kernels?

sem_open works fine (python2.6 is using it), but the python2.7 build process didn't detect it properly. This is either a bug with Ubuntu's python2.7 build configuration, or with python2.7's feature detection for sem_open.

I'm not sure which.
msg133755 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-04-14 17:47
This line...

>>> from multiprocessing.synchronize import Semaphore

... works on Ubuntu Lucid with the 2.7, 3.1, 3.2 and default
branches from:

http://hg.python.org/cpython


Since you mention Ubuntu's build configuration, I suggest that you
try to build from the above mercurial repository and see if the problem
persists.
msg133756 - (view) Author: Buck Evan (bukzor) * Date: 2011-04-14 17:56
> I suggest that you try to build from the above mercurial repository and see if the problem persists.

How do I know the configuration options that the Ubuntu packager used?
msg133759 - (view) Author: Sandro Tosi (sandro.tosi) * (Python committer) Date: 2011-04-14 18:21
> How do I know the configuration options that the Ubuntu packager used?

the make file (likely) used is at:

http://bazaar.launchpad.net/~doko/python/pkg2.7-debian/view/head:/rules

Note: he is also a python developer, so you can add him to nosy and
ask clarification.

Regards,
-- 
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi
msg133760 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-04-14 18:45
Buck Golemon <report@bugs.python.org> wrote:
> > I suggest that you try to build from the above mercurial repository and see if the problem persists.
> 
> How do I know the configuration options that the Ubuntu packager used?

I'd use the default configuration and check if sem_open is detected
correctly by ./configure and if the import works.

As Matthias Klose stated earlier, the issue seems to be the kernel version.
msg133761 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-04-14 18:48
Like Stefan says, use the default build options with the checkout.  If it works, then the problem is an Ubuntu bug, and not a Python bug.
msg134370 - (view) Author: Buck Evan (bukzor) * Date: 2011-04-25 05:03
python2.7.1+ from mercurial supports sem_open (and multiprocessing) just fine.

doko: Could you help us figure out why the ubuntu 10.10 python2.7 build has this issue? I believe this issue should be assigned to you?

Relevant lines from the config.log:


configure:9566: checking for sem_open
configure:9566: gcc -pthread -o conftest -g -O2   conftest.c -lpthread -ldl  >&5
configure:9566: $? = 0
configure:9566: result: yes
msg134385 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-04-25 14:46
Is this still a problem for people?  Ubuntu 11.04's python2.7 has been fixed:

@neurotica[~:1000]% type python2.7
python2.7 is /usr/bin/python2.7
@neurotica[~:1001]% python2.7 -c 'from _multiprocessing import SemLock'
@neurotica[~:1002]% 

The Launchpad bug for the issue is here:

https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/672209
msg134472 - (view) Author: Buck Evan (bukzor) * Date: 2011-04-26 15:35
@Barry: Yes, it's still a problem.

The ubuntu 10.10 python2.7 still has no multiprocessing.
Since the EOL is April 2012, it needs fixed.

It may be considered an invalid python bug, since it seems to be strictly related to Ubuntu packaging, but I thought the python maintainers should know.

--Buck
msg134501 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-04-26 21:11
I'm closing this as invalid for Python, since I believe this is strictly an Ubuntu bug caused by an out-of-date kernel on the build farm.  I'm working on an SRU to fix that.  Please track further status on the Launchpad bug page given below.
msg134503 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2011-04-26 21:45
> Barry A. Warsaw<barry@python.org>  added the comment:
> 
> I'm closing this as invalid for Python, since I believe this is strictly an
> Ubuntu bug caused by an out-of-date kernel on the build farm.

that's where I disagree. a configure check should not be dependent on the running kernel.  I assume in the majority of cases you won't build against a current kernel, so the a fix in python maybe could be a runtime check.  Such configure checks will fail for cross builds too.

Note that packages on Ubuntu are always built on the current kernel of the current LTS release, on Debian on current kernels of the current stable release.  So there is nothing "out-of-date".
msg134511 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-04-26 22:26
On Apr 26, 2011, at 09:45 PM, Matthias Klose wrote:

>Matthias Klose <doko@debian.org> added the comment:
>
>> Barry A. Warsaw<barry@python.org>  added the comment:
>> 
>> I'm closing this as invalid for Python, since I believe this is strictly an
>> Ubuntu bug caused by an out-of-date kernel on the build farm.
>
>that's where I disagree. a configure check should not be dependent on the
>running kernel.  I assume in the majority of cases you won't build against a
>current kernel, so the a fix in python maybe could be a runtime check.  Such
>configure checks will fail for cross builds too.

Perhaps so, but that's a totally different issue.  Such a change wouldn't be
appropriate for stable releases, so it could only make it into Python 3.3.  It
might be an interesting thing to work on, but I'd suggest opening a new bug
and seeing if anyone wants to work up a patch for that.

>Note that packages on Ubuntu are always built on the current kernel of the
>current LTS release, on Debian on current kernels of the current stable
>release.  So there is nothing "out-of-date".

"Out-of-date" was probably an incorrect choice of words.  My understanding was
that this bug was caused by a problem in the kernel that caused the configure
check to fail, at the time the Maverick Python 2.7 package was built, and that
the build machines have since been updated, correcting the problem.  At least,
current PPA builds don't have this problem.

In any case, I still think this specific issue is more appropriately tracked
in the Launchpad bug, and I have requested an SRU for a rebuild.
History
Date User Action Args
2011-04-26 22:26:41barrysetmessages: + msg134511
2011-04-26 21:45:52dokosetmessages: + msg134503
2011-04-26 21:11:10barrysetstatus: open -> closed
assignee: barry
resolution: not a bug
messages: + msg134501
2011-04-26 15:35:56bukzorsetmessages: + msg134472
2011-04-25 14:46:36barrysetmessages: + msg134385
2011-04-25 05:03:48bukzorsetmessages: + msg134370
2011-04-14 18:48:56r.david.murraysetmessages: + msg133761
2011-04-14 18:45:57skrahsetmessages: + msg133760
2011-04-14 18:21:08sandro.tosisetnosy: + sandro.tosi
messages: + msg133759
2011-04-14 17:56:13bukzorsetmessages: + msg133756
2011-04-14 17:47:15skrahsetmessages: + msg133755
2011-04-14 17:30:36bukzorsettitle: Cannot import name SemLock on Ubuntu lucid -> Cannot import name SemLock on Ubuntu
2011-04-14 17:29:39bukzorsetmessages: + msg133753
2011-04-14 11:24:49pitrousetnosy: + barry
2011-04-14 06:22:00skrahsetnosy: + skrah
messages: + msg133716
2011-04-14 05:20:45bukzorsetnosy: + bukzor
messages: + msg133711
2010-04-15 09:01:36mark.dickinsonsetmessages: + msg103188
2010-04-15 00:36:05r.david.murraysetnosy: + jnoller
2010-04-14 22:21:02dokosetnosy: + mark.dickinson
messages: + msg103156
2010-04-12 16:54:36r.david.murraysetmessages: + msg102957
2010-04-12 15:30:42coconutrbsetmessages: + msg102953
2010-04-06 12:23:49r.david.murraysetnosy: + doko

messages: + msg102460
title: Cannot import name SemLock -> Cannot import name SemLock on Ubuntu lucid
2010-04-06 12:00:13r.david.murraysetpriority: normal

nosy: + r.david.murray
messages: + msg102458

components: + Extension Modules
type: crash -> behavior
2010-04-06 10:29:50coconutrbcreate