Operating system: Ubuntu 10.04 Lucid Lynx (Beta)

This worked with Python 3.1 on 9.10 Karmic Koala. The same error appears on two separate installations of Lucid.

Issue 3770 does not seem to be relevant here.

Traceback (most recent call last):
File "/usr/lib/python3.1/multiprocessing/synchronize.py", line 28, in <module>
from _multiprocessing import SemLock
ImportError: cannot import name SemLock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
                                                                        File "main.py", line 16, in <module>
q = JoinableQueue()
File "/usr/lib/python3.1/multiprocessing/__init__.py", line 218, in JoinableQueue
from multiprocessing.queues import JoinableQueue

File "/usr/lib/python3.1/multiprocessing/queues.py", line 22, in <module>
from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition

File "/usr/lib/python3.1/multiprocessing/synchronize.py", line 33, in <module>
" function, see issue 3770.")
                                                                    ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

Why do you think issue 3770 is not relevant?  It looks from the traceback like the built version of multiprocessing decided that sem_open wasn't functional.  So this sounds like a lucid packaging problem to me.

I'm guessing Matthias might have some insight into this on the Ubuntu side. so I'm adding him as nosy.  I'm thinking this isn't a Python problem, but I could be wrong.

Issue 3770 is not relevant because this code works in Python 2.6 on both machines (x86 and x86-64). These machines run plenty of other concurrent processes also, which would imply to me that sem_open() does work.

I could be completely wrong as I'm no expert on the inner workings of either Python or Ubuntu, but that's what I've been able to deduce so far.

Ubuntu bug in package python3.1 is here: https://bugs.launchpad.net/ubuntu/+source/python3.1/+bug/556477

I guess I wasn't clear.  The error you are seeing is exactly the error that issue 3770 talks about.  The *build* process wasn't able to find a functioning sem_open implementation, and therefore you see that error.  Now, clearly your platform does have a functioning sem_open implementation, so something must have gone wrong during the build process.  Since the build works fine on our Ubuntu buildbots (http://www.python.org/dev/buildbot/builders/i386%20Ubuntu%203.1/builds/475), and since you are working from a prepackaged Lucid binary, I conclude that this is most likely an Ubuntu packaging problem.

this was caused by r76567, introduction of the sem_open configure checks. The build machines building the binary packages for lucid use kernels from the last stable LTS release:

./a.out; echo $?
sem_open: Function not implemented
1
$ uname -r
2.6.24-25-server

I can work around this, but others may be affected as well. can't the required kernel version be checked at runtime?

As an aside, I don't really understand why the sem_open check is failing on the build machine.  'man sem_overview' says:

"On a system with Linux 2.6 and a glibc that provides the NPTL threading implementation, a complete implementation of POSIX semaphores is provided."

Is some part of this not true for the Ubuntu build machines?

> can't the required kernel version be checked at runtime?

What exact check would that be, and where would you put it?  It seems to me that the check may be more complicated than just looking for a kernel version (e.g., might depend on libc version, on whether POSIX semaphore support is actually compiled into the kernel or not).  Whatever runtime check is used would also need to work for *BSD and other systems.

We could try just doing a sem_open test call somewhere at run time (e.g. when importing the relevant piece of multiprocessing?);  we'd have to find some way to catch the SIGSYS signal that generates on FreeBSD (and on Linux?) though.

On Ubuntu 10.10 (maverick), python2.6 is functioning correctly, but python2.7 is giving this error again.


$ /usr/bin/python2.7
>>> from multiprocessing.synchronize import Semaphore
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.

I cannot reproduce this on Lucid with:

$ uname -r
2.6.32-28-generic


Isn't this an Ubuntu problem if sem_open only works with some
specific kernels?

> Isn't this an Ubuntu problem if sem_open only works with some specific kernels?

sem_open works fine (python2.6 is using it), but the python2.7 build process didn't detect it properly. This is either a bug with Ubuntu's python2.7 build configuration, or with python2.7's feature detection for sem_open.

I'm not sure which.

This line...

>>> from multiprocessing.synchronize import Semaphore

... works on Ubuntu Lucid with the 2.7, 3.1, 3.2 and default
branches from:

http://hg.python.org/cpython


Since you mention Ubuntu's build configuration, I suggest that you
try to build from the above mercurial repository and see if the problem
persists.

> I suggest that you try to build from the above mercurial repository and see if the problem persists.

How do I know the configuration options that the Ubuntu packager used?

> How do I know the configuration options that the Ubuntu packager used?

the make file (likely) used is at:

http://bazaar.launchpad.net/~doko/python/pkg2.7-debian/view/head:/rules

Note: he is also a python developer, so you can add him to nosy and
ask clarification.

Regards,
-- 
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi

Buck Golemon <report@bugs.python.org> wrote:
> > I suggest that you try to build from the above mercurial repository and see if the problem persists.
> 
> How do I know the configuration options that the Ubuntu packager used?

I'd use the default configuration and check if sem_open is detected
correctly by ./configure and if the import works.

As Matthias Klose stated earlier, the issue seems to be the kernel version.

Like Stefan says, use the default build options with the checkout.  If it works, then the problem is an Ubuntu bug, and not a Python bug.

python2.7.1+ from mercurial supports sem_open (and multiprocessing) just fine.

doko: Could you help us figure out why the ubuntu 10.10 python2.7 build has this issue? I believe this issue should be assigned to you?

Relevant lines from the config.log:


configure:9566: checking for sem_open
configure:9566: gcc -pthread -o conftest -g -O2   conftest.c -lpthread -ldl  >&5
configure:9566: $? = 0
configure:9566: result: yes

Is this still a problem for people?  Ubuntu 11.04's python2.7 has been fixed:

@neurotica[~:1000]% type python2.7
python2.7 is /usr/bin/python2.7
@neurotica[~:1001]% python2.7 -c 'from _multiprocessing import SemLock'
@neurotica[~:1002]% 

The Launchpad bug for the issue is here:

https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/672209

@Barry: Yes, it's still a problem.

The ubuntu 10.10 python2.7 still has no multiprocessing.
Since the EOL is April 2012, it needs fixed.

It may be considered an invalid python bug, since it seems to be strictly related to Ubuntu packaging, but I thought the python maintainers should know.

--Buck

I'm closing this as invalid for Python, since I believe this is strictly an Ubuntu bug caused by an out-of-date kernel on the build farm.  I'm working on an SRU to fix that.  Please track further status on the Launchpad bug page given below.

> Barry A. Warsaw<barry@python.org>  added the comment:
> 
> I'm closing this as invalid for Python, since I believe this is strictly an
> Ubuntu bug caused by an out-of-date kernel on the build farm.

that's where I disagree. a configure check should not be dependent on the running kernel.  I assume in the majority of cases you won't build against a current kernel, so the a fix in python maybe could be a runtime check.  Such configure checks will fail for cross builds too.

Note that packages on Ubuntu are always built on the current kernel of the current LTS release, on Debian on current kernels of the current stable release.  So there is nothing "out-of-date".

On Apr 26, 2011, at 09:45 PM, Matthias Klose wrote:

>Matthias Klose <doko@debian.org> added the comment:
>
>> Barry A. Warsaw<barry@python.org>  added the comment:
>> 
>> I'm closing this as invalid for Python, since I believe this is strictly an
>> Ubuntu bug caused by an out-of-date kernel on the build farm.
>
>that's where I disagree. a configure check should not be dependent on the
>running kernel.  I assume in the majority of cases you won't build against a
>current kernel, so the a fix in python maybe could be a runtime check.  Such
>configure checks will fail for cross builds too.

Perhaps so, but that's a totally different issue.  Such a change wouldn't be
appropriate for stable releases, so it could only make it into Python 3.3.  It
might be an interesting thing to work on, but I'd suggest opening a new bug
and seeing if anyone wants to work up a patch for that.

>Note that packages on Ubuntu are always built on the current kernel of the
>current LTS release, on Debian on current kernels of the current stable
>release.  So there is nothing "out-of-date".

"Out-of-date" was probably an incorrect choice of words.  My understanding was
that this bug was caused by a problem in the kernel that caused the configure
check to fail, at the time the Maverick Python 2.7 package was built, and that
the build machines have since been updated, correcting the problem.  At least,
current PPA builds don't have this problem.

In any case, I still think this specific issue is more appropriately tracked
in the Launchpad bug, and I have requested an SRU for a rebuild.