classification
Title: semaphore errors on AIX 5.2
Type: Stage:
Components: Versions: Python 3.0, Python 2.6, Python 2.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: christian.heimes, lemburg, sable
Priority: normal Keywords: patch

Created on 2007-10-04 09:18 by sable, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Files
File name Uploaded Description Edit
patch_aix_broken_semaphores.diff sable, 2007-10-04 09:30
patch_aix_broken_semaphores.diff sable, 2007-10-04 09:43
Pull Requests
URL Status Linked Edit
PR 831 MSeifert, 2017-03-27 19:38
PR 552 closed dstufft, 2017-03-31 16:36
Messages (10)
msg56226 - (view) Author: Sébastien Sablé (sable) Date: 2007-10-04 09:18
There used to be a bug open concerning this problem in the old bug
tracker but I can't find it in the new bug tracker:

http://mail.python.org/pipermail/python-bugs-list/2005-June/029290.html
msg56227 - (view) Author: Sébastien Sablé (sable) Date: 2007-10-04 09:18
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: The Written Word (Albert Chin) (tww-china)
Assigned to: Nobody/Anonymous (nobody)
Summary: semaphore errors from Python 2.3.x on AIX 5.2

Initial Comment:
Newer versions of the AIX 5.2 libc implement POSIX
semaphores and thus define _POSIX_SEMAPHORES in
<unistd.h>. However, when building Python 2.3.x (and
I'm sure any other version with sem_init()), a few
semaphore errors are encountered:
  building 'pcre' extension
  ./Modules/ld_so_aix gcc -bI:Modules/python.exp  
-L/opt/TWWfsw/python232/lib/support -Wl,-brtl  
-Wl,-blibpath:/opt/TWWfsw/python232/lib/support:/usr/lib  
build/temp.aix-5.2-2.3/pcremodule.o   build/temp.aix-5.2-2.3/pypcre.o
-L/opt/TWWfsw/python232/lib   -o build/lib.aix-5.2-2.3/pcre.so
  sem_trywait: Permission denied
  sem_wait: Permission denied
  sem_post: Permission denied

I ran the Python process under truss and saw that the
sem_init() from Python/thread_pthread.h worked
successfully. fork() was then called and a
sem_trywait() operation (probably from
PyThread_acquire_lock()) was called. From the
sem_init() man page
(http://publib.boulder.ibm.com/infocenter/pseries/index.jsp?topic=/com.ibm.aix.doc/libs/basetrf2/sem_init.htm):
  If the pshared parameter has a nonzero value, the
semaphore
  is shared between processes. In this case, any
process that can
  access the sem parameter can use it for performing
sem_wait,
  sem_trywait, sem_post, and sem_destroy operations.

  Only the sem parameter itself may be used for performing
  synchronization.

  If the pshared parameter is zero, the semaphore is
shared between
  threads of the process. Any thread in this process
can use the sem
  parameter for performing sem_wait, sem_trywait,
sem_post, and
  sem_destroy operations. The use of the semaphore by
threads
  other than those created in the same process returns
an error.

Setting the 2nd value of sem_init to 1 (was 0) solved
the problem. However, I don't know if this is wise for
security. Another solution is to set
HAVE_BROKEN_POSIX_SEMAPHORES for "AIX/5" like for
"SunOS/5.8".

----------------------------------------------------------------------

Comment By: Leanid (lim_1)
Date: 2005-06-20 13:12

Message:
Logged In: YES 
user_id=701470

I can confirm this problem. I am using scons to build my 
project on AIX. I could not run parallel builds (-j4) on AIX. 
Randomly, but very often xlC invocation would stuck and I 
have 4 xlCcode hanging and taking all CPUs.


After adding HAVE_BROKEN_POSIX_SEMAPHORES for 
AIX and rebuilding Python parallel builds on AIX are fine. 
Trying first fix with changing 0 to 1 in sem_init also resolved 
this problem, but I started to get some other unexplained 
error from scons, so had to deactivate semaphores all 
together.

Albert, thanks for this report, it helped a lot.

----------------------------------------------------------------------
msg56228 - (view) Author: Sébastien Sablé (sable) Date: 2007-10-04 09:30
I can confirm that this bug is still present in Python 2.4.1 (and
probably later versions) on AIX 5.2 and 5.3.

When HAVE_BROKEN_POSIX_SEMAPHORES is not defined at compile time, Python
will spend a lot of time in sem_wait, usually when acessing files.
The problem does not always happen as it seems to depend on the type of
file system in use.

In my programm running for 3m30s, Python would call about 400000 time
sem_wait and spend about 2mins waiting. When Python is compiled with
HAVE_BROKEN_POSIX_SEMAPHORES, the same programm will only take 1m30s.

I attach a patch which solves the problem.
msg57194 - (view) Author: Sébastien Sablé (sable) Date: 2007-11-07 08:53
The bug is still present in Python 2.5.1. The same patch applies.
The patch is rather trivival, could someone please integrate it in trunk?
Thanks in advance
msg57333 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2007-11-09 21:20
I'm setting the priority to high because it sounds important for AIX
users and the patch is *really* simple, just two additional lines for
configure.in
msg57512 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2007-11-14 23:38
The problem is also present in Python 2.4 and 2.3. Confirmed on AIX 5.3.
msg57715 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-11-20 22:35
I have no way to test this.
msg59199 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-01-04 01:10
OK, accepted for 2.5 and 2.6.

Can someone with a stake and submit access (e.g. lemburg) check it in?
msg61878 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2008-01-30 21:09
Christian, could you apply this patch ? It needs a working autoconf 2.61
installation which I currently don't have.

The patch itself, does the trick, so should go in.

Thanks.
msg61879 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-01-30 22:54
Fixed in r60464 (trunk) and r60465 (25)
History
Date User Action Args
2017-03-31 16:36:19dstufftsetpull_requests: + pull_request930
2017-03-27 19:49:38gvanrossumsetnosy: - gvanrossum
2017-03-27 19:38:27MSeifertsetpull_requests: + pull_request753
2008-01-30 22:55:02christian.heimessetstatus: open -> closed
resolution: accepted -> fixed
messages: + msg61879
2008-01-30 21:09:15lemburgsetassignee: lemburg -> christian.heimes
messages: + msg61878
2008-01-04 01:10:56gvanrossumsetassignee: lemburg
messages: + msg59199
resolution: accepted
2008-01-04 00:44:12christian.heimessetversions: - Python 2.4, Python 2.3
2007-11-20 22:35:00gvanrossumsetpriority: high -> normal
assignee: gvanrossum -> (no value)
messages: + msg57715
2007-11-20 01:16:13christian.heimessetassignee: gvanrossum
nosy: + gvanrossum
2007-11-14 23:38:29lemburgsetnosy: + lemburg
messages: + msg57512
versions: + Python 2.4, Python 2.3
2007-11-09 21:20:05christian.heimessetpriority: high
keywords: + patch
messages: + msg57333
nosy: + christian.heimes
versions: + Python 2.6, Python 2.5, Python 3.0
2007-11-07 08:53:52sablesetmessages: + msg57194
2007-10-04 09:43:07sablesetfiles: + patch_aix_broken_semaphores.diff
2007-10-04 09:30:54sablesetfiles: + patch_aix_broken_semaphores.diff
messages: + msg56228
2007-10-04 09:18:56sablesetmessages: + msg56227
2007-10-04 09:18:14sablecreate