classification
Title: fcntl extension fails to build on AIX 6.1
Type: compile error Stage: resolved
Components: Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: darkside, drj, haubi, inkblotter, pitrou, sable
Priority: normal Keywords: patch

Created on 2008-10-03 08:32 by drj, last changed 2010-09-10 18:48 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
py26aix61script2 drj, 2008-10-03 08:32
Python-2.6.6_flock_AIX.diff sable, 2010-09-02 14:18
Python-2.7_flock_AIX.diff sable, 2010-09-06 14:17
flock-check-py3k.patch haubi, 2010-09-07 20:15
patch_Michael_Haubenwallner_flock_msg115868.diff sable, 2010-09-10 13:05
Messages (22)
msg74231 - (view) Author: David Jones (drj) * Date: 2008-10-03 08:32
After hacking the configure script to work around the issues http://bugs.python.org/issue4025 and http://bugs.python.org/issue1633863 
the build still fails:

building 'fcntl' extension
xlc_r -DNDEBUG -O -I. -I/home/u0006584/Python-2.6/./Include -I. -
IInclude -I./Include -I/home/u0006584/Python-2.6/Include -
I/home/u0006584/Python-2.6 -c /home/u0006584/Python-
Modules/fcntlmodule.c -o build/temp.aix-6.1-
2.6/home/u0006584/Python-2.6/Modules/fcntlmodule.o
./Modules/ld_so_aix xlc_r -bI:Modules/python.exp build/temp.aix-6.1-
2.6/home/u0006584/Python-2.6/Modules/fcntlmodule.o -o build/lib.aix-6.1-
2.6/fcntl.so
ld: 0711-317 ERROR: Undefined symbol: .flock
ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more 
information.
*** WARNING: renaming "fcntl" since importing it failed:        0509-022 
Cannot load module build/lib.aix-6.1-2.6.
        0509-026 System error: A file or directory in the path name does 
not exist.
error: No such file or directory
make: 1254-004 The error code from the last command is 1.

(Full transcript attached)

I'm mystified and I can't log in to the AIX box right now.

This does not appear to be the same as http://bugs.python.org/issue1756343 and http://bugs.python.org/issue1694442 which both appear to be the same, 
but documenting a differen AIX build problem.
msg75260 - (view) Author: inkblotter (inkblotter) Date: 2008-10-27 19:35
This occurs because it must link with -lbsd on AIX 6.1.  If you hand
link the failing module by adding -lbsd it builds quietly.

If you build the prerequisites for the _tkinter module on AIX 6.1, you
will also encounter an error becuase this module is not linked with the
-lXext library.  Again, you can hand link it with that library.  

Notice that the modules ignore these two libraries when they are passed
to the python build.  The modules should build against the libraries
that are mentioned in the --with-libs=LIBS argument to python configure.

I will eventually figure out how to modify setup.py so that these two
failures no longer occur on AIX 6.1.
msg79289 - (view) Author: Jeremy Olexa (darkside) Date: 2009-01-06 21:41
This also happens with Python 2.5.2 (not the latest 2.5 series, I know)
on AIX 6.1.
msg79340 - (view) Author: Jeremy Olexa (darkside) Date: 2009-01-07 15:31
I have now confirmed that the fix described here[1] works as desired. I
don't know if this is proper or not but it matches what inkblotter said.

[1]: http://www.ibm.com/developerworks/forums/thread.jspa?threadID=226339
msg102526 - (view) Author: Michael Haubenwallner (haubi) * Date: 2010-04-07 09:40
This very same problem happens (with Python-2.6.2) on AIX5.3 now too, after upgrading to:
$ oslevel -s
5300-08-09-1013

Unlike before (comparing with old build logs), this AIX5.3 now provides flock() in <sys/file.h> and libbsd.a[shr.o] like AIX6.1.

Interesting enough, /usr/lib/libbsd.a contains 32bit shared objects only, so -lbsd does not help in 64bit mode (don't know if python actually supports 64bit on AIX). I don't have an AIX6.1 to check this.

Because of this, upgrading "checking for flock" from compile- to link-check (eventually trying -lbsd a second time) might help?
msg102529 - (view) Author: Michael Haubenwallner (haubi) * Date: 2010-04-07 10:03
Ohw, looking on another machine, being AIX5.3 TL6:
$ oslevel -s
5300-06-00-0000

Here flock() is provided in libbsd.a(shr.o) (32bit only) too, but it isn't declared in any header-file. So that recent AIX5.3 patchset just adds the flock() declaration, not the function itself (is already there).
msg115377 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-02 14:18
Hi,

Here is a patch that solves this problem. It was tested with Python 2.6.6 on AIX 6.1.

The same problem applies to Python 2.7 and 3.x, but since the syntax has been changed in configure.in for Python 2.7 and 3.x, I need to adapt a little bit my patch (coming soon).

regards
msg115706 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-06 14:17
Here is the version of the patch for Python 2.7.
I had to change AC_TRY_COMPILE to AC_COMPILE_IFELSE and AC_TRY_LINK to AC_LINK_IFELSE.

The syntax in configure.in is the same between Python 2.7 and the trunk. It is also the same between Python 2.6.6 and Python 3.1.2.

So this patch can be applied on the trunk and the patch Python-2.6.6_flock_AIX.diff can be applied on Python 3.1.2.
msg115711 - (view) Author: Michael Haubenwallner (haubi) * Date: 2010-09-06 15:12
While I've not tested these patches myself, I do think there still is an unhandled case: building a 64bit python.
As libbsd.a contains 32bit objects only, there is no 64bit flock().
So AC_CHECK_LIB(bsd,flock) may still fail, which should result in have_flock=no then.
Thank you!
msg115732 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-06 19:59
[rant: grrr, AIX is really a crappy platform; no consistency between releases; there are dozens of obvious bugs like that... anyway...]

The fact that there is no 64 bits objects in libbsd.a on AIX 6.1 is a documented bug:
http://www-01.ibm.com/support/docview.wss?uid=isg1IZ45155

They recommend to install patch APAR IZ45155.

I suppose there is nothing more we can do if the function is not available on the system. Maybe we could display a warning or error message and suggest to upgrade to a more recent patchlevel if we are on AIX and flock cannot be found?

Though if the installation is relatively up to date, there should be no problem.
msg115733 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-06 20:11
Well, I suppose if flock needs libbsd on AIX (which is quite weird in itself), and AIX fails to ship libbsd for some executable formats, then it's AIX's problem.
msg115744 - (view) Author: Michael Haubenwallner (haubi) * Date: 2010-09-07 06:43
While I do agree this being an AIX bug, it is not a blocker here:
fcntl extension does not _depend_ on flock, it just does _prefer_ flock:
If not available, fcntl extension uses something else (fcntl IIRC), as it did (even without linking libbsd) before the AIX5.3-patch, which just added flock to the headers (the implementation in libbsd already has been there before the patch).
The problem after that AIX5.3-patch was that the compile-check found flock, and then fcntl extension failed to link due to missing libbsd.
So it's just for how to detect flock, where the most safe variant is both the compile- and link-check. When one of them fails, flock should not be flagged as available and the alternative gets used.
msg115768 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-07 14:51
Here is a new test for flock. HAVE_FLOCK is defined if we can link a C application calling flock, or if flock is defined in libbsd.
FLOCK_NEEDS_LIBBSD is also defined in the second case.

AC_MSG_CHECKING(for flock)
have_flock=no
AC_TRY_LINK([
#include "confdefs.h" 
#include <sys/file.h>
], [void* p = flock; flock(0, 0)],
  [AC_CHECK_LIB(bsd,flock, [
    AC_DEFINE(FLOCK_NEEDS_LIBBSD, 1, Define if flock needs to be linked with bsd library.)
    have_flock=yes
    ])],
  [have_flock=yes]
)
if test "$have_flock" = yes ; then
  AC_DEFINE(HAVE_FLOCK, 1, Define if you have the 'flock' function.)
fi
AC_MSG_RESULT($have_flock)

I think that this new test would also cover your case with a broken AIX libbsd? [I haven't actually tried to compile it yet, if the behavior is OK I will test it in both autoconf formats and provide some new patches tomorrow]
msg115770 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-07 15:02
I've just merged Sébastien's patches in r84584 (3.x), r84585 (2.7) and r84586 (3.1). If you'd like to improve the fixes (as per your latest message), please say so, otherwise I'll close the issue.

(and thanks for your contributions!)
msg115789 - (view) Author: Michael Haubenwallner (haubi) * Date: 2010-09-07 17:39
Sorry to be pedantic - but it looks like (don't have a build environment at hand at the moment) it will fail to build as 64bit on the not-so-up-to-date AIX with missing 64bit flock implementation:
There, the compile-check will succeed and define HAVE_FLOCK, but the link-check will fail - which really should reset have_flock to 'no'.
IMO, HAVE_FLOCK should be defined when _both_ checks succeed only, not upon the compile-check only as it is done now.
Thank you!
msg115796 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-07 19:07
> Sorry to be pedantic - but it looks like (don't have a build
> environment at hand at the moment) it will fail to build as 64bit on
> the not-so-up-to-date AIX with missing 64bit flock implementation:
> There, the compile-check will succeed and define HAVE_FLOCK, but the
> link-check will fail - which really should reset have_flock to 'no'.
> IMO, HAVE_FLOCK should be defined when _both_ checks succeed only, not
> upon the compile-check only as it is done now.

It might be better indeed. Patches against the current SVN branches are
welcome (I don't have an AIX system to test them on, and besides I'm not
an autoconf expert).
msg115801 - (view) Author: Michael Haubenwallner (haubi) * Date: 2010-09-07 20:15
Using this patch based on current py3k branch I've been able now to build the fcntl module for 32bit aix5.3 (with flock) and 64bit aix5.3 (without flock), both using an old gcc-3.3.6, as well as for 64bit linux.
If necessary, I should be able to create a similar patch for the other branches tomorrow.
msg115805 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-07 20:49
I inverted the actions in the test I proposed in msg115768 (quick note from home, should have tested at work before posting). It should be:

AC_MSG_CHECKING(for flock)
have_flock=no
AC_TRY_LINK([
#include "confdefs.h" 
#include <sys/file.h>
], [void* p = flock; flock(0, 0)],
  [have_flock=yes],
  [AC_CHECK_LIB(bsd,flock, [
    AC_DEFINE(FLOCK_NEEDS_LIBBSD, 1, Define if flock needs to be linked with bsd library.)
    have_flock=yes
    ])]
)
if test "$have_flock" = yes ; then
  AC_DEFINE(HAVE_FLOCK, 1, Define if you have the 'flock' function.)
fi
AC_MSG_RESULT($have_flock)

Mickael, does that new test works for you? I am not sure I follow your explanation: there is no more 'compile-check' (AC_TRY_COMPILE) in the new test I provided; just a 'link-check' (AC_TRY_LINK - which tests that compilation and link did work).
HAVE_FLOCK is undefined by default.
If the link-check succeeds, then HAVE_FLOCK is defined.
If it fails, then we try to look for flock in libbsd; if that test works, then HAVE_FLOCK is defined as well as FLOCK_NEEDS_LIBBSD.

In the case of broken AIX 64 bits, the link-check will fail (no flock in default C libraries); then the test to find flock in libbsd will also fail (no 64 bits objets in libbsd). So HAVE_FLOCK will be undefined, which is what you expect.

m4/autoconf is quite unintuitive so I may have made a mistake, or it may be easily misunderstood. But I intend my test to define HAVE_FLOCK only if compilation _and_ linking works, like you expect.
msg115868 - (view) Author: Michael Haubenwallner (haubi) * Date: 2010-09-08 13:29
Sébastien, while this gives expected results on the AIX box here, it still has one subtle problem:
AC_TRY_LINK may fail due to missing declaration in <sys/file.h> _or_ due to missing implementation in libc.
The subsequent AC_CHECK_LIB won't fail when the implementation is in libbsd but the declaration is missing - this is the case on not-so-recent AIX5.3 (and the reason python worked there before).

Another minor nit is configure's output when libbsd is needed:
> checking for flock... checking for flock in -lbsd... yes
> yes

Anyway - I've hacked up another variant (with recent autoconf manual at hand), which really should cover all known situations now (additionally using cache-checks):

AC_CACHE_CHECK([for flock declaration], [ac_cv_flock_decl],
  [AC_COMPILE_IFELSE(
    [AC_LANG_PROGRAM(
      [#include <sys/file.h>],
      [void* p = flock]
    )],
    [ac_cv_flock_decl=yes],
    [ac_cv_flock_decl=no]
  )
])
if test "x${ac_cv_flock_decl}" = xyes; then
  AC_CHECK_FUNCS(flock,,
    AC_CHECK_LIB(bsd,flock,
      [AC_DEFINE(HAVE_FLOCK)
       AC_DEFINE(FLOCK_NEEDS_LIBBSD, 1, Define if flock needs to be linked with bsd library.)
    ])
  )
fi

Thank you anyway!
msg115886 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-08 16:53
Thanks Michael, your new test looks fine to me.

I wanted to test it on my AIX system with branch py3k but I had some other issues (issue 9799), but I will test it tomorrow.
msg116016 - (view) Author: Sébastien Sablé (sable) Date: 2010-09-10 13:05
I tested Michael test in msg115868 on my AIX system (AIX 6.1 with flock correctly present in libbsd) with py3k and it works fine:

checking for flock declaration... yes
checking for flock... no
checking for flock in -lbsd... yes

[103/344] test_fcntl
= OK

I attach it as a patch against py3k.
msg116035 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-10 18:48
Ok, thank you. The improvement has been committed in all three branches (3.2, 3.1, 2.7).
History
Date User Action Args
2010-09-10 18:48:12pitrousetstatus: open -> closed

stage: resolved
messages: + msg116035
versions: - Python 2.6, Python 3.3
2010-09-10 13:05:54sablesetfiles: + patch_Michael_Haubenwallner_flock_msg115868.diff

messages: + msg116016
2010-09-08 16:53:33sablesetmessages: + msg115886
versions: + Python 2.6, Python 3.3
2010-09-08 13:29:14haubisetmessages: + msg115868
2010-09-07 20:49:58sablesetmessages: + msg115805
2010-09-07 20:15:17haubisetfiles: + flock-check-py3k.patch

messages: + msg115801
2010-09-07 19:07:31pitrousetmessages: + msg115796
2010-09-07 17:39:51haubisetstatus: pending -> open

messages: + msg115789
2010-09-07 15:02:19pitrousetstatus: open -> pending
resolution: fixed
messages: + msg115770

versions: - Python 2.6, Python 3.3
2010-09-07 14:51:17sablesetmessages: + msg115768
2010-09-07 06:43:19haubisetmessages: + msg115744
2010-09-06 20:11:09pitrousetnosy: + pitrou
messages: + msg115733
2010-09-06 19:59:59sablesetmessages: + msg115732
2010-09-06 15:12:24haubisetmessages: + msg115711
2010-09-06 14:17:50sablesetfiles: + Python-2.7_flock_AIX.diff

messages: + msg115706
2010-09-02 14:18:48sablesetfiles: + Python-2.6.6_flock_AIX.diff
versions: + Python 3.1, Python 2.7, Python 3.2, Python 3.3
nosy: + sable

messages: + msg115377

keywords: + patch
2010-04-07 10:03:19haubisetmessages: + msg102529
2010-04-07 09:41:00haubisetnosy: + haubi
messages: + msg102526
2009-01-07 15:31:39darksidesetmessages: + msg79340
2009-01-06 21:41:43darksidesetnosy: + darkside
messages: + msg79289
2008-10-27 19:35:41inkblottersetnosy: + inkblotter
messages: + msg75260
2008-10-03 08:32:41drjcreate