I inverted the actions in the test I proposed in msg115768 (quick note from home, should have tested at work before posting). It should be:

AC_MSG_CHECKING(for flock)
have_flock=no
AC_TRY_LINK([
#include "confdefs.h" 
#include <sys/file.h>
], [void* p = flock; flock(0, 0)],
  [have_flock=yes],
  [AC_CHECK_LIB(bsd,flock, [
    AC_DEFINE(FLOCK_NEEDS_LIBBSD, 1, Define if flock needs to be linked with bsd library.)
    have_flock=yes
    ])]
)
if test "$have_flock" = yes ; then
  AC_DEFINE(HAVE_FLOCK, 1, Define if you have the 'flock' function.)
fi
AC_MSG_RESULT($have_flock)

Mickael, does that new test works for you? I am not sure I follow your explanation: there is no more 'compile-check' (AC_TRY_COMPILE) in the new test I provided; just a 'link-check' (AC_TRY_LINK - which tests that compilation and link did work).
HAVE_FLOCK is undefined by default.
If the link-check succeeds, then HAVE_FLOCK is defined.
If it fails, then we try to look for flock in libbsd; if that test works, then HAVE_FLOCK is defined as well as FLOCK_NEEDS_LIBBSD.

In the case of broken AIX 64 bits, the link-check will fail (no flock in default C libraries); then the test to find flock in libbsd will also fail (no 64 bits objets in libbsd). So HAVE_FLOCK will be undefined, which is what you expect.

m4/autoconf is quite unintuitive so I may have made a mistake, or it may be easily misunderstood. But I intend my test to define HAVE_FLOCK only if compilation _and_ linking works, like you expect.