classification
Title: ctypes module doesn't build on FreeBSD, RHEL (x86) - Undefined symbol "ffi_call_win32"
Type: compile error Stage: patch review
Components: Build, ctypes Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, Danya.Alexeyevsky, benjamin.peterson, berker.peksag, davin, koobs, larry, lemburg, pitrou, python-dev
Priority: release blocker Keywords: patch

Created on 2014-12-12 20:40 by lemburg, last changed 2015-05-09 02:24 by benjamin.peterson. This issue is now closed.

Files
File name Uploaded Description Edit
issue_23042_fix_windows_ifdefs_py35_and_py34_and_py27.patch davin, 2015-03-19 13:51 Patch changes ifdefs in ffi.c for 3.5, 3.4, and 2.7 branches review
ffi.patch lemburg, 2015-05-08 11:16 Fix ffi.c to compile correctly with 2.7, 3.4 and 3.5
Messages (19)
msg232574 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-12-12 20:40
With Python 2.7.8, ctypes builds fine on FreeBSD x86, but with Python 2.7.9, the build fails with:

*** WARNING: renaming "_ctypes" since importing it failed: build/lib.freebsd-8.3-RELEASE-p3-i386-2.7/_ctypes.so: Undefined symbol "ffi_call_win32"

Since this is FreeBSD, there shouldn't really be any calls to ffi_call_win32() in the _ctypes module or libffi.

Looking at the code changes in libffi/src/x86/ffi.c, it looks as if some of the #ifdefs for X86_WIN64 and X86_WIN32 were mixed up, causing calls to the Windows function to be compiled on FreeBSD x86, without also compiling the function definition in the same file.
msg232575 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-12-12 20:50
The cause seems to be these changes of the file (diff between the 2.7.8 and 2.7.9 version):

@@ -368,14 +374,21 @@ void ffi_call(ffi_cif *cif, void (*fn)(v
 #ifdef X86_WIN64
     case FFI_WIN64:
       ffi_call_win64(ffi_prep_args, &ecif, cif->bytes,
                      cif->flags, ecif.rvalue, fn);
       break;
-#elif defined(X86_WIN32)
+#else
+#ifndef X86_WIN32
+    case FFI_SYSV:
+      ffi_call_SYSV(ffi_prep_args, &ecif, cif->bytes, cif->flags, ecif.rvalue,
+                    fn);
+      break;
+#else
     case FFI_SYSV:
-    case FFI_STDCALL:
     case FFI_MS_CDECL:
+#endif
+    case FFI_STDCALL:
       ffi_call_win32(ffi_prep_args, &ecif, cif->abi, cif->bytes, cif->flags,
                     ecif.rvalue, fn);
       break;
     case FFI_THISCALL:
     case FFI_FASTCALL:
@@ -404,15 +417,10 @@ void ffi_call(ffi_cif *cif, void (*fn)(v
          abi = FFI_STDCALL;
         ffi_call_win32(ffi_prep_args, &ecif, abi, cif->bytes, cif->flags,
                        ecif.rvalue, fn);
       }
       break;
-#else
-    case FFI_SYSV:
-      ffi_call_SYSV(ffi_prep_args, &ecif, cif->bytes, cif->flags, ecif.rvalue,
-                    fn);
-      break;
 #endif
     default:
       FFI_ASSERT(0);
       break;
     }
@@ -785,14 +797,21 @@ ffi_raw_call(ffi_cif *cif, void (*fn)(vo
     ecif.rvalue = rvalue;


   switch (cif->abi)
     {
-#ifdef X86_WIN32
+#ifndef X86_WIN32
+    case FFI_SYSV:
+      ffi_call_SYSV(ffi_prep_args_raw, &ecif, cif->bytes, cif->flags,
+                    ecif.rvalue, fn);
+      break;
+#else
     case FFI_SYSV:
-    case FFI_STDCALL:
     case FFI_MS_CDECL:
+#endif
+#ifndef X86_WIN64
+    case FFI_STDCALL:
       ffi_call_win32(ffi_prep_args_raw, &ecif, cif->abi, cif->bytes, cif->flags,
                     ecif.rvalue, fn);
       break;
     case FFI_THISCALL:
     case FFI_FASTCALL:
@@ -821,15 +840,10 @@ ffi_raw_call(ffi_cif *cif, void (*fn)(vo
          cif->abi = abi = FFI_STDCALL;
         ffi_call_win32(ffi_prep_args_raw, &ecif, abi, cif->bytes, cif->flags,
                        ecif.rvalue, fn);
       }
       break;
-#else
-    case FFI_SYSV:
-      ffi_call_SYSV(ffi_prep_args_raw, &ecif, cif->bytes, cif->flags,
-                    ecif.rvalue, fn);
-      break;
 #endif
     default:
       FFI_ASSERT(0);
       break;
     }

If neither X86_WIN64 nor X86_WIN32 are defined, the #ifndefs cause the functions calls to be compiled in.
msg233304 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2015-01-02 02:46
@lemburg, does reverting this patch fix the regression for you?
msg238058 - (view) Author: Danya Alexeyevsky (Danya.Alexeyevsky) Date: 2015-03-13 22:16
Faced the same problem on FreeBSD.

I confirm: applying the reversed patch fixed it.

Thanks!
msg238446 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-18 14:08
Python 3.4 and 3.5 are also affected: I marked issue #22521 as a duplicate of this issue.
msg238452 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-03-18 15:31
To provide supporting information, using the i386 release of FreeBSD 10.1:

* ctypes fails to build with the latest from Python 2.7 branch with the message:

    build/temp.freebsd-10.1-RELEASE-i386-2.7-pydebug/usr/home/davin/pycoredev/cpython/Modules/_ctypes/libffi/src/x86/ffi.o: In function `ffi_prep_closure_loc':
    /usr/home/davin/pycoredev/cpython/Modules/_ctypes/libffi/src/x86/ffi.c:677: undefined reference to `ffi_closure_FASTCALL'
    /usr/home/davin/pycoredev/cpython/Modules/_ctypes/libffi/src/x86/ffi.c:683: undefined reference to `ffi_closure_THISCALL'
    /usr/home/davin/pycoredev/cpython/Modules/_ctypes/libffi/src/x86/ffi.c:689: undefined reference to `ffi_closure_STDCALL'
    /usr/bin/ld: build/lib.freebsd-10.1-RELEASE-i386-2.7-pydebug/_ctypes.so: hidden symbol `ffi_closure_FASTCALL' isn't defined
    /usr/bin/ld: final link failed: Nonrepresentable section on output
    cc: error: linker command failed with exit code 1 (use -v to see invocation)
    
    ...
    
    Failed to build these modules:
    _ctypes                                               


* Note that the above failure message was described in issue22521 by @Danya.Alexeyevsky

* ctypes also fails to build with the latest from default/3.5 branch with the very same message


Apparently the compilation failure messages differ between FreeBSD 8.3 and 10.1 (both i386).
msg238507 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-03-19 13:12
Reading issue22634 suggests that perhaps this issue is not limited to FreeBSD -- issue22634 appears to suffer the same behavior on RHEL 6.4 and 5.5 when compiling to 32-bit (the -m32 compiler flag is being used there).
msg238522 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-03-19 13:51
Attaching a patch which enabled successful compilation of ctypes on FreeBSD 10.1 i386-RELEASE in each of default/3.5, 3.4, and 2.7 branches.

This patch attempts to fix the incomplete logic regarding Windows target platforms in the ifdef's inside Modules/_ctypes/libffi/src/x86/ffi.c which at times tested for X86_WIN64 but not X86_WIN32 inside corresponding #else clauses (and vice versa).

This especially needs to be tested on a 32-bit Windows system -- I do not have one handy so help testing it there (and elsewhere) would be much appreciated.
msg238588 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-03-20 00:06
Inheriting the priority from issue22634 which has been marked closed->duplicate (duplicate of this issue).

Previously issue22634 was given a priority of "release blocker" by @pitrou.  Issue22634 described how this issue also afflicts 32-bit builds on multiple RHEL release versions.
msg238657 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2015-03-20 12:40
Modules/_ctypes/libffi in branches 2.7, 3.4, default is currently a copy of libffi 3.1.
Is problem reproducible with libffi 3.2.1? If not, then it might be better to update Modules/_ctypes/libffi to libffi 3.2.1.
msg238667 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2015-03-20 13:23
I would prefer to have someone with good libffi knowledge to chime in here. It's easy to just revert the patch, but it may be even better to upgrade the included libffi lib.

But probably not to 3.2.1, since this still has the same bug it seems:

https://github.com/atgreen/libffi/blob/v3.2.1/src/x86/ffi.c#L401
msg238748 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2015-03-21 02:00
@Marc, if upstream 3.2.1 also has the issue, then that would mean the current FreeBSD Python ports, which use --use-system-ffi and the security/libffi port, currently at version 3.2.1 [1], can reproduce the issue. 

I'm not aware of reports that they fail in this manner though.

[1] http://www.freshports.org/devel/libffi/
msg238767 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2015-03-21 07:29
See also: 

Issue: https://github.com/atgreen/libffi/issues/180
Title: OSX/i386 build is broken in v3.2.1
msg238793 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2015-03-21 11:55
@koobs: I was just looking at the 3.2.1 code where it still looks like ffi_call_win32() gets called even on non-Win32 platforms.

That said, it's possible that ffi_call_win32() is indeed defined on other platforms than Win32 as well in 3.2.1 - after all, it just implements a particular calling method. I haven't checked that or actually tried compiling _ctypes with 3.2.1.

Perhaps we could try switch to 3.2.1 and then check the buildbots for possible issues ?!
msg238802 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2015-03-21 13:56
@marc I took a look at the code upstream and it does indeed appear to be the same. It was introduced in 3.1 [1].

I cant explain however how or why our Python ports work with libffi 3.2.1.

See msg238767 for a link to another similar (same?) issue, with failure of OSX on 3.2.1 (building libffi, not python)

[1] https://github.com/atgreen/libffi/commit/e1911f78df113ca58738b66089a070d4cf747de7
msg242754 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2015-05-08 11:16
Here's a patch which I have tested on Linux, FreeBSD and Mac OS X.

It solves the problem with compiling in Windows calls on non-Windows platforms and resynchronizes the ffi_raw_call() function with the ffi_call() implementation. Both functions had the same issue and the only difference between the two is the use of ffi_prep_args_raw instead of ffi_prep_args.

As with davin's patch, this would need to be tested on Windows.
msg242790 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2015-05-09 01:31
Let's see what the buildbots think.
msg242791 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-05-09 01:31
New changeset 96a7b401d5e4 by Benjamin Peterson in branch '2.7':
fix libffi compilation on FreeBSD (#23042)
https://hg.python.org/cpython/rev/96a7b401d5e4
msg242792 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-05-09 01:32
New changeset a04b3de18c4c by Benjamin Peterson in branch '3.4':
fix libffi compilation on FreeBSD (#23042)
https://hg.python.org/cpython/rev/a04b3de18c4c

New changeset 987b30a88653 by Benjamin Peterson in branch 'default':
merge 3.4 (#23042)
https://hg.python.org/cpython/rev/987b30a88653
History
Date User Action Args
2015-05-09 02:24:14benjamin.petersonsetstatus: open -> closed
resolution: fixed
2015-05-09 01:32:50python-devsetmessages: + msg242792
2015-05-09 01:31:44python-devsetnosy: + python-dev
messages: + msg242791
2015-05-09 01:31:03benjamin.petersonsetmessages: + msg242790
2015-05-08 11:16:35lemburgsetfiles: + ffi.patch

messages: + msg242754
2015-03-21 13:56:04koobssetmessages: + msg238802
2015-03-21 11:55:08lemburgsetmessages: + msg238793
2015-03-21 07:29:54koobssetmessages: + msg238767
2015-03-21 02:00:11koobssetmessages: + msg238748
title: ctypes module doesn't build on FreeBSD x86 -> ctypes module doesn't build on FreeBSD, RHEL (x86) - Undefined symbol "ffi_call_win32"
2015-03-20 13:23:26lemburgsetmessages: + msg238667
2015-03-20 12:40:03Arfreversetmessages: + msg238657
2015-03-20 00:06:45davinsetpriority: normal -> release blocker
nosy: + benjamin.peterson, pitrou, berker.peksag, larry
messages: + msg238588

2015-03-19 15:32:35davinlinkissue22634 superseder
2015-03-19 13:54:21davinsetstage: patch review
2015-03-19 13:51:58davinsetfiles: + issue_23042_fix_windows_ifdefs_py35_and_py34_and_py27.patch
keywords: + patch
messages: + msg238522
2015-03-19 13:12:09davinsetmessages: + msg238507
2015-03-18 15:47:42vstinnersetnosy: - vstinner
2015-03-18 15:31:50davinsetnosy: + davin
messages: + msg238452
2015-03-18 14:08:19vstinnersetnosy: + vstinner
title: Python 2.7.9 ctypes module doesn't build on FreeBSD x86 -> ctypes module doesn't build on FreeBSD x86
messages: + msg238446

versions: + Python 3.4, Python 3.5
2015-03-18 14:08:17vstinnerlinkissue22521 superseder
2015-03-13 22:16:35Danya.Alexeyevskysetnosy: + Danya.Alexeyevsky
messages: + msg238058
2015-01-02 02:46:12koobssetnosy: + koobs
type: compile error
messages: + msg233304
2014-12-18 23:57:08Arfreversetnosy: + Arfrever
2014-12-12 20:50:31lemburgsetmessages: + msg232575
2014-12-12 20:40:59lemburgcreate