ctypes module doesn't build on FreeBSD, RHEL (x86) - Undefined symbol "ffi_call_win32" #67231

malemburg · 2014-12-12T20:41:00Z

BPO	23042
Nosy	@malemburg, @pitrou, @larryhastings, @benjaminp, @berkerpeksag, @koobs, @applio
Files	issue_23042_fix_windows_ifdefs_py35_and_py34_and_py27.patch: Patch changes ifdefs in ffi.c for 3.5, 3.4, and 2.7 branches ffi.patch: Fix ffi.c to compile correctly with 2.7, 3.4 and 3.5

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-05-09.02:24:14.641>
created_at = <Date 2014-12-12.20:40:59.777>
labels = ['ctypes', 'build', 'release-blocker']
title = 'ctypes module doesn\'t build on FreeBSD, RHEL (x86) - Undefined symbol "ffi_call_win32"'
updated_at = <Date 2015-05-09.02:24:14.640>
user = 'https://github.com/malemburg'

bugs.python.org fields:

activity = <Date 2015-05-09.02:24:14.640>
actor = 'benjamin.peterson'
assignee = 'none'
closed = True
closed_date = <Date 2015-05-09.02:24:14.641>
closer = 'benjamin.peterson'
components = ['Build', 'ctypes']
creation = <Date 2014-12-12.20:40:59.777>
creator = 'lemburg'
dependencies = []
files = ['38561', '39321']
hgrepos = []
issue_num = 23042
keywords = ['patch']
message_count = 19.0
messages = ['232574', '232575', '233304', '238058', '238446', '238452', '238507', '238522', '238588', '238657', '238667', '238748', '238767', '238793', '238802', '242754', '242790', '242791', '242792']
nosy_count = 10.0
nosy_names = ['lemburg', 'pitrou', 'larry', 'benjamin.peterson', 'Arfrever', 'Danya.Alexeyevsky', 'python-dev', 'berker.peksag', 'koobs', 'davin']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue23042'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

malemburg · 2014-12-12T20:40:59Z

With Python 2.7.8, ctypes builds fine on FreeBSD x86, but with Python 2.7.9, the build fails with:

*** WARNING: renaming "_ctypes" since importing it failed: build/lib.freebsd-8.3-RELEASE-p3-i386-2.7/_ctypes.so: Undefined symbol "ffi_call_win32"

Since this is FreeBSD, there shouldn't really be any calls to ffi_call_win32() in the _ctypes module or libffi.

Looking at the code changes in libffi/src/x86/ffi.c, it looks as if some of the #ifdefs for X86_WIN64 and X86_WIN32 were mixed up, causing calls to the Windows function to be compiled on FreeBSD x86, without also compiling the function definition in the same file.

malemburg · 2014-12-12T20:50:31Z

The cause seems to be these changes of the file (diff between the 2.7.8 and 2.7.9 version):

@@ -368,14 +374,21 @@ void ffi_call(ffi_cif *cif, void (*fn)(v
 #ifdef X86_WIN64
     case FFI_WIN64:
       ffi_call_win64(ffi_prep_args, &ecif, cif->bytes,
                      cif->flags, ecif.rvalue, fn);
       break;
-#elif defined(X86_WIN32)
+#else
+#ifndef X86_WIN32
+    case FFI_SYSV:
+      ffi_call_SYSV(ffi_prep_args, &ecif, cif->bytes, cif->flags, ecif.rvalue,
+                    fn);
+      break;
+#else
     case FFI_SYSV:
-    case FFI_STDCALL:
     case FFI_MS_CDECL:
+#endif
+    case FFI_STDCALL:
       ffi_call_win32(ffi_prep_args, &ecif, cif->abi, cif->bytes, cif->flags,
                     ecif.rvalue, fn);
       break;
     case FFI_THISCALL:
     case FFI_FASTCALL:
@@ -404,15 +417,10 @@ void ffi_call(ffi_cif *cif, void (*fn)(v
          abi = FFI_STDCALL;
         ffi_call_win32(ffi_prep_args, &ecif, abi, cif->bytes, cif->flags,
                        ecif.rvalue, fn);
       }
       break;
-#else
-    case FFI_SYSV:
-      ffi_call_SYSV(ffi_prep_args, &ecif, cif->bytes, cif->flags, ecif.rvalue,
-                    fn);
-      break;
 #endif
     default:
       FFI_ASSERT(0);
       break;
     }
@@ -785,14 +797,21 @@ ffi_raw_call(ffi_cif *cif, void (*fn)(vo
     ecif.rvalue = rvalue;


   switch (cif->abi)
     {
-#ifdef X86_WIN32
+#ifndef X86_WIN32
+    case FFI_SYSV:
+      ffi_call_SYSV(ffi_prep_args_raw, &ecif, cif->bytes, cif->flags,
+                    ecif.rvalue, fn);
+      break;
+#else
     case FFI_SYSV:
-    case FFI_STDCALL:
     case FFI_MS_CDECL:
+#endif
+#ifndef X86_WIN64
+    case FFI_STDCALL:
       ffi_call_win32(ffi_prep_args_raw, &ecif, cif->abi, cif->bytes, cif->flags,
                     ecif.rvalue, fn);
       break;
     case FFI_THISCALL:
     case FFI_FASTCALL:
@@ -821,15 +840,10 @@ ffi_raw_call(ffi_cif *cif, void (*fn)(vo
          cif->abi = abi = FFI_STDCALL;
         ffi_call_win32(ffi_prep_args_raw, &ecif, abi, cif->bytes, cif->flags,
                        ecif.rvalue, fn);
       }
       break;
-#else
-    case FFI_SYSV:
-      ffi_call_SYSV(ffi_prep_args_raw, &ecif, cif->bytes, cif->flags,
-                    ecif.rvalue, fn);
-      break;
 #endif
     default:
       FFI_ASSERT(0);
       break;
     }

If neither X86_WIN64 nor X86_WIN32 are defined, the #ifndefs cause the functions calls to be compiled in.

koobs · 2015-01-02T02:46:13Z

@lemburg, does reverting this patch fix the regression for you?

DanyaAlexeyevsky · 2015-03-13T22:16:36Z

Faced the same problem on FreeBSD.

I confirm: applying the reversed patch fixed it.

Thanks!

vstinner · 2015-03-18T14:08:20Z

Python 3.4 and 3.5 are also affected: I marked issue bpo-22521 as a duplicate of this issue.

applio · 2015-03-18T15:31:50Z

To provide supporting information, using the i386 release of FreeBSD 10.1:

ctypes fails to build with the latest from Python 2.7 branch with the message:

build/temp.freebsd-10.1-RELEASE-i386-2.7-pydebug/usr/home/davin/pycoredev/cpython/Modules/_ctypes/libffi/src/x86/ffi.o: In function ffi_prep_closure_loc': /usr/home/davin/pycoredev/cpython/Modules/_ctypes/libffi/src/x86/ffi.c:677: undefined reference to ffi_closure_FASTCALL'
/usr/home/davin/pycoredev/cpython/Modules/_ctypes/libffi/src/x86/ffi.c:683: undefined reference to ffi_closure_THISCALL' /usr/home/davin/pycoredev/cpython/Modules/_ctypes/libffi/src/x86/ffi.c:689: undefined reference to ffi_closure_STDCALL'
/usr/bin/ld: build/lib.freebsd-10.1-RELEASE-i386-2.7-pydebug/_ctypes.so: hidden symbol `ffi_closure_FASTCALL' isn't defined
/usr/bin/ld: final link failed: Nonrepresentable section on output
cc: error: linker command failed with exit code 1 (use -v to see invocation)

...

Failed to build these modules:
_ctypes
Note that the above failure message was described in bpo-22521 by @Danya.Alexeyevsky
ctypes also fails to build with the latest from default/3.5 branch with the very same message

Apparently the compilation failure messages differ between FreeBSD 8.3 and 10.1 (both i386).

applio · 2015-03-19T13:12:09Z

Reading bpo-22634 suggests that perhaps this issue is not limited to FreeBSD -- bpo-22634 appears to suffer the same behavior on RHEL 6.4 and 5.5 when compiling to 32-bit (the -m32 compiler flag is being used there).

applio · 2015-03-19T13:51:58Z

Attaching a patch which enabled successful compilation of ctypes on FreeBSD 10.1 i386-RELEASE in each of default/3.5, 3.4, and 2.7 branches.

This patch attempts to fix the incomplete logic regarding Windows target platforms in the ifdef's inside Modules/_ctypes/libffi/src/x86/ffi.c which at times tested for X86_WIN64 but not X86_WIN32 inside corresponding #else clauses (and vice versa).

This especially needs to be tested on a 32-bit Windows system -- I do not have one handy so help testing it there (and elsewhere) would be much appreciated.

applio · 2015-03-20T00:06:46Z

Inheriting the priority from bpo-22634 which has been marked closed->duplicate (duplicate of this issue).

Previously bpo-22634 was given a priority of "release blocker" by @pitrou. bpo-22634 described how this issue also afflicts 32-bit builds on multiple RHEL release versions.

Arfrever · 2015-03-20T12:40:03Z

Modules/_ctypes/libffi in branches 2.7, 3.4, default is currently a copy of libffi 3.1.
Is problem reproducible with libffi 3.2.1? If not, then it might be better to update Modules/_ctypes/libffi to libffi 3.2.1.

malemburg · 2015-03-20T13:23:27Z

I would prefer to have someone with good libffi knowledge to chime in here. It's easy to just revert the patch, but it may be even better to upgrade the included libffi lib.

But probably not to 3.2.1, since this still has the same bug it seems:

https://github.com/atgreen/libffi/blob/v3.2.1/src/x86/ffi.c#L401

koobs · 2015-03-21T02:00:10Z

@marc, if upstream 3.2.1 also has the issue, then that would mean the current FreeBSD Python ports, which use --use-system-ffi and the security/libffi port, currently at version 3.2.1 [1], can reproduce the issue.

I'm not aware of reports that they fail in this manner though.

[1] http://www.freshports.org/devel/libffi/

koobs · 2015-03-21T07:29:54Z

See also:

Issue: https://github.com/atgreen/libffi/issues/180
Title: OSX/i386 build is broken in v3.2.1

malemburg · 2015-03-21T11:55:08Z

@koobs: I was just looking at the 3.2.1 code where it still looks like ffi_call_win32() gets called even on non-Win32 platforms.

That said, it's possible that ffi_call_win32() is indeed defined on other platforms than Win32 as well in 3.2.1 - after all, it just implements a particular calling method. I haven't checked that or actually tried compiling _ctypes with 3.2.1.

Perhaps we could try switch to 3.2.1 and then check the buildbots for possible issues ?!

koobs · 2015-03-21T13:56:05Z

@marc I took a look at the code upstream and it does indeed appear to be the same. It was introduced in 3.1 [1].

I cant explain however how or why our Python ports work with libffi 3.2.1.

See msg238767 for a link to another similar (same?) issue, with failure of OSX on 3.2.1 (building libffi, not python)

[1] atgreen/libffi@e1911f7

malemburg · 2015-05-08T11:16:35Z

Here's a patch which I have tested on Linux, FreeBSD and Mac OS X.

It solves the problem with compiling in Windows calls on non-Windows platforms and resynchronizes the ffi_raw_call() function with the ffi_call() implementation. Both functions had the same issue and the only difference between the two is the use of ffi_prep_args_raw instead of ffi_prep_args.

As with davin's patch, this would need to be tested on Windows.

benjaminp · 2015-05-09T01:31:04Z

Let's see what the buildbots think.

python-dev · 2015-05-09T01:31:45Z

New changeset 96a7b401d5e4 by Benjamin Peterson in branch '2.7':
fix libffi compilation on FreeBSD (bpo-23042)
https://hg.python.org/cpython/rev/96a7b401d5e4

python-dev · 2015-05-09T01:32:50Z

New changeset a04b3de18c4c by Benjamin Peterson in branch '3.4':
fix libffi compilation on FreeBSD (bpo-23042)
https://hg.python.org/cpython/rev/a04b3de18c4c

New changeset 987b30a88653 by Benjamin Peterson in branch 'default':
merge 3.4 (bpo-23042)
https://hg.python.org/cpython/rev/987b30a88653

malemburg added build The build process and cross-build topic-ctypes labels Dec 12, 2014

koobs added the build The build process and cross-build label Jan 2, 2015

vstinner changed the title ~~Python 2.7.9 ctypes module doesn't build on FreeBSD x86~~ ctypes module doesn't build on FreeBSD x86 Mar 18, 2015

applio added the release-blocker label Mar 20, 2015

koobs changed the title ~~ctypes module doesn't build on FreeBSD x86~~ ctypes module doesn't build on FreeBSD, RHEL (x86) - Undefined symbol "ffi_call_win32" Mar 21, 2015

benjaminp closed this as completed May 9, 2015

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ctypes module doesn't build on FreeBSD, RHEL (x86) - Undefined symbol "ffi_call_win32" #67231

ctypes module doesn't build on FreeBSD, RHEL (x86) - Undefined symbol "ffi_call_win32" #67231

malemburg commented Dec 12, 2014

malemburg commented Dec 12, 2014

malemburg commented Dec 12, 2014

koobs commented Jan 2, 2015

DanyaAlexeyevsky mannequin commented Mar 13, 2015

vstinner commented Mar 18, 2015

applio commented Mar 18, 2015

applio commented Mar 19, 2015

applio commented Mar 19, 2015

applio commented Mar 20, 2015

Arfrever mannequin commented Mar 20, 2015

malemburg commented Mar 20, 2015

koobs commented Mar 21, 2015

koobs commented Mar 21, 2015

malemburg commented Mar 21, 2015

koobs commented Mar 21, 2015

malemburg commented May 8, 2015

benjaminp commented May 9, 2015

python-dev mannequin commented May 9, 2015

python-dev mannequin commented May 9, 2015

ctypes module doesn't build on FreeBSD, RHEL (x86) - Undefined symbol "ffi_call_win32" #67231

ctypes module doesn't build on FreeBSD, RHEL (x86) - Undefined symbol "ffi_call_win32" #67231

Comments

malemburg commented Dec 12, 2014

malemburg commented Dec 12, 2014

malemburg commented Dec 12, 2014

koobs commented Jan 2, 2015

DanyaAlexeyevsky mannequin commented Mar 13, 2015

vstinner commented Mar 18, 2015

applio commented Mar 18, 2015

applio commented Mar 19, 2015

applio commented Mar 19, 2015

applio commented Mar 20, 2015

Arfrever mannequin commented Mar 20, 2015

malemburg commented Mar 20, 2015

koobs commented Mar 21, 2015

koobs commented Mar 21, 2015

malemburg commented Mar 21, 2015

koobs commented Mar 21, 2015

malemburg commented May 8, 2015

benjaminp commented May 9, 2015

python-dev mannequin commented May 9, 2015

python-dev mannequin commented May 9, 2015