mingw32 fails to link with libpython27.a fails with the following error

/cygdrive/C/Python27/libs/libpython27.a: error adding symbols: File format not recognized

extracting all the objects from libpython27.a and checking them with objdump shows that two objects (dmmet.o and dmmeh.o) are not recognized, file returns 'data' for their file types.

objdump from a 64-bit version of mingw recognizes these as pe-x86-64.

Given I can't generate the file any other way that will be compatible for everyone, unless someone contributes a fix I'm going to stop shipping these files and let people generate them using whatever tools they have.

If anyone wants to suggest instructions then I'll make a readme file that has some suggestions on how to generate them. Please include information about what build/fork/version of mingw your instructions are for.

Here's how I can build the regex module on Windows 8.1, 64-bit, using only MinGW64.

For Python 3.5, I can link against "python35.dll", but for earlier versions, including Python 2.7, I need "libpython??.a".

I have built regex module for all of the 16 supported versions of Python (2.5-2.7, 3.1-3.5, 64-bit and 32-bit) and they have all passed the tests.


rem For Python 3.5, 64-bit.
rem Can link against the Python DLL.

rem Compile
"C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python35-64\include" -c 
"D:\mrab-regex\source\_regex_unicode.c" -o 
"D:\mrab-regex\release\3.5-64\_regex_unicode.o"

"C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python35-64\include" -c 
"D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.5-64\_regex.o"

rem Link
"C:\MinGW64\bin\gcc.exe" -m64 -shared -s 
"D:\mrab-regex\release\3.5-64\_regex_unicode.o" 
"D:\mrab-regex\release\3.5-64\_regex.o" -L"C:\Python35" -lpython35 -o 
"D:\mrab-regex\release\3.5-64\_regex.pyd"


rem For Python 3.5, 32-bit.
rem Can link against the Python DLL.

rem Compile
"C:\MinGW64\bin\gcc.exe" -mdll -m32  -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python35-32\include" -c 
"D:\mrab-regex\source\_regex_unicode.c" -o 
"D:\mrab-regex\release\3.5-32\_regex_unicode.o"

"C:\MinGW64\bin\gcc.exe" -mdll -m32  -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python35-32\include" -c 
"D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.5-32\_regex.o"

rem Link
"C:\MinGW64\bin\gcc.exe" -m32 -shared -s 
"D:\mrab-regex\release\3.5-32\_regex_unicode.o" 
"D:\mrab-regex\release\3.5-32\_regex.o" -L"C:\Python35-32" -lpython35 -o 
"D:\mrab-regex\release\3.5-32\_regex.pyd"


rem For Python 3.4, 64-bit.
rem Need to link against the Python .a file.

rem Make libpython34.a
"C:\MinGW64\x86_64-w64-mingw32\bin\gendef.exe" - 
"C:\Windows\System32\python34.dll" >"C:\Python34-64\libs\libpython34.def"

"C:\MinGW64\bin\dlltool.exe" --dllname python34.dll --def 
"C:\Python34-64\libs\libpython34.def" --output-lib 
"C:\Python34-64\libs\libpython34.a"

rem Compile
"C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python34-64\include" -c 
"D:\mrab-regex\source\_regex_unicode.c" -o 
"D:\mrab-regex\release\3.4-64\_regex_unicode.o"

rem Link
"C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python34-64\include" -c 
"D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.4-64\_regex.o"

"C:\MinGW64\bin\gcc.exe" -m64 -shared -s 
"D:\mrab-regex\release\3.4-64\_regex_unicode.o" 
"D:\mrab-regex\release\3.4-64\_regex.o" -L"C:\Python34-64\libs" 
-lpython34 -o "D:\mrab-regex\release\3.4-64\_regex.pyd"


rem For Python 3.4, 32-bit.
rem Need to link against the Python .a file.

rem Make libpython34.a
"C:\MinGW64\x86_64-w64-mingw32\bin\gendef.exe" - 
"C:\Windows\SysWOW64\python34.dll" >"C:\Python34-32\libs\libpython34.def"

"C:\MinGW64\x86_64-w64-mingw32\bin\dlltool.exe" --as-flags=--32 -m i386 
--dllname python34.dll --def "C:\Python34-32\libs\libpython34.def" 
--output-lib "C:\Python34-32\libs\libpython34.a"

rem Compile
"C:\MinGW64\bin\gcc.exe" -mdll -m32  -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python34-32\include" -c 
"D:\mrab-regex\source\_regex_unicode.c" -o 
"D:\mrab-regex\release\3.4-32\_regex_unicode.o"

"C:\MinGW64\bin\gcc.exe" -mdll -m32  -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python34-32\include" -c 
"D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.4-32\_regex.o"

rem Link
"C:\MinGW64\bin\gcc.exe" -m32 -shared -s 
"D:\mrab-regex\release\3.4-32\_regex_unicode.o" 
"D:\mrab-regex\release\3.4-32\_regex.o" -L"C:\Python34-32\libs" 
-lpython34 -o "D:\mrab-regex\release\3.4-32\_regex.pyd"


rem For earlier versions of Python, follow the pattern of Python 3.4.

The Python-2.7.9 version of the Windows-x86 MSI installed version works fine with the 32-bit mingw. In fact I rolled back to that release and built my code successfully.

Somehow the fix for https://bugs.python.org/issue23199 must have broken the 32-bit libpython27.a.

In fact, just tried running the commands in the comments to that previous bug report by hand which produces a similarly broken 32-bit libpython27.a.

   x86_64-w64-mingw32-dlltool --dllname python27.dll --def mingwlib.def --output-lib win32\libpython27.a -m i386

This creates an archive with two 64-bit object files. However the following builds a correct import library with all 32-bit objects.

   i686-w64-mingw32-dlltool --dllname python27.dll --def mingwlib.def --output-lib win32\libpython27.a -m i386

Just noticed in Matthew Barnett's comment that he has an additional "--as-flags=--32" argument.

  x86_64-w64-mingw32-dlltool --as-flags=--32 --dllname python27.dll --def mingwlib.def --output-lib win32\libpython27.a -m i386

Creates a correct libpython27.a where every object is 32-bit.

The fix for issue23199 "broke" 32-bit mingw because nobody agrees which fork of mingw should be supported.

Before the next releases I intend to remove the lib completely and add a "libpython.txt" file in its place with commands or tools to try for generating the file with the fork/version you are currently using.

Thanks for the info.

@mrabarnett: are the `-L"C:\Python35" -lpython35` flags the ones that link to the DLL? (i.e. -L == search path, -l == link-to, and C:\Python35\python35.dll is the DLL)

Steve, I don't think that's the right characterization of this bug.  Putting 64-bit .o files in a 32-bit import library seems wrong for both MinGW and MinGW-w64.

msg244874 says that if you use the dlltool from the *32-bit* MinGW-w64 to build the 32-bit import library, the problem does not occur.  msg244875 says that you can even use the 64-bit MinGW-w64 dlltool by passing one additional command-line option.  Why not pursue one of those solutions?

Ah, I misread msg244875 and thought it was still requiring different tools. I'll add that option in for the next release then, and if there are still issues I'll come back to removing the libraries.

New changeset e4cde2f928fe by Steve Dower in branch '3.5':
Issue 24385: Adds "--as-flags=--32" when generating 32-bit MinGW library.
https://hg.python.org/cpython/rev/e4cde2f928fe

New changeset 5fa77372e675 by Steve Dower in branch 'default':
Issue 24385: Adds "--as-flags=--32" when generating 32-bit MinGW library.
https://hg.python.org/cpython/rev/5fa77372e675

New changeset 686f9a607b98 by Steve Dower in branch '2.7':
Issue 24385: Adds "--as-flags=--32" when generating 32-bit MinGW library.
https://hg.python.org/cpython/rev/686f9a607b98

If I'm not mistaken, if you're building an extension with some flavor of MinGW, you've got the tools to build the lib yourself, right?  It strikes me as being much more reliable to provide hints towards how to do it yourself than to provide a binary for something that's only marginally supported anyway.

@steve.dower: Yes.

For Python 35, it appears that it'll link to libpython??.a or python??.dll, whichever it finds in the given folder, so it doesn't actually need libpython??.a anymore. Which is nice. :-)

Zachary: gendef is an auxiliary tool; I'm not sure a basic installation of MinGW-w64 will have it.  However, I've checked Cygwin and Fedora (my two points of reference) and a gendef package is available in both.

I'm still somewhat confused as to why we're looking at this in the context of manually building an extension. It's *certainly* true that anyone attempting to build a Python extension by hand (as per Matthew Barnett's instructions) should be able and willing to build the import libraries by hand.

On the other hand, I've no *objection* to shipping the .a files (and I agree 100% that shipping a 64-bit library in a 32-bit installer is a bug). But if we do so, then we should be shipping whatever makes setup.py build_ext --compiler=mingw (i.e. distutils) happy. Distutils is the only way of building extensions that I think we should support (whether via shipping .a files, or in terms of how we handle any other bug reports).

The most robust way to support a specific mingw-w64 distribution is to copy a python import library generated by this specific toolchain and the provided import library for the corresponding msvcrXX.dll runtime into the libs folder.
This has to be performed by the user with the help of gendef and dlltool. Another atempt is to install and use a specific mingw-toolchain that copies these files into the libs folder during install. Such a toolchain will be provided in the near future as 'mingwpy' toolchain for python2.7-3.4.

Is there any reason an import library generated by a particular MinGW version shouldn't be usable by any other MinGW version?

I experienced this behaviour by accident several times. Most likely this is due to inconsistenticies in different binutils and mingw-runtime versions/patches. 
See i.e. http://article.gmane.org/gmane.comp.gnu.binutils/46799
A robust solution is to exclusively use import libraries created by the toolchain used for building python extensions.

@Carl.Kleffner: that is an interesting message but it discusses using binutils(/mingw?) generated import libraries with the MS linker.

This bugreport was about two 64-bit objects that were accidentally included in an otherwise 32-bit library and it has been resolved.

I am not entirely sure why people are anticipating possible problems for which I have seen no actual bug reports or build failures. Searching this bugtracker for mingw related reports shows 187 reports over the past 13.5 years and at first glance I don't see any about compatibility issues between versions. There are only 3 for mingw+libpython, one of which is this report.

The various windows forks of mingw, maybe in an attempt to reach Visual C compatibility, may have as stable a format but the only failure I have personally seen was this 64-bit objects in a 32-bit library problem, which does look to me like a tool chain bug but luckily there was a relatively simple "--as-flags=--32" workaround.

I just reread my comment and I it reads ruder than I meant it to be, I'm truly sorry for that. I appreciated reading the link you included to see some of those demons lurking in the depths.

Doesn't seem to be anything left to do here, so closing as fixed.