Title: libpython27.a in python-2.7.10 i386 (windows msi release) contains 64-bit objects
Type: Stage: resolved
Components: Build, Installation, Windows Versions: Python 3.6, Python 3.5, Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: bgilbert, carlkl, jaharkes, mrabarnett, paul.moore, python-dev, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2015-06-04 20:52 by jaharkes, last changed 2015-08-11 18:16 by steve.dower. This issue is now closed.

Messages (19)
msg244842 - (view) Author: Jan Harkes (jaharkes) Date: 2015-06-04 20:52
mingw32 fails to link with libpython27.a fails with the following error

/cygdrive/C/Python27/libs/libpython27.a: error adding symbols: File format not recognized

extracting all the objects from libpython27.a and checking them with objdump shows that two objects (dmmet.o and dmmeh.o) are not recognized, file returns 'data' for their file types.

objdump from a 64-bit version of mingw recognizes these as pe-x86-64.
msg244848 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-06-04 23:21
Given I can't generate the file any other way that will be compatible for everyone, unless someone contributes a fix I'm going to stop shipping these files and let people generate them using whatever tools they have.

If anyone wants to suggest instructions then I'll make a readme file that has some suggestions on how to generate them. Please include information about what build/fork/version of mingw your instructions are for.
msg244853 - (view) Author: Matthew Barnett (mrabarnett) * (Python triager) Date: 2015-06-05 02:46
Here's how I can build the regex module on Windows 8.1, 64-bit, using only MinGW64.

For Python 3.5, I can link against "python35.dll", but for earlier versions, including Python 2.7, I need "libpython??.a".

I have built regex module for all of the 16 supported versions of Python (2.5-2.7, 3.1-3.5, 64-bit and 32-bit) and they have all passed the tests.

rem For Python 3.5, 64-bit.
rem Can link against the Python DLL.

rem Compile
"C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python35-64\include" -c 
"D:\mrab-regex\source\_regex_unicode.c" -o 

"C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python35-64\include" -c 
"D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.5-64\_regex.o"

rem Link
"C:\MinGW64\bin\gcc.exe" -m64 -shared -s 
"D:\mrab-regex\release\3.5-64\_regex.o" -L"C:\Python35" -lpython35 -o 

rem For Python 3.5, 32-bit.
rem Can link against the Python DLL.

rem Compile
"C:\MinGW64\bin\gcc.exe" -mdll -m32  -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python35-32\include" -c 
"D:\mrab-regex\source\_regex_unicode.c" -o 

"C:\MinGW64\bin\gcc.exe" -mdll -m32  -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python35-32\include" -c 
"D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.5-32\_regex.o"

rem Link
"C:\MinGW64\bin\gcc.exe" -m32 -shared -s 
"D:\mrab-regex\release\3.5-32\_regex.o" -L"C:\Python35-32" -lpython35 -o 

rem For Python 3.4, 64-bit.
rem Need to link against the Python .a file.

rem Make libpython34.a
"C:\MinGW64\x86_64-w64-mingw32\bin\gendef.exe" - 
"C:\Windows\System32\python34.dll" >"C:\Python34-64\libs\libpython34.def"

"C:\MinGW64\bin\dlltool.exe" --dllname python34.dll --def 
"C:\Python34-64\libs\libpython34.def" --output-lib 

rem Compile
"C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python34-64\include" -c 
"D:\mrab-regex\source\_regex_unicode.c" -o 

rem Link
"C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python34-64\include" -c 
"D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.4-64\_regex.o"

"C:\MinGW64\bin\gcc.exe" -m64 -shared -s 
"D:\mrab-regex\release\3.4-64\_regex.o" -L"C:\Python34-64\libs" 
-lpython34 -o "D:\mrab-regex\release\3.4-64\_regex.pyd"

rem For Python 3.4, 32-bit.
rem Need to link against the Python .a file.

rem Make libpython34.a
"C:\MinGW64\x86_64-w64-mingw32\bin\gendef.exe" - 
"C:\Windows\SysWOW64\python34.dll" >"C:\Python34-32\libs\libpython34.def"

"C:\MinGW64\x86_64-w64-mingw32\bin\dlltool.exe" --as-flags=--32 -m i386 
--dllname python34.dll --def "C:\Python34-32\libs\libpython34.def" 
--output-lib "C:\Python34-32\libs\libpython34.a"

rem Compile
"C:\MinGW64\bin\gcc.exe" -mdll -m32  -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python34-32\include" -c 
"D:\mrab-regex\source\_regex_unicode.c" -o 

"C:\MinGW64\bin\gcc.exe" -mdll -m32  -O2 -Wall -Wsign-compare 
-Wconversion -I"C:\Python34-32\include" -c 
"D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.4-32\_regex.o"

rem Link
"C:\MinGW64\bin\gcc.exe" -m32 -shared -s 
"D:\mrab-regex\release\3.4-32\_regex.o" -L"C:\Python34-32\libs" 
-lpython34 -o "D:\mrab-regex\release\3.4-32\_regex.pyd"

rem For earlier versions of Python, follow the pattern of Python 3.4.
msg244874 - (view) Author: Jan Harkes (jaharkes) Date: 2015-06-05 19:35
The Python-2.7.9 version of the Windows-x86 MSI installed version works fine with the 32-bit mingw. In fact I rolled back to that release and built my code successfully.

Somehow the fix for must have broken the 32-bit libpython27.a.

In fact, just tried running the commands in the comments to that previous bug report by hand which produces a similarly broken 32-bit libpython27.a.

   x86_64-w64-mingw32-dlltool --dllname python27.dll --def mingwlib.def --output-lib win32\libpython27.a -m i386

This creates an archive with two 64-bit object files. However the following builds a correct import library with all 32-bit objects.

   i686-w64-mingw32-dlltool --dllname python27.dll --def mingwlib.def --output-lib win32\libpython27.a -m i386
msg244875 - (view) Author: Jan Harkes (jaharkes) Date: 2015-06-05 19:40
Just noticed in Matthew Barnett's comment that he has an additional "--as-flags=--32" argument.

  x86_64-w64-mingw32-dlltool --as-flags=--32 --dllname python27.dll --def mingwlib.def --output-lib win32\libpython27.a -m i386

Creates a correct libpython27.a where every object is 32-bit.
msg245015 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-06-08 16:12
The fix for issue23199 "broke" 32-bit mingw because nobody agrees which fork of mingw should be supported.

Before the next releases I intend to remove the lib completely and add a "libpython.txt" file in its place with commands or tools to try for generating the file with the fork/version you are currently using.

Thanks for the info.

@mrabarnett: are the `-L"C:\Python35" -lpython35` flags the ones that link to the DLL? (i.e. -L == search path, -l == link-to, and C:\Python35\python35.dll is the DLL)
msg245018 - (view) Author: Benjamin Gilbert (bgilbert) Date: 2015-06-08 16:48
Steve, I don't think that's the right characterization of this bug.  Putting 64-bit .o files in a 32-bit import library seems wrong for both MinGW and MinGW-w64.

msg244874 says that if you use the dlltool from the *32-bit* MinGW-w64 to build the 32-bit import library, the problem does not occur.  msg244875 says that you can even use the 64-bit MinGW-w64 dlltool by passing one additional command-line option.  Why not pursue one of those solutions?
msg245019 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-06-08 16:52
Ah, I misread msg244875 and thought it was still requiring different tools. I'll add that option in for the next release then, and if there are still issues I'll come back to removing the libraries.
msg245020 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-06-08 16:56
New changeset e4cde2f928fe by Steve Dower in branch '3.5':
Issue 24385: Adds "--as-flags=--32" when generating 32-bit MinGW library.

New changeset 5fa77372e675 by Steve Dower in branch 'default':
Issue 24385: Adds "--as-flags=--32" when generating 32-bit MinGW library.
msg245021 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-06-08 16:57
New changeset 686f9a607b98 by Steve Dower in branch '2.7':
Issue 24385: Adds "--as-flags=--32" when generating 32-bit MinGW library.
msg245022 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2015-06-08 16:58
If I'm not mistaken, if you're building an extension with some flavor of MinGW, you've got the tools to build the lib yourself, right?  It strikes me as being much more reliable to provide hints towards how to do it yourself than to provide a binary for something that's only marginally supported anyway.
msg245023 - (view) Author: Matthew Barnett (mrabarnett) * (Python triager) Date: 2015-06-08 17:04
@steve.dower: Yes.

For Python 35, it appears that it'll link to libpython??.a or python??.dll, whichever it finds in the given folder, so it doesn't actually need libpython??.a anymore. Which is nice. :-)
msg245027 - (view) Author: Benjamin Gilbert (bgilbert) Date: 2015-06-08 17:15
Zachary: gendef is an auxiliary tool; I'm not sure a basic installation of MinGW-w64 will have it.  However, I've checked Cygwin and Fedora (my two points of reference) and a gendef package is available in both.
msg245028 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-06-08 17:29
I'm still somewhat confused as to why we're looking at this in the context of manually building an extension. It's *certainly* true that anyone attempting to build a Python extension by hand (as per Matthew Barnett's instructions) should be able and willing to build the import libraries by hand.

On the other hand, I've no *objection* to shipping the .a files (and I agree 100% that shipping a 64-bit library in a 32-bit installer is a bug). But if we do so, then we should be shipping whatever makes build_ext --compiler=mingw (i.e. distutils) happy. Distutils is the only way of building extensions that I think we should support (whether via shipping .a files, or in terms of how we handle any other bug reports).
msg245091 - (view) Author: Carl Kleffner (carlkl) Date: 2015-06-09 20:18
The most robust way to support a specific mingw-w64 distribution is to copy a python import library generated by this specific toolchain and the provided import library for the corresponding msvcrXX.dll runtime into the libs folder.
This has to be performed by the user with the help of gendef and dlltool. Another atempt is to install and use a specific mingw-toolchain that copies these files into the libs folder during install. Such a toolchain will be provided in the near future as 'mingwpy' toolchain for python2.7-3.4.
msg245106 - (view) Author: Benjamin Gilbert (bgilbert) Date: 2015-06-10 02:57
Is there any reason an import library generated by a particular MinGW version shouldn't be usable by any other MinGW version?
msg245118 - (view) Author: Carl Kleffner (carlkl) Date: 2015-06-10 07:25
I experienced this behaviour by accident several times. Most likely this is due to inconsistenticies in different binutils and mingw-runtime versions/patches. 
See i.e.
A robust solution is to exclusively use import libraries created by the toolchain used for building python extensions.
msg245126 - (view) Author: Jan Harkes (jaharkes) Date: 2015-06-10 11:38
@Carl.Kleffner: that is an interesting message but it discusses using binutils(/mingw?) generated import libraries with the MS linker.

This bugreport was about two 64-bit objects that were accidentally included in an otherwise 32-bit library and it has been resolved.

I am not entirely sure why people are anticipating possible problems for which I have seen no actual bug reports or build failures. Searching this bugtracker for mingw related reports shows 187 reports over the past 13.5 years and at first glance I don't see any about compatibility issues between versions. There are only 3 for mingw+libpython, one of which is this report.

The various windows forks of mingw, maybe in an attempt to reach Visual C compatibility, may have as stable a format but the only failure I have personally seen was this 64-bit objects in a 32-bit library problem, which does look to me like a tool chain bug but luckily there was a relatively simple "--as-flags=--32" workaround.

I just reread my comment and I it reads ruder than I meant it to be, I'm truly sorry for that. I appreciated reading the link you included to see some of those demons lurking in the depths.
msg248421 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-08-11 18:16
Doesn't seem to be anything left to do here, so closing as fixed.
Date User Action Args
2015-08-11 18:16:32steve.dowersetstatus: open -> closed
resolution: fixed
messages: + msg248421

stage: resolved
2015-06-10 11:38:56jaharkessetmessages: + msg245126
2015-06-10 07:26:00carlklsetmessages: + msg245118
2015-06-10 02:57:27bgilbertsetmessages: + msg245106
2015-06-09 20:18:54carlklsetnosy: + carlkl
messages: + msg245091
2015-06-08 17:29:57paul.mooresetmessages: + msg245028
2015-06-08 17:15:24bgilbertsetmessages: + msg245027
2015-06-08 17:04:55mrabarnettsetmessages: + msg245023
2015-06-08 16:58:03zach.waresetmessages: + msg245022
2015-06-08 16:57:19python-devsetmessages: + msg245021
2015-06-08 16:56:21python-devsetnosy: + python-dev
messages: + msg245020
2015-06-08 16:52:52steve.dowersetmessages: + msg245019
2015-06-08 16:48:13bgilbertsetmessages: + msg245018
title: libpython.a does not work for all MinGW forks -> libpython27.a in python-2.7.10 i386 (windows msi release) contains 64-bit objects
2015-06-08 16:12:34steve.dowersettitle: libpython27.a in python-2.7.10 i386 (windows msi release) contains 64-bit objects -> libpython.a does not work for all MinGW forks
messages: + msg245015
versions: + Python 3.5, Python 3.6
2015-06-05 19:40:35jaharkessetmessages: + msg244875
2015-06-05 19:35:05jaharkessetmessages: + msg244874
2015-06-05 02:46:06mrabarnettsetnosy: + mrabarnett
messages: + msg244853
2015-06-05 00:04:00r.david.murraysetnosy: + paul.moore
2015-06-04 23:21:33steve.dowersetmessages: + msg244848
2015-06-04 22:34:30bgilbertsetnosy: + bgilbert
2015-06-04 22:22:06zach.waresetassignee: steve.dower

components: + Build, Installation, Windows, - Library (Lib)
nosy: + tim.golden, zach.ware, steve.dower
2015-06-04 20:52:13jaharkescreate