classification
Title: libpython27.a in amd64 release is 32-bit
Type: compile error Stage: resolved
Components: Library (Lib), Windows Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: franke-daniel, paul.moore, python-dev, steve.dower, tim.golden, zach.ware, zwelch
Priority: normal Keywords: patch

Created on 2015-01-09 00:45 by zwelch, last changed 2015-04-09 10:46 by steve.dower. This issue is now closed.

Files
File name Uploaded Description Edit
23199.diff steve.dower, 2015-01-24 16:32
Messages (15)
msg233702 - (view) Author: Zach Welch (zwelch) Date: 2015-01-09 00:45
I tried to link a program against the libpython27.a provided by the latest 2.7.9 amd64 installer, only to discover that the provided library is a 32-bit version.  I had to go through the gendef/dlltool dance in order to produce a useable 64-bit library from the DLL.
msg233886 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-01-12 17:27
Thanks for the notice. I'll try and get that fixed up for the next 2.7 release.
msg233889 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-01-12 21:49
Is the libpython27.a file actually a 32-bit version or is it just corrupted? It seems to be considerably smaller than the version in the 32-bit installer, but it is certainly not being generated from the 32-bit version.

I didn't write this code originally, and I don't use MinGW, so I'm not entirely familiar with how it should work. Do I need separate 64-bit tools to build the library for the 64-bit DLL or will the 32-bit tools suffice? Are the interfaces stable enough within 64-bit versions of MinGW that shipping the library makes sense, or are people likely to always need to regenerate it anyway?
msg233892 - (view) Author: Zach Welch (zwelch) Date: 2015-01-12 22:25
The libpython27.a is an actual 32-bit version, as confirmed by running objdump -t on it.  It reports the sections' file format as pe-i386 instead of pe-x86-64.  I am only using it for building for the 64-bit target, so I cannot confirm its viability for 32-bit builds.

To build 64-bit Windows binaries, you need a MinGW-w64 toolchain (which is a completely separate project from the original MinGW project).  The target triplet is x86_64-w64-mingw32.

I cannot speak to the stability of the MinGW-w64 library interface, but I would expect it to be stable.  To wit, I would be shocked if future changes required regeneration of third-party .a files.
msg234196 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-01-17 20:21
So I've grabbed gendef and dlltool from the latest mingw-w64 and will look at using those in the future for both 2.7 and 3.5.

According to objdump, I can use these to create file format pe-i386 and pe-x86-64 with the same tools. Are these the correct formats for 32-bit and 64-bit respectively?

My commands are (approximately):

  gendef - python35.dll > mingwlib.def

  dlltool --dllname python35.dll --def mingwlib.def --output-lib amd64\libpython35.a -m i386:x86-64

  dlltool --dllname python35.dll --def mingwlib.def --output-lib win32\libpython35.a -m i386
msg234317 - (view) Author: Zach Welch (zwelch) Date: 2015-01-19 16:39
Yes, pe-i386 and pe-x86-64 are the respective 32-bit and 64-bit object formats.

Your commands seem reasonable.  With gendef, I just let it create a .def file with the same name (i.e. skip the '-' and redirection); in my mind, that reinforces the association between the dll and def files.  With dlltool, I did not have to use the -m flag (as x86-64 is the default for me), but I don't see anything wrong with being explicit.
msg234398 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-01-20 22:20
Just came across this advice on https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows:

> Do not use MinGW-w64. As you will notice, the MinGW import library for
> Python (e.g. libpython27.a) is omitted from the AMD64 version of
> Python. This is deliberate. Do not try to make one using dlltool.
> There is no official MinGW-w64 release yet, it is still in "beta" and
> considered unstable, although you can get a 64-bit build from e.g.
> TDM-GCC. There have also been issues with the mingw runtime
> conflicting with the MSVC runtime; this can happen from places you
> don't expect, such as inside runtime libraries for g++ or gfortran. To
> stay on the safe side, avoid MinGW-w64 for now.

How accurate is this? Would we be better to omit the mingw libraries from the installer and instead provide the commands (or even a shell script?) to generate it with whatever toolset you're currently using?
msg234407 - (view) Author: Zach Welch (zwelch) Date: 2015-01-20 23:38
That's certainly an interesting data point.  We are just beginning to use MinGW-w64 internally, so I do not have enough experience to confirm or deny that advice.  For various reasons, we must use cross-compiling on a Linux host, so the advice to use a native compiler is moot for our situation.

Certainly, documenting the absense of the 64-bit library would be good.  Providing a documented script to generate one is better.  Providing the library would be ideal, if there will not be forward compatibility or runtime issues.

It would be nice to see concrete details about the current state of affairs.  The cython project's warning would carry more weight with me if it contained links to specific details: mailing list discussion that led to the "deliberate" decision to omit the 64-bit library, bug reports filed against the mingw-w64 project about the "runtime issues", etc..  That said, such details probably do exist, but my cursory searching has failed to turn them up.
msg234619 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-01-24 16:32
It looks like #1088716 is where the change was made a bit over 10 years ago. Adding Paul to see if he has any further insight, but I'm pretty sure at that point there was no 64-bit Python (there certainly wasn't a 64-bit Windows that people were using), and probably nobody noticed/cared that it wasn't there. It may even have been missing completely from the 64-bit installer until I took over building and started doing it slightly differently from Martin.

I've attached a patch for msi.py that will use mingw-w64's tools when building the installer, but I suspect we're better off putting the instructions there rather than hoping build versions match.

Is there a mingw list we can post this thread to for opinions from more people who depend on this lib?
msg234630 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-01-24 18:42
There's a lot of politics around the mingw vs mingw64 situation, but in practice I believe the various mingw64 builds are fine. So I see no reason not to supply a correct 64-bit libpythonXY.a.

When the patch was made to include the libpythonXY.a file to the msi (wow! I'd forgotten I'd ever done a patch for the MSI build) there was only 32-bit systems, and there certainly wasnt't a 64-bit mingw.

The added patch seems like a reasonable approach.
msg235127 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-01-31 20:28
New changeset 536a816e7232 by Steve Dower in branch '2.7':
Issue #23199: libpython27.a in amd64 release is 32-bit
https://hg.python.org/cpython/rev/536a816e7232
msg236392 - (view) Author: Daniel Franke (franke-daniel) Date: 2015-02-21 22:19
I've downloaded and installed python-2.7.9.amd.msi and spent some hours trying to understand the myriad of i386/x86-64 related linker errors I get when attempting to build an extension module. Now that I found this report I can at least stop doubting my sanity.

As I take it that there won't be an updated package any time soon, would someone knowledgeable please be so kind and place copy-paste ready step-by-step instructions here on what to do to get a working 64bit .a file for the 2.7.9 amd64 package? This would be much appreciated. Thanks.
msg236400 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-02-22 06:58
I posted these commands above (modulo version number). If you have MinGW and binutils then it should Just Work, but I'm not very experienced with the range of installations you could have, so you may need to track down the gendef and dlltool tools:

  gendef - python27.dll > mingwlib.def

  dlltool --dllname python27.dll --def mingwlib.def --output-lib amd64\libpython27.a -m i386:x86-64

  dlltool --dllname python27.dll --def mingwlib.def --output-lib win32\libpython27.a -m i386
msg240302 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2015-04-09 01:57
Steve: is this fixed and ready for next time?
msg240316 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-04-09 10:46
Yep
History
Date User Action Args
2015-04-09 10:46:12steve.dowersetstatus: pending -> closed

messages: + msg240316
2015-04-09 01:57:25zach.waresetstatus: open -> pending
resolution: fixed
messages: + msg240302

stage: resolved
2015-02-22 06:58:57steve.dowersetmessages: + msg236400
2015-02-21 22:19:44franke-danielsetnosy: + franke-daniel
messages: + msg236392
2015-01-31 20:28:31python-devsetnosy: + python-dev
messages: + msg235127
2015-01-24 18:42:16paul.mooresetmessages: + msg234630
2015-01-24 16:32:12steve.dowersetfiles: + 23199.diff

nosy: + paul.moore
messages: + msg234619

keywords: + patch
2015-01-20 23:38:50zwelchsetmessages: + msg234407
2015-01-20 22:20:22steve.dowersetmessages: + msg234398
2015-01-19 16:39:03zwelchsetmessages: + msg234317
2015-01-17 20:21:03steve.dowersetmessages: + msg234196
2015-01-12 22:25:44zwelchsetmessages: + msg233892
2015-01-12 21:49:28steve.dowersetmessages: + msg233889
2015-01-12 17:27:39steve.dowersetassignee: steve.dower
messages: + msg233886
2015-01-09 00:45:37zwelchcreate