Created on 2008-03-30 07:01 by mhammond, last changed 2008-04-09 18:57 by loewis. This issue is now closed.
|windows-cross-compile.patch||mhammond, 2008-03-30 07:01||patch for cross-compilation on Windows|
|pcbuild.diff||theller, 2008-03-31 18:52||Changes to PC\pcbuild.sln|
|msg64743 - (view)||Author: Mark Hammond (mhammond) *||Date: 2008-03-30 07:01|
I've taken the liberty of adding Trent, Christian and Martin to the nosy list as I know they are actively, if reluctantly interested in this. This patch allows the distutils to cross-compile on Windows. It has been tested on x86 and amd64 platforms, with both platforms successfully able to natively and cross-compile extension modules and create binary distributions. To cross-compile, specify '--plat-name' to the build command (valid values are 'win32', 'win-amd64' and 'win-ia64'). This option name was chosen to be consistent with the bdist_dumb command. I've included the docs I added below (which are also in the patch), but note that as with native compilation using distutils, it's not necessary to set any environment variables or do anything else special with your environment to make this work. The patch also adds a x64 target for the 'bdist_wininst' target, which it creates as distutils/command/wininst-9.0-amd64.exe. This executable is necessary even for bdist_wininst to work natively on x64, but is still included here for simplicity. To assist with testing, I've also added a distutils setup.py script to the PC/example_nt directory. This is capable of creating bdist_wininst executables for both native and cross platforms; 'setup.py build --platname=win-amd64 bdist_wininst' will create an amd64 installer on an x86 machine. The patch has not been tested with a Visual Studio environment without cross-compile tools installed - it will obviously fail, but its not clear how ugly this failure will be. Below is the text I added to docs/distutils/builtdist.rst: Cross-compiling on Windows ===================== Starting with Python 2.6, distutils is capable of cross-compiling between Windows platforms. In practice, this means that with the correct tools installed, you can use a 32bit version of Windows to create 64bit extensions and vice-versa. To build for an alternate platform, specify the :option:`--plat-name` option to the build command. Valid values are currently 'win32', 'win-amd64' and 'win-ia64'. For example, on a 32bit version of Windows, you could execute:: python setup.py build --plat-name=win-amd64 to build a 64bit version of your extension. The Windows Installers also support this option, so the command:: python setup.py build --plat-name=win-amd64 bdist_wininst would create a 64bit installation executable on your 32bit version of Windows. Note that by default, Visual Studio 2008 does not install 64bit compilers or tools. You may need to reexecute the Visual Studio setup process and select these tools.
|msg64790 - (view)||Author: Thomas Heller (theller) *||Date: 2008-03-31 18:52|
I had to make additional changes to PCBuild\pcbuild.sln to create a 64-bit wininst-9.0-amd64.exe (but I was not able to try out if it works or not).
|msg64825 - (view)||Author: Marc-Andre Lemburg (lemburg) *||Date: 2008-04-01 22:02|
Hi Mark, I hope this patch makes it into 2.6. One thing I'm curious about is the version of "Visual Studio 2008" that is needed to support this kind cross-compilation and which version will be or is used to build the Python versions on 64-bit Windows.
|msg64826 - (view)||Author: Marc-Andre Lemburg (lemburg) *||Date: 2008-04-01 22:09|
Nevermind, diving deep into the MSDN, I found the answer: http://msdn2.microsoft.com/en-us/library/hs24szh9.aspx So for cross-compilation, the Visual Studio 2008 Standard Edition is sufficient. Is the Professional Edition used for the standard Python binaries ? Does it generate faster code ? A bit off-topic, I know, but it's really hard to find this important information elsewhere and may well be of use for other Python extension authors as well. Thanks.
|msg64828 - (view)||Author: Mark Hammond (mhammond) *||Date: 2008-04-01 22:30|
Hi Marc-Andre, The PCBuild/README.txt file has some info about compatibility with VS versions (but probably needs to say more about x64 builds). There is also talk about releasing "Profile Guided Optimization" built binaries, which does require a professional version. I believe, but am not sure, that Martin uses the professional versions to make the official binaries. Either way, this might be getting off-topic for this patch.
|msg64834 - (view)||Author: Trent Nelson (trent) *||Date: 2008-04-02 05:26|
Hi guys, have been on holiday since PyCon ended, only getting back into the swing of things now. With regards to the x64 Windows build, indeed, PCbuild/readme.txt is definitely in need of an update, especially with the details of the slightly new build system I checked in a few weeks ago. As it currently stands, the Professional version of Visual Studio 2008 is required for a complete x64 build, however, this is because a cross- compilation 'vsvars32.bat'-type file needs to be called in order to do command line (i.e. nmake) x64 builds. (The particular file I'm referring to is "C:\Program Files (x86) \Microsoft Visual Studio 9.0\VC\vcvarsall.bat", which needs to be invoked with 'x86_amd64' as the first parameter. This is only shipped with the Professional version of Visual Studio 2008 -- the buildbots currently rely on it to produce x64 builds, see http://svn.python.org/projects/python/trunk/Tools/buildbot/build- amd64.bat.) We need to invoke this particular batch file in order to build x64 versions tcl/tk and openssl because we're currently reliant on the nmake-oriented builds each of these projects ship with. That said, the next step in my build-related improvements is to remove this reliance and mimick the nmake-oriented builds with custom .vcproj builds that inherit all of our property sheets -- this is the approach I took with bsddb and sqlite3 and it worked *very*, *very* well. (See _bsddb44.vcproj and sqlite3.vcproj in the PCbuild directory.) There are many advantages with this approach. To name a few: the release versions of the modules will benefit from profile guided optimisation, just like the rest of the python build, the modules are significantly easier to debug, and there is no chance for conflicting compiler/linker flags (which was a continual source of pain with bsddb/sqlite3 when they were built with the vendor-supplied build system). So, eventually, you should be able to build and entire release version of Python x64 with just the Express edition of Visual Studio 2008 (note that the Express edition won't support a PGO build though). The 'official' release though will be built with a Professional version though, as Mark states, thus allowing us to take advantage of profile guided optimisation. Hopefully this isn't too much off topic. (I can't comment on the patches Mark has provided just yet as I haven't had a chance to review them -- just wanted to clear up the x64 cross-compilation questions.) Perhaps I should create a separate issue to track the work needed to cross compile an x64 build with the Express edition only. Any objections?
|msg64835 - (view)||Author: Christian Theune (ctheune)||Date: 2008-04-02 05:33|
Mark, I think you mixed me up with Christian Heimes, so I'm removing myself here (at least for now).
|msg64836 - (view)||Author: Trent Nelson (trent) *||Date: 2008-04-02 05:35|
Humm, just saw the MSDN page Marc-Andre referred to, which indeed seems to indicate the Express edition doesn't support x64 cross-compilation *at all*. AFAICT the latest Platform SDK (6.1) ships with an x64 cross compilation environment, but it's command line only, so unless there's a way to hook it up such that 'vcbuild' uses it with the Express edition as well -- it seems like you do need Standard edition or higher for x64 builds.
|msg64843 - (view)||Author: Christian Heimes (christian.heimes) *||Date: 2008-04-02 09:48|
Could somebody with a 64bit CPU please extend the PCbuild/README.txt? I don't have the means to test 64bit builds or cross compilation of a 32bit build on a 64bit system. I only have 32bit CPUs at home. +1 for nmake if you can get it to work. A while ago I suggested to ship both 32bit and 64bit lib files with every flavor of Python. With both library sets developers could cross compile Windows extensions. Comments?
|msg64849 - (view)||Author: Trent Nelson (trent) *||Date: 2008-04-02 12:13|
Christian, regarding shipping x64 .lib files as well as 32-bit ones -- sensible idea, but where would we place the x64 version libs? In Python [xx]\libs\amd64? (This would mirror the approach used with PCbuild and PCbuild\amd64 -- although there's currently an ongoing discussion on this topic in python-dev.) With regards to updating PCbuild/readme.txt, I'm planning on doing that by EOD today.
|msg64860 - (view)||Author: Trent Nelson (trent) *||Date: 2008-04-02 15:07|
Updated PCbuild/readme.txt in r62105 and r62106 for trunk and py3k respectively.
|msg64884 - (view)||Author: Mark Hammond (mhammond) *||Date: 2008-04-02 23:39|
I'd like to keep this issue specifically about cross-compilation in the current tree, and while some of the other ideas may have merit, let's not bog this issue down with them unless they directly impact the patch. Does anyone have any comments or concerns about my patch? Anything people would like to see changed before I check it in? Martin - my patch creates a new executable in Lib\distutils\command, and IMO it should be included in binary builds for all windows platforms. Is there something I should change in the MSI process?
|msg65059 - (view)||Author: Mark Hammond (mhammond) *||Date: 2008-04-07 02:00|
|msg65257 - (view)||Author: Martin v. Löwis (loewis) *||Date: 2008-04-09 18:57|
> Martin - my patch creates a new executable in Lib\distutils\command, and > IMO it should be included in binary builds for all windows platforms. > Is there something I should change in the MSI process? Yes, I fixed that in r62255: it now packages wininst*.exe from that directory. Regards, Martin
|2008-04-09 18:57:01||loewis||set||messages: + msg65257|
|2008-04-07 02:00:19||mhammond||set||status: open -> closed|
messages: + msg65059
|2008-04-02 23:39:15||mhammond||set||messages: + msg64884|
|2008-04-02 15:07:42||trent||set||messages: + msg64860|
|2008-04-02 12:13:09||trent||set||messages: + msg64849|
|2008-04-02 09:48:06||christian.heimes||set||messages: + msg64843|
|2008-04-02 05:35:09||trent||set||messages: + msg64836|
|2008-04-02 05:34:31||ctheune||set||nosy: - ctheune|
lemburg, loewis, mhammond, theller, ctheune, christian.heimes, trent|
messages: + msg64835
|2008-04-02 05:26:08||trent||set||messages: + msg64834|
|2008-04-01 22:30:36||mhammond||set||messages: + msg64828|
|2008-04-01 22:09:35||lemburg||set||messages: + msg64826|
messages: + msg64825
messages: + msg64790
|2008-03-31 18:31:50||theller||set||nosy: + theller|
|2008-03-31 11:49:04||christian.heimes||set||nosy: + christian.heimes|