classification
Title: cross and native build of python for mingw32 with distutils
Type: feature request Stage:
Components: Build Versions: Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: davidfraser, giampaolo.rodola, lkcl, rpetrov, tarek, zooko (6)
Priority: Keywords patch

Created on 2008-09-14 23:08 by rpetrov, last changed 2009-11-04 22:06 by rpetrov.

Files
File name Uploaded Description Edit Remove
python-trunk-20090722-MINGW.patch rpetrov, 2009-07-22 20:09
python-trunk-20091104-MINGW.patch rpetrov, 2009-11-04 22:06
Messages (19)
msg73239 - (view) Author: Roumen Petrov (rpetrov) Date: 2008-09-14 23:08
This is a completely new patch against trunk that try to resolve mingw32
build. The first version to show problems with current python builds
system, to propose solutions/work-arounds and to be stating point for
discussion. Also I prefer some issues to be resolved later when we agree
on the basic issues.

The patch is tested in cross-compilation environment and native build
has to be tested too. Also the patch show how to use cross-compilation
environment and require patch posted in
issue 3754 ("minimal cross-compilation support for configure") to be
applied first.

The new patch share many commons with issue 841454 "Cross building
python for mingw32" posted by Andreas Ames, on 2003-11-13), but build
process use distutils instead scons.
Also some mingw related changes from issue 1597850 ("Cross compiling
patches for MINGW" - Han-Wen Nienhuys, from 2006-11-16) are shared too,
but we differ in concept how to use cross-compilation environment.
The correct way is to use $host variable (see issue 3754).

Also my patch use python installed on build system along with some hacks
to overcome distutils limitations.

The main issue "how to select correct compiler" in cross-compilation
environment isn't resolved at all. The patch is tested with installed
python version 2.4. 
Usually the cross-compiler is installed in same directories as native
compiler but tools are prefixed by <CPU>-<VENDOR>- (see meanign of "host
triplet"). In this environment we can't cross-compile even if we select
mingw as compiler in arguments for "setup.py":
The method get_versions() from cygwinccompiler.py call find_executable
for to find gcc, ld, dllwrap but on build system they point to native
compiler and that aren't prefixed. Also the expression "result =
re.search('(\d+\.\d+(\.\d+)*)',out_string)" return as example
'2.17.50.0.6' and later StrictVersion fail since value is not in format
N.N{.N{A{N}}}. If we replace '*' with '?" in regular expression we will
get correct  result, but this isn't right solution.
So the question is 'how to pass compiler to distutils' (as example for
python 2.4) remain open.

Work-around: install mingw cross-suite in own directory (MINGW_ROOT) and
prepend PATH environment variable with both paths:
- $MINGW_ROOT/bin
- $MINGW_ROOT/<MINGW_HOST>/bin where <MINGW_HOST> is something like
i586-pc-mingw32 (depend from you host selected when you build suite).
In this case find_executable will find mingw ld and since -v option
return something like "GNU ld version 2.17.50 20060824", regular
expresion value that make StrictVersion happy.

Note that in both environment configure script find compiler tools
prefixed as example by "i586-pc-mingw32-".
My mingw is build for host "i386-mingw32msvc"(this isn't correct, but is
acceptable for host. For next version I plan to use for build the
correct one, i.e i386-pc-mingw32). So in the final Makefile I see:
CC=             i386-mingw32msvc-gcc
AR=             i386-mingw32msvc-ar
RANLIB=         i386-mingw32msvc-ranlib

Also setup.py set compiler attribute linker_so - for details see
comments in the patch.

Another point is to link installed python in top-build directory so that
on posix system distutils read variables from our makefile instead from
installed in build system. In this case python_build (an internal to
distutils flag) is set.

A, but not common, build issue is that some python 2.4 versions will
search for pyconfig-32.h instead pyconfig.h.
Symptoms:
=======================================
....
running build
running build_ext
error: ./pyconfig-32.h: No such file or directory
....
=======================================
I see this on cent-os (5.0?), but I can't see on slackware 11.0.
This patch don't address this.
Just go in top-build directory and create a link, i.e. "ln -s
./pyconfig.h pyconfig-32.h".


If you system support emulation after build you may run python.
The build python search for modules. After patch modules are with suffix
same as for native build - .pyd. You may thin them to the top-build
directory or to follow binary distribution - create directory DLLs,
enter into it and link pyd-files:
$ for f in ../build/lib.*/*.pyd; do ln -sf $f; done


Differences between mingw build python and binary distribution(native
build):
Mingw build follow posix rules and will create more modules than native
build. In the native build they are part from main executable(buildin).
This is the list:
array, audioop, binascii, _bisect, _bytesio, cmath, _codecs_cn,
_codecs_hk, _codecs_iso2022, _codecs_jp, _codecs_kr, _codecs_tw,
_collections, cPickle, cStringIO, _csv, datetime, _fileio, _functools,
future_builtins, _heapq, _hotshot, imageop, itertools, _json, _locale,
_lsprof, math, mmap, msvcrt, _multibytecodec, operator, parser, _random,
strop, _struct, _subprocess, time, _weakref, _winreg, zlib.
I think that this inconsistency is problem of native build and for now I
don't address it.

Because in my environment I still don't have installed mingw port for
some externals, build of following modules isn't tested:
_bsddb.pyd
_msi.pyd
_sqlite3.pyd
_tkinter.pyd



Known issues:
The float, math and other related tests fail(under emulation and on nt5.1).
The reason is that C runtime function strtod() can't parse as example
numbers with exponent lower than -308, can't parse inf, nan, etc.. The
python source contain a file Python/strtod.c and my attempt to modify
and use it didn't succeed.
Also the configure lack test for "working" strtod and I guess that tests
fail on posix systems without C99 support.
The library mingwex from mingw runtime version 3.15 has working
replacement for stdtod function.
For now my patch will not address old strtod function.

In mingw definition of some functions(as example getaddrinfo and
getnameinfo) depend of value of WINVER. This issue is commented in the
patch. If user define WINVER geater the 0x0500 to CPPFLAGS at configure
time the build will use C-runtime function otherwise fake-function from
getaddrinfo.c and getnameinfo.c. As is commented in the patch if program
is linked with C-runtime functions I expect to fail to run on w2k.
I prefer to left selection of C-runtime for future mingw patches.

The other issue is that I see failure for some tests that use sockets.
This problem isn't investigated yet.

We may group some changes in configure but this require to reorder some
commands as example '--with-pydebug'. Since this isn't mingw issue I
prefer don't change current order. Another example is LIBM - it is set
by configure but setup.py use own rule.
msg73241 - (view) Author: Roumen Petrov (rpetrov) Date: 2008-09-14 23:36
P.S.: this patch cover changes in the python C-code proposed in issue
1412448 as include only necessary modifications.
msg73699 - (view) Author: Roumen Petrov (rpetrov) Date: 2008-09-24 10:01
For the protocol: issue2445 impact proposed patch.
Also I finish the tests and I will upload soon new patch - I the current
patch (  	 rpetrov, 2008-09-15 02:08) "Modules/selectmodule.c" isn't
ported and this prevent socked and related modules to work.
Also I understand the one of the reasons python 2.6x to be linked with
msvcr90 - many of bugs related to math functions are fixed in this
version of runtime.
msg74090 - (view) Author: Roumen Petrov (rpetrov) Date: 2008-09-30 18:17
note: updated patch contain unsynchronized with trunk code in
./Objects/fileobject.c (after /* EINVAL is returned when an invalid
filename or ... )
msg76699 - (view) Author: Zooko O'Whielacronx (zooko) Date: 2008-12-01 15:57
I'm interested in this patch so I'm adding myself to the Nosy list.
msg77243 - (view) Author: Roumen Petrov (rpetrov) Date: 2008-12-07 19:22
The new patch is attached. Now the patch allow native build for mingw*
host OS. The build id tested in MSYS environment. The issue title is
updated to reflect this.
The main changes:
- native build require four more build-in modules (_functools, operator
,_locale, _winreg) otherwise setup.py fail to load;
- changes in distutils for python 2.6+. The previous patch work on
python 2.4 run in cross environment;
- don't build with libintl;
- use long double functions ldexp() atan2() from mingwex library as
replacement for buggy msvcrt implementation.


The new patch don't include changes related to issue 4288 (see also
issue 4279).
msg77246 - (view) Author: Roumen Petrov (rpetrov) Date: 2008-12-07 20:07
About the regression test results test.
The result from 2008-11-29 is:
306 tests OK.
1 test failed:
    test_sundry
54 tests skipped:
....
Those skips are all expected on win32.

To get those results you mingwex library has to contain fixes for two
issues(see mingw32 bug tracker).
- 2136252 (fixed in trunk): c99, printf and float zero precision;
- 2117590 (pending): correction of function asinh - inverse hyperbolic
sine. For the python regression tests copysign is enough.

About "test_sundry" depend on _msi module. I don't plan to work on mingw
build for _msi module. Note that mingw build is with OS system default
msvcrt so there is no "Assembly Hell".

The trunk from 2008-12-07 fail in addition on test_anydbm, test_bsddb
and test_whichdb.
No idea what is wrong. It seems to that break is related to updates. As
example dbm module still fail to build on my linux.
msg77259 - (view) Author: Roumen Petrov (rpetrov) Date: 2008-12-07 21:29
The changes related to issue 4483 break build on dbm based on "Berkeley
DB" so I update mingw patch to restore bulld to previous state.

Now I expect test_anydbm, test_bsddb and test_whichdb to succeed as
before one week.
msg79959 - (view) Author: Luke Kenneth Casson Leighton (lkcl) Date: 2009-01-16 15:53
roumen, hi,
i'm interested in collaborating with you to get python compiled
under wine (running msys+mingw32 under wine, on linux).
#4954 incorporates much of your work, and takes a slightly
different direction for the configure setup - the time taken to
complete configure under wine is intolerable (3 hours).
also i have found that by setting PATH=%PATH%;c:/mingw/bin in the
environment vars, the python.exe that's produced can successfully
be used to run setup.py build.
msg80019 - (view) Author: Roumen Petrov (rpetrov) Date: 2009-01-17 15:24
Hi Luke,
Initially I start build in cross environment. Later I setup a posix
environment, based on msys and in the last patch is commented which
module has to be build-in for native build.

I'm not surprised that you succeed to build in emulated environment and
I know that born shell work slow in this environment.
msg80028 - (view) Author: Luke Kenneth Casson Leighton (lkcl) Date: 2009-01-17 17:44
roumen, hi,
can you add:

       BASECFLAGS="-mthreads $BASECFLAGS"
       LDFLAGS="-mthreads $LDFLAGS"

when compiling with threads, and... a second request: _block_ people
from compiling without mthreads, because there's a bug in wine's
msvcrt _filbuf routine where it doesn't perform CRLF (at all)
and it should.

http://bugs.winehq.org/show_bug.cgi?id=16970
msg80037 - (view) Author: Luke Kenneth Casson Leighton (lkcl) Date: 2009-01-17 18:53
roumen, hi,
can you add:

       BASECFLAGS="-mthreads $BASECFLAGS"
       LDFLAGS="-mthreads $LDFLAGS"

when compiling with threads, and... a second request: _block_ people
from compiling without mthreads, because there's a bug in wine's
msvcrt _filbuf routine where it doesn't perform CRLF (at all)
and it should.

http://bugs.winehq.org/show_bug.cgi?id=16970
msg80347 - (view) Author: Terry J. Reedy (tjreedy) Date: 2009-01-21 21:25
Does this supercede #3754? If so, please add this as superseder and
close it.
msg80349 - (view) Author: Roumen Petrov (rpetrov) Date: 2009-01-21 21:53
Does this supercede #3754?  - no.
The issue #3754 introduce macro AC_CANONICAL_HOST(add standardize "host
triplets") + fixes for cross-compilation.
Changes to configure script in this patch show how to use "host triplet"
for mingw builds.
msg80970 - (view) Author: Tarek Ziadé (tarek) Date: 2009-02-02 17:43
maybe you can group all your patch into one single issue since they are
related, and remove old patches,
msg80997 - (view) Author: Roumen Petrov (rpetrov) Date: 2009-02-02 21:10
The proposed patch for this issue include parts of other pending issues
- so its all is single file. If python team don't like idea for
"canonical host names" (part of issue 3754) this patch can be modified
do not use host-triplet. Also some discussions in py-dev list show that
small patches are preferred. I'm ready to split into small patches to be
reviewed.
msg81170 - (view) Author: Roumen Petrov (rpetrov) Date: 2009-02-04 22:30
attached patch for trunk-20090204:
- removed patch from issue4587 (not relevant for mingw) ;
- improved static build now succeed (use --disable-shared). this is
follow-up of issue4494 + ... (don't expect loadable modules to work on
windows) ;
- restore win32.S removed in r69260. thanks thomas.
msg82114 - (view) Author: Roumen Petrov (rpetrov) Date: 2009-02-14 19:50
updated patch do not include Lib/distutils/... :

- .../command/build_scripts.py, fixed in another non-related issue4524 (
distutils build_script command failed with --with-suffix=3)

- .../tests/test_build_ext.py, fixed by issues related to builddir <>
srcdir. Now distutils support srcdir on all platforms and this include
win32.

and to apply cleanly to trunk 2009-02-14 (after minimal cross patch -
see above).
msg90819 - (view) Author: Roumen Petrov (rpetrov) Date: 2009-07-22 20:12
The last patch is updated to support build with GNU C Compiler v 4.4.0
for windows.
History
Date User Action Args
2009-11-04 22:06:49rpetrovsetfiles: + python-trunk-20091104-MINGW.patch
2009-10-29 11:10:47davidfrasersetnosy: + davidfraser
2009-07-22 20:12:22rpetrovsetmessages: + msg90819
2009-07-22 20:10:31rpetrovsetfiles: - python-trunk-20090612-MINGW.patch
2009-07-22 20:10:25rpetrovsetfiles: - python-trunk-20090416-MINGW.patch
2009-07-22 20:10:19rpetrovsetfiles: - python-trunk-20090214-MINGW.patch
2009-07-22 20:10:12rpetrovsetfiles: + python-trunk-20090722-MINGW.patch
2009-06-12 21:27:31rpetrovsetfiles: + python-trunk-20090612-MINGW.patch
2009-04-15 21:02:55rpetrovsetfiles: + python-trunk-20090416-MINGW.patch
2009-04-15 21:02:18rpetrovsetfiles: - python-trunk-20090415-MINGW.patch
2009-04-15 20:43:14rpetrovsetfiles: + python-trunk-20090415-MINGW.patch
2009-02-14 22:41:12tjreedysetnosy: - tjreedy
2009-02-14 19:50:44rpetrovsetfiles: - python-trunk-20090204-MINGW.patch
2009-02-14 19:50:38rpetrovsetfiles: - python-trunk.patch-MINGW-20090110
2009-02-14 19:50:29rpetrovsetfiles: + python-trunk-20090214-MINGW.patch
messages: + msg82114
2009-02-04 22:31:59rpetrovsetfiles: - python-trunk-MINGW.patch
2009-02-04 22:31:53rpetrovsetfiles: - python-trunk-MINGW.patch
2009-02-04 22:31:21rpetrovsetfiles: + python-trunk-20090204-MINGW.patch
messages: + msg81170
2009-02-03 16:57:03giampaolo.rodolasetnosy: + giampaolo.rodola
2009-02-03 15:12:15gvanrossumsetmessages: - msg81041
2009-02-03 10:33:28lkclsetmessages: + msg81041
2009-02-02 21:10:50rpetrovsetmessages: + msg80997
2009-02-02 17:43:09tareksetnosy: + tarek
messages: + msg80970
2009-01-21 21:53:25rpetrovsetmessages: + msg80349
2009-01-21 21:25:49tjreedysetnosy: + tjreedy
messages: + msg80347
2009-01-17 18:53:21lkclsetmessages: + msg80037
2009-01-17 17:44:04lkclsetmessages: + msg80028
2009-01-17 15:24:59rpetrovsetmessages: + msg80019
2009-01-16 15:53:56lkclsetmessages: + msg79959
2009-01-15 20:53:11lkclsetnosy: + lkcl
2009-01-10 18:41:55rpetrovsetfiles: + python-trunk.patch-MINGW-20090110
2008-12-07 21:30:04rpetrovsetfiles: - python-trunk-MINGW.patch
2008-12-07 21:29:34rpetrovsetfiles: + python-trunk-MINGW.patch
messages: + msg77259
2008-12-07 20:07:24rpetrovsetmessages: + msg77246
2008-12-07 19:22:20rpetrovsetfiles: + python-trunk-MINGW.patch
messages: + msg77243
title: cross building python for mingw32 with distutils -> cross and native build of python for mingw32 with distutils
2008-12-01 15:57:36zookosetnosy: + zooko
messages: + msg76699
2008-09-30 18:17:46rpetrovsetmessages: + msg74090
2008-09-30 18:03:51rpetrovsetfiles: - python-trunk.patch-MINGW
2008-09-30 18:03:40rpetrovsetfiles: + python-trunk-MINGW.patch
keywords: + patch
2008-09-24 10:01:06rpetrovsetmessages: + msg73699
2008-09-14 23:36:59rpetrovsetmessages: + msg73241
2008-09-14 23:08:18rpetrovcreate