This is a completely new patch against trunk that try to resolve mingw32
build. The first version to show problems with current python builds
system, to propose solutions/work-arounds and to be stating point for
discussion. Also I prefer some issues to be resolved later when we agree
on the basic issues.

The patch is tested in cross-compilation environment and native build
has to be tested too. Also the patch show how to use cross-compilation
environment and require patch posted in
issue 3754 ("minimal cross-compilation support for configure") to be
applied first.

The new patch share many commons with issue 841454 "Cross building
python for mingw32" posted by Andreas Ames, on 2003-11-13), but build
process use distutils instead scons.
Also some mingw related changes from issue 1597850 ("Cross compiling
patches for MINGW" - Han-Wen Nienhuys, from 2006-11-16) are shared too,
but we differ in concept how to use cross-compilation environment.
The correct way is to use $host variable (see issue 3754).

Also my patch use python installed on build system along with some hacks
to overcome distutils limitations.

The main issue "how to select correct compiler" in cross-compilation
environment isn't resolved at all. The patch is tested with installed
python version 2.4. 
Usually the cross-compiler is installed in same directories as native
compiler but tools are prefixed by <CPU>-<VENDOR>- (see meanign of "host
triplet"). In this environment we can't cross-compile even if we select
mingw as compiler in arguments for "setup.py":
The method get_versions() from cygwinccompiler.py call find_executable
for to find gcc, ld, dllwrap but on build system they point to native
compiler and that aren't prefixed. Also the expression "result =
re.search('(\d+\.\d+(\.\d+)*)',out_string)" return as example
'2.17.50.0.6' and later StrictVersion fail since value is not in format
N.N{.N{A{N}}}. If we replace '*' with '?" in regular expression we will
get correct  result, but this isn't right solution.
So the question is 'how to pass compiler to distutils' (as example for
python 2.4) remain open.

Work-around: install mingw cross-suite in own directory (MINGW_ROOT) and
prepend PATH environment variable with both paths:
- $MINGW_ROOT/bin
- $MINGW_ROOT/<MINGW_HOST>/bin where <MINGW_HOST> is something like
i586-pc-mingw32 (depend from you host selected when you build suite).
In this case find_executable will find mingw ld and since -v option
return something like "GNU ld version 2.17.50 20060824", regular
expresion value that make StrictVersion happy.

Note that in both environment configure script find compiler tools
prefixed as example by "i586-pc-mingw32-".
My mingw is build for host "i386-mingw32msvc"(this isn't correct, but is
acceptable for host. For next version I plan to use for build the
correct one, i.e i386-pc-mingw32). So in the final Makefile I see:
CC=             i386-mingw32msvc-gcc
AR=             i386-mingw32msvc-ar
RANLIB=         i386-mingw32msvc-ranlib

Also setup.py set compiler attribute linker_so - for details see
comments in the patch.

Another point is to link installed python in top-build directory so that
on posix system distutils read variables from our makefile instead from
installed in build system. In this case python_build (an internal to
distutils flag) is set.

A, but not common, build issue is that some python 2.4 versions will
search for pyconfig-32.h instead pyconfig.h.
Symptoms:
=======================================
....
running build
running build_ext
error: ./pyconfig-32.h: No such file or directory
....
=======================================
I see this on cent-os (5.0?), but I can't see on slackware 11.0.
This patch don't address this.
Just go in top-build directory and create a link, i.e. "ln -s
./pyconfig.h pyconfig-32.h".


If you system support emulation after build you may run python.
The build python search for modules. After patch modules are with suffix
same as for native build - .pyd. You may thin them to the top-build
directory or to follow binary distribution - create directory DLLs,
enter into it and link pyd-files:
$ for f in ../build/lib.*/*.pyd; do ln -sf $f; done


Differences between mingw build python and binary distribution(native
build):
Mingw build follow posix rules and will create more modules than native
build. In the native build they are part from main executable(buildin).
This is the list:
array, audioop, binascii, _bisect, _bytesio, cmath, _codecs_cn,
_codecs_hk, _codecs_iso2022, _codecs_jp, _codecs_kr, _codecs_tw,
_collections, cPickle, cStringIO, _csv, datetime, _fileio, _functools,
future_builtins, _heapq, _hotshot, imageop, itertools, _json, _locale,
_lsprof, math, mmap, msvcrt, _multibytecodec, operator, parser, _random,
strop, _struct, _subprocess, time, _weakref, _winreg, zlib.
I think that this inconsistency is problem of native build and for now I
don't address it.

Because in my environment I still don't have installed mingw port for
some externals, build of following modules isn't tested:
_bsddb.pyd
_msi.pyd
_sqlite3.pyd
_tkinter.pyd



Known issues:
The float, math and other related tests fail(under emulation and on nt5.1).
The reason is that C runtime function strtod() can't parse as example
numbers with exponent lower than -308, can't parse inf, nan, etc.. The
python source contain a file Python/strtod.c and my attempt to modify
and use it didn't succeed.
Also the configure lack test for "working" strtod and I guess that tests
fail on posix systems without C99 support.
The library mingwex from mingw runtime version 3.15 has working
replacement for stdtod function.
For now my patch will not address old strtod function.

In mingw definition of some functions(as example getaddrinfo and
getnameinfo) depend of value of WINVER. This issue is commented in the
patch. If user define WINVER geater the 0x0500 to CPPFLAGS at configure
time the build will use C-runtime function otherwise fake-function from
getaddrinfo.c and getnameinfo.c. As is commented in the patch if program
is linked with C-runtime functions I expect to fail to run on w2k.
I prefer to left selection of C-runtime for future mingw patches.

The other issue is that I see failure for some tests that use sockets.
This problem isn't investigated yet.

We may group some changes in configure but this require to reorder some
commands as example '--with-pydebug'. Since this isn't mingw issue I
prefer don't change current order. Another example is LIBM - it is set
by configure but setup.py use own rule.

P.S.: this patch cover changes in the python C-code proposed in issue
1412448 as include only necessary modifications.

For the protocol: issue2445 impact proposed patch.
Also I finish the tests and I will upload soon new patch - I the current
patch (  	 rpetrov, 2008-09-15 02:08) "Modules/selectmodule.c" isn't
ported and this prevent socked and related modules to work.
Also I understand the one of the reasons python 2.6x to be linked with
msvcr90 - many of bugs related to math functions are fixed in this
version of runtime.

note: updated patch contain unsynchronized with trunk code in
./Objects/fileobject.c (after /* EINVAL is returned when an invalid
filename or ... )

I'm interested in this patch so I'm adding myself to the Nosy list.

The new patch is attached. Now the patch allow native build for mingw*
host OS. The build id tested in MSYS environment. The issue title is
updated to reflect this.
The main changes:
- native build require four more build-in modules (_functools, operator
,_locale, _winreg) otherwise setup.py fail to load;
- changes in distutils for python 2.6+. The previous patch work on
python 2.4 run in cross environment;
- don't build with libintl;
- use long double functions ldexp() atan2() from mingwex library as
replacement for buggy msvcrt implementation.


The new patch don't include changes related to issue 4288 (see also
issue 4279).

About the regression test results test.
The result from 2008-11-29 is:
306 tests OK.
1 test failed:
    test_sundry
54 tests skipped:
....
Those skips are all expected on win32.

To get those results you mingwex library has to contain fixes for two
issues(see mingw32 bug tracker).
- 2136252 (fixed in trunk): c99, printf and float zero precision;
- 2117590 (pending): correction of function asinh - inverse hyperbolic
sine. For the python regression tests copysign is enough.

About "test_sundry" depend on _msi module. I don't plan to work on mingw
build for _msi module. Note that mingw build is with OS system default
msvcrt so there is no "Assembly Hell".

The trunk from 2008-12-07 fail in addition on test_anydbm, test_bsddb
and test_whichdb.
No idea what is wrong. It seems to that break is related to updates. As
example dbm module still fail to build on my linux.

The changes related to issue 4483 break build on dbm based on "Berkeley
DB" so I update mingw patch to restore bulld to previous state.

Now I expect test_anydbm, test_bsddb and test_whichdb to succeed as
before one week.

roumen, hi,
i'm interested in collaborating with you to get python compiled
under wine (running msys+mingw32 under wine, on linux).
#4954 incorporates much of your work, and takes a slightly
different direction for the configure setup - the time taken to
complete configure under wine is intolerable (3 hours).
also i have found that by setting PATH=%PATH%;c:/mingw/bin in the
environment vars, the python.exe that's produced can successfully
be used to run setup.py build.

Hi Luke,
Initially I start build in cross environment. Later I setup a posix
environment, based on msys and in the last patch is commented which
module has to be build-in for native build.

I'm not surprised that you succeed to build in emulated environment and
I know that born shell work slow in this environment.

roumen, hi,
can you add:

       BASECFLAGS="-mthreads $BASECFLAGS"
       LDFLAGS="-mthreads $LDFLAGS"

when compiling with threads, and... a second request: _block_ people
from compiling without mthreads, because there's a bug in wine's
msvcrt _filbuf routine where it doesn't perform CRLF (at all)
and it should.

http://bugs.winehq.org/show_bug.cgi?id=16970

roumen, hi,
can you add:

       BASECFLAGS="-mthreads $BASECFLAGS"
       LDFLAGS="-mthreads $LDFLAGS"

when compiling with threads, and... a second request: _block_ people
from compiling without mthreads, because there's a bug in wine's
msvcrt _filbuf routine where it doesn't perform CRLF (at all)
and it should.

http://bugs.winehq.org/show_bug.cgi?id=16970

Does this supercede #3754? If so, please add this as superseder and
close it.

Does this supercede #3754?  - no.
The issue #3754 introduce macro AC_CANONICAL_HOST(add standardize "host
triplets") + fixes for cross-compilation.
Changes to configure script in this patch show how to use "host triplet"
for mingw builds.

maybe you can group all your patch into one single issue since they are
related, and remove old patches,

The proposed patch for this issue include parts of other pending issues
- so its all is single file. If python team don't like idea for
"canonical host names" (part of issue 3754) this patch can be modified
do not use host-triplet. Also some discussions in py-dev list show that
small patches are preferred. I'm ready to split into small patches to be
reviewed.

attached patch for trunk-20090204:
- removed patch from issue4587 (not relevant for mingw) ;
- improved static build now succeed (use --disable-shared). this is
follow-up of issue4494 + ... (don't expect loadable modules to work on
windows) ;
- restore win32.S removed in r69260. thanks thomas.

updated patch do not include Lib/distutils/... :

- .../command/build_scripts.py, fixed in another non-related issue4524 (
distutils build_script command failed with --with-suffix=3)

- .../tests/test_build_ext.py, fixed by issues related to builddir <>
srcdir. Now distutils support srcdir on all platforms and this include
win32.

and to apply cleanly to trunk 2009-02-14 (after minimal cross patch -
see above).

The last patch is updated to support build with GNU C Compiler v 4.4.0
for windows.

system python has to be at least revision 77704 (trunk)

updated patch to trunk:
- apply cleanly
- support builddir <> sourcedir
- regression tests pass if run from any drive (windows hosts)

distutils from trunk is restored to state before 9 months ago

Roumen, I am moving all the new work in distutils into distutils2, and distutils is going to be frozen.

Do you want to work on this feature in distutils2 ? (which is distutils code base before I started to revert things)

Let me know so I mark this issue under "Distutils2"

Tarek Ziadé wrote:
> Roumen, I am moving all the new work in distutils into distutils2, and distutils is going to be frozen.

Last patch is uploaded as the previous patches  didn't apply cleanly (as 
compiler_obj is restored to compiler and etc.)

> Do you want to work on this feature in distutils2 ? (which is distutils code base before I started to revert things)

Yes.
Also I don't know where is distutils2 repository.

> Let me know so I mark this issue under "Distutils2"

No idea, because I expect Distutils2 to work with python 3+ and this 
patch is not tested(ported) to 3+.

Roumen

Common to all platforms part of patches lets call it  "build modules with system python"  is moved to issue 3754.

Roumen thanks for this work, but will you upload a new version?
Just because i can't get it to work, i can't even patch it^^

Maybe it's the SVN Client i use (Tortoise SVN) or that i don't know how to apply this patch :D It would be nice when you help me

Kind regards

Patches to this issue require first patch from issue 3754. The last patch here now is from 2010-05-16. This patch cannot be applied to trunk as it will fail after update for readline in setup.py. You could use trunk version before this readline update.

Now I'm working on readline and ncurses support for mingw . All patches before skip this as MSVC build don't support them. As I could not found reason to limit mingw users I will post soon version without limitation.

P.S. Many thanks to евроком (www.ekk.com now blizzo) - a company that left me for a week without internet connection.

- require patch "...20100626-CROSS..." from issue3871;
- setup.py don't ignore readline and ncurses for host platform  : python tests with readline pass, ncurses (current development)  is not usable yet;
- follow restored build with python specific windows version of libffi.

Could someone with knowledge of the Python build systems please comment on this, thanks.

Here's an alternative patch that enables Python building with MinGW/MSys
I've skimmed rpetrov's patch, and it seems that my patch is more about building Python than about distutils (i've patched distutils only as far as the compilation of Python itself (with the extensions already included in the trunk) requires.

Brief patch description - what i did, and why:
Python/import.c:
* briefly fixed an issue with mkdir() already pointed out in code comments

Python/dynload_win.c:
* renamed strcasecmp to pystrcasecmp to avoid conflicts with the same function defined in MinGW

Python/thread.c:
Python/ceval_gil.h:
* Made sure that Python won't try to include pthreads header when NT threads are available

Include/osdefs.h:
* Made sure that the PYCC_GCC won't lead Python into defining *nix path separation scheme
* Fixed redefinition of MAXPATHLEN (one of the few redefinitions i did fix, there are just too many of them)

configure.in:
* Added corret MACHDEP="win32" for MinGW
* Added lots of new AC_SUBST'ed variables, they are used later to alter Makefile and Setup functionality
  MACHDEP_PATH_SEPARATOR - to put ';' instead of ':' as path separator in machdep
  NTOBJS - extra object files to insert into makefile
  FROZEN_MAIN - controls the inclusion of Python/frozenmain.o into libpython. Why is it even there? Prevents the library from linking.
  NT_LDFLAGS - extra LDFLAGS to add on Windows
  SHAREDINSTALL - controls which sharedinstall target is built (works around MSys weirdness, see below)
  USE_*_MODULE - controls the inclusion of modules in Setup.config.in
  ms_windows - indicates that we're building on Windows through the rest of configure.in
  Special MINGW32_NT*/1.* case entry sets all of the above as well as some some #defines that traditionally are just hard-coded in PC/pyconfig.h (not all the defines from PC/pyconfig.h are here, there might be a need to add some more)
* Added MINGW32* to trigger CYGWIN* cases or added separate MINGW32* cases in most uname tests.
* Added MS_COREDLL definition (why is it separate from other Windows-specific AC_DEFINEs? I don't remember...)
* Added a separate check for CreateThread, a special variable (ac_cv_nt_threads) to indicate its success, and some code to switch on NT threads and switch off any other threads (including pthreads which ARE available on Windows)
* Added checks for some headers (pwd.h, sys/ioctl.h etc) and functions (spawn, spawnv, system etc) that are (are they?) traditionally available on *nix and are not checked for.
* Added correct DYNLOADFILE definition for MinGW
* Worked around AC_STRUCT_ST_BLOCKS weirdness by removing fileblocks.o from the list of object files (this one might not be portable, should check that it works on *nix)
* Fixed a strange bug in AC_COMPILE_IFELSE around "have_prototypes=yes" - when left with an empty argument [] it probuces broken (?) code in configure script (or maybe it's bash that is broken, i don't know)
* Fixed /dev/ptmx detection - MSys will tell you it's available, but it is only an MSys thing, Python shouldn't think it is there.
* Fixed PY_FORMAT_SIZE_T definition - AFAIK MinGW does not allow %z format in printf (in my test cases i've found that some definitions can prevent GCC from throwing an error about this format, but i couldn't reproduce this GCC behaviour in Python build system).
* Added ws2tcpip.h into socket test
* Added PC as a subdirectory

setup.py:
* Added module's location into sys.path. This is required to correctly import multiprocessing (it imports itself recursively, and since its .pyd file is in unusual location, it couldn't find itself)
* Added some config_h_vars.get() checks for some extensions (which is why some extra function/header checks were necessary in configure.in) - no setup.py won't try to build extensions that can't be built
* Added "'msys' in sys.builtin_module_names" kind of checks in some places. When successful, this check indicates that Python is built in MSys, and alters otherwise win32-oriented logic in setup.py and some other modules to adapt to MSys environment
* Changed the way MODULE_NAME is passed to sqlite3 extension (see below)
* Added extra library dependencies for multiprocessing extension
* Prevented setup.py from adding libX11 dependency to tkinter on Windows
* Added some msysize() calls (see below) to fix the paths passed to os.system() calls
* Added extra library dependencies for ctypes extension

PC/errmap.h:
* Re-generated it with new generrmap.c (see below)

PC/generrmap.c:
* Added an implementation of _dosmaperr() function, which is available in MSVC MSVCRT static library, but not in its MinGW counterpart (code is from PostgreSQL, should be compatible with Python license)

Lib/distutils/spawn.py:
* Fixed spawn_nt to use os.system() in conjunction with 'env.exe' program to call external tools - works like a charm in MSys ('env.exe' is an MSys tool, lives in /bin, and it hooks up MSys shell just as well as a call to sh.exe would). I don't know why i'm not calling sh.exe directly here. Just saw os.system()/env pair somewhere else, and it stuck.

Lib/distutils/util.py:
* Added a primitive msysize() function - an MSys path converter. Keeps MSys shell happy by avoiding paths with '\' path separators and '[A-Za-z]:' drive letters

Lib/distutils/ccompiler.py:
* Rigged ccompiler to use unixcompiler subclass for MSys. I didn't know at the time that there is the cygwincompiler, which should have been closer to MSys than unixcompiler. However from the looks of things it took only few lines to fix unixcompiler to work with MSys, so i kept it this way.

Lib/distutils/command/install.py:
* Added special installation schemes for MSys. Not sure if they are used during the installation of Python itself, but they look cool. It always bothered me that Python's layout on Windows is significantly different from the one on *nix, that breaks most autotools magic aimed at detecting/using Python. Now i have a chance to set things right.

Lib/distutils/command/build_ext.py:
* OK, this one is a real hack. There is virtually no way to find include and library paths for GCC: they USUALLY point to /mingw/include/* and /mingw/lib , but even that is configurable in MSys, and Python doesn't have direct access to that data -  asking GCC should always work, so we call GCC via os.system() and ask it to provide include and lib directories (GCC output is fed into a file, since os.system() doesn't have any piping)
* Added libpython naming scheme for Msys (similar to the one used for 'unix')

Lib/distutils/sysconfig.py:
* Added _init_msys() initialization function which combines _init_nt(), _init_posix() and some custom fixes (which come from PC/pyconfig.h for MSVC)

Lib/sysconfig.py:
* Added some more "'msys' in sys.builtin_module_names" checks to make sure that some code branches get executed or not.

Makefile.pre.in:
* Added all the variables from configure.in to insert correct object modules and compiler/linker flags
* Added explicit --export-all-symbols for building libpython
* Added some extravagant rules to build PC/generrmap.h and build exceptions.o which depends on it
* Added special msys sharedinstall variant with fixed paths (otherwise, with normal sharedinstall, some stuff ends up in $(DESTDIR)/MSys/1.0.11/usr/local or something like that due to MSys substituting /usr/local or any other prefix for its Win32 counterpart, which in this particular case is completely wrong, since prefix is always a relative path which shouldn't be fixed)
* Added some Windows-specific clean target commands

Modules/makesetup:
* Added some MinGW specific definitions (cloned from Cygwin's ones, didn't have to alter them after that)

Modules/_ctypes/callproc.c:
* Replaced some MS_WIN32 with _MSC_VER, since the code they guard works only with MSVC

Modules/_ctypes/_ctypes_test.c:
* Prevented preprocessor from inserting dllexport attributes (their presence disables automatic symbol exporting for module initialization function)

Modules/getpath.c:
* Fixed reduce() to work with paths with mixed separators and with paths with Windows-style drive letters

Modules/socketmodule.h:
* Avoided some clever MSSDK detection trap

Modules/_multiprocessing/semaphore.c:
* Fixed some #ifdef guards (the test is ambiguous: it can refer to broken sem_getvalue() - on *nix, or to the lack of one - on Windows)

Modules/Setup.dist:
* Used MACHDEP_PATH_SEPARATOR instead of hard-coding ':' separator into MACHDEPPATH
* Removed posixmodule and pwdmodule from the list of unconditionals modules, now they are conditional (see below)

Modules/_sqlite/connection.c:
Modules/_sqlite/cache.c:
Modules/_sqlite/statement.c:
Modules/_sqlite/row.c:
Modules/_sqlite/sqlitecompat.h:
Modules/_sqlite/cursor.c:
Modules/_sqlite/prepare_protocol.c:
Modules/_sqlite/module.c:
* I had some annoying problems passing MODULE_NAME=\"sqlite3\" - with escaping backslashes. It doesn't work well with my msysize(). Eventually i gave up and removed the need for doublequotes in the definitions by using stringifying feature of C preprocessor. Looks clumsy, but i do not insist on naming the macro Py_SQLITE_TO_STRING(), it can be something shorter...

Modules/python.c:
* Fixed main function prototype - MinGW doesn't work with wmain()

Modules/Setup.config.in:
* Made some modules configurable - some of them are not available on Windows, while others are only required to be built early on Windows. Adds MACHDEP_PATH_SEPARATOR definition as well.

Modules/posixmodule.c:
* Widened some include guards to include MinGW, added some includes for MinGW, disabled some structure re-definitions (it should be noted that i've used a patched version of MinGW's w32api package that already includes symlink functionality - the patch should eventually make it into official w32api release, so this particular fix in posixmodule might not be necessary for a while)

Modules/main.c:
* Fixed PATH_MAX redefinition

New version of my patch. Addresses some issues i've found since the last one. Diff against a recent trunk is attached.

Changes (relative to previous patch):

Python/pythonrun.c:
* Added a call to _set_abort_behavior(). That should fix http://bugs.python.org/issue9116

Python/dynamic_annotations.c:
* Changed _MSC_VER to MS_WINDOWS, since the code does not seem to be MSVC-specific

configure.in:
* Added AC_DEFINE for Py_SOCKET_FD_CAN_BE_GE_FD_SETSIZE (for MSVC-based builds it is hardcoded in PC/pyconfig.h)
* Added -no-undefined LDFLAG for Windows builds - helps find missing library dependencies at buildtime
* Added -fno-common and -mms-bitfields CFLAGS for Windows builds. -fno-common could fix some hidden problem with aligned commons, and -mms-bitfields should improve compatibility with MSVC-built binaries
* Added -D_GNU_SOURCE CFLAG. Defining this enables MinGW non-ANSI extensions, including the custom printf() implementation that supports %z format
* Added AC_DEFINE for HAVE_GETPEERNAME, because testing lws2_32 for it is tricky
* Removed PY_FORMAT_SIZE_T fiddling, since with GNU extensions %z is supported

setup.py:
* _subprocess module is now built on win32 to support subprocess extension
* msvcrt, winreg and winsound modules are built as well

PC/winreg.c:
* Added REG_LEGAL_CHANGE_FILTER definition (MinGW does not define it)

PC/dl_nt.c:
* Fixed PFN_ADDREFACTCTX and PFN_RELEASEACTCTX - according to MSDN, they return nothing
* Fixed the way AddRefActCtx is called (GetLastError() is being checked for errors instead of return value)

PC/generrmap.c:
* Added return 0 to prevent generrmap.exe from "failing" (returning something other than 0)

PC/msvcrtmodule.c:
* Added #ifdef guards around crtdbg.h, because only MSVC provides it

Lib/site.py:
* Extended os.sep == '/' (why isn't os.name tested?) to msys
* Extended addbuilddir calling condition to msys

Lib/distutils/sysconfig.py:
* Extended some posix conditions to msys, restricted some nt conditions to non-msys

Lib/distutils/tests/test_bdist_msi.py:
* Disabled this test for MSys (MinGW can't build msi module)

Lib/distutils/ccompiler.py:
* An extra hack to add $(srcdir) to library_dirs at build time

Lib/distutils/sysconfig.py:
* Some MSys-specific configuration magic

Lib/ntpath.py:
* Added a reminder that realpath() can now be implemented

Lib/sysconfig.py:
* Added msys-only variation of directory layout (copied from posix)

Lib/test/test_site.py:
* Widened os.sep == '/' condition to MSys

Makefile.pre.in:
* Fixed CFLAGS order for Objects/exceptions.o target (won't include pyconfig.h from PC anymore)

Modules/getpath.c:
* Massive fixes. Most thigs are copied from PC/getpathp.c (who had the bright idea of having two separate, but almost identical headers?)
* Ripped a readlink() implementation from posixmodule.c to use here. By the way, it was buggy, so i had to fix it as well.

Modules/socketmodule.h:
* Added some definitions for MinGW - these aren't in current MinGW w32api package, and will (probably) never get there.

Modules/_multiprocessing/win32_functions.c:
Modules/_multiprocessing/multiprocessing.h:
* Added more gurads on crtdbg stuff, because it is MSVC-only

Modules/selectmodule.c:
* Changed some guards from _MSC_VER to MS_WINDOWS, because the code is not MSVC-specific

Modules/posixmodule.c:
* Changed some guards to work for MinGW as well
* Discovered why i previously had to fix mkdir() call. Now that's not necessary anymore (but i left previous fix as it is, can't hurt to be careful)

I ran some tests and tried to make my Python build pass them, which partially fueled my bugfixing efforts.

_ctypes test (namely - test_callconv_1) crashes the interpreter (SIGSEGV in libffi), so i had to comment it out.

test_capi fails (but that should be now fixed in MSVC builds, can't test it in a MinGW build)
test_cmath fails (i have no idea why)
test_cmd_line fails (i think it is not adapted to MSys; and i'm not sure it needs to be - after all, this Python build is NOT intended to be ran in MSys most of the time)
test_distutils fails (i've fixed that only partially)
test_email fails (wrong crlfs)
test_format fails (?)
test_heapq fails (but when i re-run it, it passes; because of pre-compilation?)
test_platform fails (doesn't work well with symlinked python, because realpath isn't implemented on NT yet)
test_smtplib fails (no idea - why)
test_sundry fails (?)
test_sysconfig fails (didn't try to fix)
test_tarfile fails (didn't try to fix)
test_tcl fails (didn't try to fix)
test_trace (didn't try to fix)
test_urllib2 (didn't try to fix)

test_multiprocessing is skipped because there's no test for multiprocessing in Lib/test/

282 tests OK.

FYI, distutils is frozen because even minor bug fixes have broken third-party tools in the past, that’s why new features and bug fixes land in distutils2. Only some bug fixes that are sure not to break things make their way into distutils. distutils2 will be reintegrated into the stdlib in 3.2 or 3.3, but standalone releases will be made for Python 2.4 up to 2.7 and 3.x.

Where's the distutils2 code?

See http://bitbucket.org/tarek/distutils2

@LRN
what program did u used to create the patch?
And what to apply it?

Just because i can't get Tortoise to apply the patch (An unknown line type was found in line 678 !)
and MSYS's patch also does not work.

Uh, sorry about that. I've replaced the second diff with the correct one.

update patch still for 2.7 with
- restored build of ctypes (broken after recent updates)
- python build with ncurses library, test pass
- python build with readline library, test pass

On Sun, Aug 8, 2010 at 12:27 AM, Éric Araujo <report@bugs.python.org> wrote:
>
> Éric Araujo <merwok@netwok.org> added the comment:
>
> FYI, distutils is frozen because even minor bug fixes have broken third-party tools in the past, that’s why new features and bug fixes land in distutils2. Only some bug fixes that are sure not to break things make their way into distutils. distutils2 will be reintegrated into the stdlib in 3.2 or 3.3, but standalone releases will be made for Python 2.4 up to 2.7 and 3.x.

 eric,

 it's been a year since i did the mingw32 port so i'll try to recall
what it was that i did, but i have recollections of also having to add
to distutils, specifically adding an extra compiler class and an extra
linker class, derived from cygwin i believe (and then adding,
obviously, platform detection logic and options to make use of them).

 using logic analysis on the conditions, it can be shown that there is
nothing that can satisfy the conditions (one of which is "distutils is
frozen", another is "distutils will not go into python2.N or 3.x"
etc).  thus, something has to give.

 detailed analysis and recommendations follow:

 if distutils2 is to be "standalone" it literally makes it impossible
to build python 2.4 up to 2.7 and 3.x using mingw32, because the build
infrastructure will be entirely missing.  how can you build python 2.7
using mingw32 if one of the critical dependencies - actually saying
"use mingw32" - is missing?!!

 the decision to keep distutils frozen therefore forces the mingw32
developers to create a "special" version of distutils, one which has
to be added as a patch to python - let's call it distutils_mingw32 -
which has the exact same functionality as distutils except that it is
extended to have a mingw32 compiler and linker class.   adding such a
patch (which adds a copy of distutils called distutils_mingw32) would
be extremely disruptive, as it would defeat the entire purpose of
distutils, and i would be extremely surprised if it was accepted.

 so, unpalatable-as-defined-by-previous-negative-experience as it may
be, the only sensible option is to do an extremely careful and
thorough review of LRN's distutils compiler and linker class
additions, and ensure that they are _purely_ additional classes, even
at the expense of duplicated code.

 please remember that mingw32 is an _additional_ platform, not a
change to any _existing_ platforms.

l.

p.s. the only other option i can think of would be to add in
build-time options where you can specify the fully-qualified path to
where distutils2 can be found.  this would be somewhat... contrived,
but would "work".

sorry to have to ask, but could we get some feedback please so that this issue may move forward?  currently there is a conflict between what is required and what is stated as being "absolute law".

let's imagine that it is reasonable to expect distutils2 to be used.  python proceeds with the mingw32 patches using distutils2, python 2.N is compiled and released.  an average user installs python 2.N mingw32 and they run "python setup.py install" on a package - what happens?

they get a compile error, don't they?

why do they get a compile error?

because the setup.py tries to build some c code, and at the top of *their* setup.py is "import distutils".

so you now force thousands of developers to patch allll their setup.py scripts - scripts which have worked for years - _just_ so that those packages can cope with the use of distutils2 for the mingw32 platform?

so we have another "reducto ad absurdum" which demonstrates that it is impractical to expect distutils to be frozen.

thus, we are forced to consider alternative options, such as "monkey-patching" of distutils, to satisfy the requirements.

would the following be acceptable, to be inserted somewhere in the path so that it is guaranteed to work at all times?

import mingw32_distutils_compiler
import sys
if "distutils.compiler" in sys.modules:
    sys.modules['distutils.compiler'] = mingw32_distutils_compiler


where mingw32_distutils_compiler performs an import of distutils.compiler and performs a monkey-patch mish-mash to combine and replace various parts of the compiler module at run-time, to get round the "freeze" objections.

would that be acceptable?  please say no, because the long-term viability and maintainability of such practices is virtually nil.

basically i'm pointing out to you, eric, that the freeze on distutils is unworkable and impractical and unnecessary.

this isn't "bug-fixing" of distutils, it's absolutely necessary and critically required because this is an entirely new platform.

* it's not cygwin: cygwin uses standard gcc but with weird outputs and quirks.

* it's not standard unix gcc: you need to output ".dll" not ".so"

* it's not msvc: mingw32-gcc doesn't accept "/this /that" for options.

therefore i'm very sorry to have to spell it out but a new compiler and linker type - a NEW compiler and linker type - an ADDITIONAL compiler and linker type - IS required in order to cater for this platform, and there's no getting away from that fact.

please could we have some acknowledgement of this fact, and acceptance of this fact, and a plan for moving forward with careful review of this patch, so that LRN and others do not have their time spent pursuing a direction involving distutils2 which will be completely fruitless?

many many thanks,

l.

A small correction to previous post: two new compiler-and-linker types, not one.

First - MinGW toolset without MSys, working with normal Windows command interpreter, which can be invoked simply by calling "cmd" (or whatever we get from %ComSpec%). Obviously, it doesn't work. Well, it works, but only ~5% of all software packages will have a makefile that doesn't have to be generated by configure script (which we can't run, since this variant of MinGW doesn't have bash) and that doesn't call any shell builtins or *nix tools. On the other hand, SCons, CMake and other build systems not based on autotools should work fine without bash and/or should be able to generate makefiles that don't rely on bash or *nix tools.

Second - MinGW toolset with MSys, working with bash, which can't be invoked by calling "sh.exe" (even if you have full absolute path to sh.exe), because you must cast a few black magic spells by calling sh.exe with "--login" to hook up /etc/profile, which sets up PATH and other environment variables; without it you won't be able to run anything that isn't in c:/msys/1.0/bin - including mingw32-gcc.exe, which lives in c:/mingw/bin, unless you'd want to duplicate /etc/profile in Python (which is possible, remind me to try that), and you have to be careful to not give it any \-separated paths (even in scaped form) at command line, because bash won't understand them, and path mangling doesn't kick in until you're *within* bash, and that includes the name of the script you want to run - it must be /-separated and not absolute (because bash will pass it as $0 without mangling).

And the patch for py3k (future 3.2) . Note require python-py3k-20101112-CROSS.patch from issue 3754 to be applied first.

I'm trying to read the patch. It contains many interesting things (and others I have no opinon about), but it is very large, and makes it difficult to comment or find why some change were made etc.

I do believe that better supporting the mingw32 platform is a worthy goal, but a single large patch, even after review, is something difficult to trust.

I suggest to present this work in another form. Maybe something like a Mercurial Patch Queue?  This would give us a collection of smaller patches, each with its own interest.  Each one can be discussed, modified, reviewed separately and by different people. Large easy tasks (like a massive rename) are distinct from small-but-hard changes. And when something is updated, the reviewers don't have to parse the whole file again; they just look at the recent patches.

As an example: among the many changes, you had to handle the Modules/getpath.c vs. PC/getpathp.c inconsistency.  You chose to have both filenames in the Makefile, but in my opinion it would be better to try to merge the files.

Let’s try again.  The current situation is that building with mingw32 is not supported.  Adding support for mingw32 is by definition a new feature.  You seem to agree on that: “a NEW compiler and linker type - an ADDITIONAL compiler and linker type”.

Second, distutils is frozen.  That’s the policy decided by Tarek, who’s spent literal months working on PEPs to move the situation forward.  It’s just not possible to fix things in distutils, so it’s frozen and new work happens in distutils2, at the suggestion of Python creator Guido van Rossum.  Please accept the feature freeze.

In conclusion, such pervasive changes will not happen in distutils.

erik, i'm really sorry, but the freeze on distutils simply cannot be accepted: there really is no other way, as you can see from the previous walkthrough analysis, and is reinforced by the further analysis below.

simply put: if the freeze on distutils is not lifted, then this entire set of work, which has been going on for years and _precedes_ the distutils freeze by at least 18 months, is completely, utterly and totally wasted.

let's walk through the situation where distutils2 is forced to be used.

what you're asking for is, basically, that every single third party package, of which there must be tens of thousands, must be patched for compilation on mingw32... _just_ so that it says "if sys.platform == 'mingw32': from distutils2.core import setup else: from distutils.core import setup", is that correct?

does that strike you as being completely and utterly unreasonable an expectation, to ask third parties to modify setup.py scripts which have worked perfectly well for years, many of which are likely to no longer even have a maintainer?

that leaves patching - which should be nothing more than _adding_ to - not "changing existing compilers" - ADDING an extra compiler - distutils as the only option.

now, that can be done monkey-patch style (i.e. at runtime, by adding in code very early on in python startup, or perhaps by patching the import system to replace the word "distutils" with "distutilsmingw32") or it can be done... by lifting the distutils freeze.

perhaps i should ask: what _exactly_ is the problem, and why do several teams complete and utter failure to do the correct thing by making changes to _existing_ compile platforms have anything to do with _this_ team's patches which add a new totally separate platform that has absolutely nothing to do with any of the other platforms?

i could say more, but i believe the point is clear: none of the people involved in seeing mingw32 added as a platform had _anything_ to do with the past failures, so why "punish" and mis-trust the python-mingw32 for other peoples' failures?

We don't punish anyone here. We try to avoid breaking distutils because it's really easy to get broken and get people upset. (you ask me)

This bug has been opened for years and no one commited it because it's very very hard to review. it involves many parts, many people.

I am not sure how we should do this, but here's my proposal for distutils2 at least:

- make this new feature a standalone package that patches distutils
- release it for 2.x
- let's add your work in distutils2 as well, so it's back in the stdlib in 3.x

> I'm trying to read the patch. It contains many interesting things (and 
> others I have no opinon about), but it is very large, and makes it
> difficult to comment or find why some change were made etc.

amaury: unfortunately, the development on adding mingw32 as a platform has been ongoing for well over three years, with absolute silence and rejection of its existence.  it was only when a fourth person showed interest in it (LRN) that it became "accepted" - but that was... only two months ago!

basically, this situation should never have been allowed to get this
far: the very first patch that was created, three nearly four years ago, should have been accepted, and then an incremental process could have been taken, _and_ the silly situation in which distutils gets frozen during the time when people have been completely ignoring this ongoing work would never have occurred.

but, that's the situation: the bed has been made, and now developers have to lie in it.  sorry to be the one that's pointing this out, but...

anyway.

your idea to split this into a series has merit: personally i much prefer git, because of git-format-patch, but i have to say i've never done "patch regeneration" based on a "review / change-patch-in-middle-of-series / regenerate git-format-patch" cycle.  should be fun! :)

but the very very first thing that has to happen - before any of this work is begun - is for the distutils freeze to be lifted, or for someone to come up with a _sensible_ alternative solution.

if that cannot be done, then roumen and LRN won't _stop_ working on python-mingw32: the end result will be that they just... continue to create a single patch file that will just get larger and larger.

I understand your frustration, but I can't lift the freeze because any change to distutils potentially break third party projects out there. We suffered from that in the past and we decided to proceed on a new version and freeze this one.

Having this compiler released on its own project is absolutely possible and acceptable and can be done in a way that works with distutils1 and distutils2, and I'd happily integrate it to distutils2 today once we manage to review it.

> but the very very first thing that has to happen - before any of this
> work is begun - is for the distutils freeze to be lifted, or for 
> someone to come up with a _sensible_ alternative solution.

Tarek's approach looks right to me: mingw32 support in distutils should take the form of an external module that subclasses (when possible) or monkeypatches (on last resort) the standard distutils behavior.

The current patch makes too many changes in core distutils functions; it cannot be accepted in this form.  I'm sure that most of the needed changes can be made in a subclass of the present Mingw32CCompiler.

> I am not sure how we should do this, but here's my proposal
> for distutils2 at least:

> - make this new feature a standalone package that patches distutils
> - release it for 2.x
> - let's add your work in distutils2 as well, so it's back in the stdlib in 3.x

 that would work very well: people doing new setup.py files, converting from python2 etc., will, duh, need to do _some_ conversion: they'll expect breakage and to be using distutils2 anyway.

thank you tarek.

> The current patch makes too many changes in core distutils functions;
> it cannot be accepted in this form.  I'm sure that most of the needed
> changes can be made in a subclass of the present Mingw32CCompiler.

 that's what i did when creating the _other_ (yet another)
 mingw32 patch - however _some_ minimal changes to core distutils
 _are_ unfortunately required.  and to sys.py and os.py - this is,
 after all, a new platform!  it starts off with
 "sys.platform == 'mingw32'", requiring detection of gcc compiler
 type BUT and os type of win32, and goes from there.

 right now, detection logic is:

 * "if gcc on win32 platform, platform MUST be cygwin"
 * "if msvc compiler, platform MUST be win32"

 both of which are... well... wrong! :)

 so, i found that it was necessary to start in os.py and sys.py,
 create a platform-type "mingw32" and _then_ it was easy to do
 a MingW32Compiler etc. with (mostly) additions not modifications
 to distutils, but _necessary_ to add in detection of the type.

 if distutils was designed to do "exec import distutils.compiler.%s as compiler" % sys.platform and go from there then this would be an entirely non-issue: no modifications to distutils would be required, just a completely separate module containing the new compiler...

 ... but distutils isn't designed that way, is it? :)

 l.

> this is, after all, a new platform! it starts off with 
> "sys.platform == 'mingw32'"

I disagree; programs compiled with mingw32 run on Windows, and use the MSVC runtime. It's the same platform as the current win32 build.
It's even possible to use mingw32 to compile extensions for the VS9.0 based python.exe.
A different compiler does not make a new platform.

But different build system make new plaform !

Roumen

> But different build system make new plaform!
Maybe, but not a new "sys.platform".

VC6 and VS9.0 don't share any project file for the compilation; but they both build a sys.platform=='win32'.
Distutils has to use another way to make the difference and choose the right compiler (in distutils.mscvcompiler.get_build_version())

> I disagree;

 i would say that you're entitled to disagree, but i have to point
 out that unless you've actually been through the process of trying
 to port python to mingw32 you're not really in a position of ...
 how can i put this best... you're entitled to say "i disagree"
 but that doesn't actually carry any weight.

 if you said "i've tried compiling this patch, and i looked at
 it and i disagreed with the decision to create a new platform,
 so i removed that code, here's an updated patch, i found it to
 work absolutely fine" THEN i would say that you are in a position
 of authority to "disagree".

 so - question: have you actually _tried_ compiling python with
 mingw32, with the latest patch?


> programs compiled with mingw32 run on Windows, and use the MSVC 
> runtime.

 they do indeed.  this however has absolutely no relevance.

> It's the same platform as the current win32 build.
> It's even possible to use mingw32 to compile extensions
> for the VS9.0 based python.exe.
 
 it is indeed.  the patch that i did allowed you to specify
 a gcc spec file which did exactly that: i added options to
 compile not only extensions but also the entire python.exe
 to use a different MSVCRT runtime.

 _and_ it did assemblies, too.


> A different compiler does not make a new platform.

 ok.

 unfortunately, as the work that i did was well over a year ago,
 i'm going from memory - but basically, i'm very very sorry to
 have to point out that you don't know what you're talking about,
 here.

 let me try and go through it.

 look at the platform detection code: it parses the gcc version
 string.  it goes "if compiler has string gcc but also has win32
 then it must be cygwin platform".

 otherwise it goes "this must be MSVC win32 platform".

 this _simply_ doesn't work, hence the need to do further
 detection, hence the need to have a separate compiler type.
 ... but it doesn't end there: there are subtle differences
 between MSVC win32 and MINGW32 win32 (differences in the
 build .lib files that specify what functions are available
 in the DLLs. mingw32 is a reverse-engineering project,
 remember?)

 to be honest i can't remember if i actually set sys.platform to
 mingw32 - but the more time i spent getting more and more modules
 to compile, the more blindingly obvious it was that a new platform
 type was needed.

 i encountered dozens of assumptions that "if sys.platform == 'win32'
 then you MUST be building using visual studio: f*** off with your
 attempt to compile this module using gcc".
 
 over the eight to ten week period in which i focussed on this
 non-stop for about 13 hours a day, the list just went on and on
 of discrepancies that had to be solved, and if i _hadn't_ set a
 new platform type, it would have been necessary to add extra
 tests instead:

  "if sys.platform == 'win32' and not {something to detect mingw32}":

 mingw32 _really_ does fall between both worlds: not just the
 compiler type is different, but there are even features and
 functions _missing_ from mingw32 that are present in MSVC.
 i had to work with roumen to get patches to mingw32 upstream
 in some cases!

 so please _do_ stop putting road-blocks in the way.  this is
 a complex enough project, having to fit half way between
 two disparate worlds, without it being stymied by "disagreement"
 when you haven't _actually_ tried compiling this code (if you
 have, i apologise).

 btw if you'd like to try compiling it, but are adamant about
 staying away from proprietary operating systems, i _did_
 manage to get python 2.5 and 2.6 cross-compiled to run under wine.
 ironically there were long-standing bugs in wine that ended up
 getting fixed as a result of running the 25,000 python unit tests,
 but that's another story... :)

 l.

 p.s. msys runs under wine as well, but the configure stage takes
 well over an hour.  the patch i created cut out most of configure
 and replaced it with a pcconfig.h just like win32, which i had
 to create by hand.  this was quicker than waiting for configure
 to run.

perhaps, amaury, you might like to, instead of saying "i disagree", you might like instead to say something like this:

"that sounds... interesting, and a little scary - creating an entirely new platform!  are you absolutely sure it's necessary??  could you please perhaps elaborate and give a good justification behind why that decision was taken?"

you see how radically different that is?  on the one hand you're telling three volunteers who have spent considerable time and resources - unpaid - on improving python's reach that they are, to put it bluntly, complete ignorant f*****g morons, and on the other you're engaging with them, encouraging them, and generally trusting them.

i'm really sorry: i really don't like having to be the one to point
these kinds of things out, but... you see what i'm saying?

No it is win32 for the resulting binaries.

Idea is following (issue3754) . Travial patch - move method get_platform 
global _get_platform and variable host_platform is initialized to its 
return value. Then replace all calls of sys.platform and get_platform() 
to use host_platform.

In the patch for this issue _get_platform is modiffed to return mingw32 
if ....  so host_platform will be 'minwg32'. Including compiler and etc. 
don't work well in cross-compilation environment as the build script is 
run with python from build system.

Roumen

Luke,
Please please! Don't make me say what I did not say. I just said "I disagree", without any bad word. This may have been a bit terse, but I don't consider the issue tracker as a place to express one's feelings; hey, other core developers could have written "-1" instead.
I'm not trying to block this development, on the contrary, I'm getting interested in it so that we can merge it into CPython development tree. I'm here to help. Really.

I did try to port python to the mingw32 compiler some years ago, and in the proposed patch I retrieve some of the changes I did at the time.  IIRC I gave up because there were issues with the localtime function, and because of differences in the sprintf format. I'm glad that you found how to solve them.
FYI, I'm also the maintainer of pypy on Windows, and we support both MSVC (8, 9, 10) and mingw32.

> i encountered dozens of assumptions that "if sys.platform == 'win32'
> then you MUST be building using visual studio: f*** off with your
> attempt to compile this module using gcc".

There are not so many of them in distutils. Did you see them in third-party packages?
Code like this is wrong and should be changed, just like "#ifdef MS_WINDOWS" (=the platform) is not equivalent to "#ifdef _MSC_VER" (=the compiler)

fixed test case failure for 32-bit time_t

I touched the version field by mistake while trying to add myself to the nosy list. My sincere apologies for the SPAM to all involved.

Follow up updated patch to #3754

...py3k-20110520...:  with updates of integrated recently distutil2 (require patch from #3754 with same time-stamp).

> ...py3k-20110520...
For successive versions of the same patch, it’s best to remove the obsolete ones and add the new one, all with the same name.

> with updates of integrated recently distutil2
Your patch touches distutils, which is not acceptable.  Also, it contains a lot of FIXME comments.  I suggest you open reports for those bugs and make this one depend on them.  Please post a patch when it’s ready for review; in-development patches don’t really help.

follow recent changes to distutils and packaging modules and platform tests without os major version ...
(require python-py3k-20110825-CROSS.patch from #3754)

Please read msg136382 and take it into account.  Also, I suggest you wait a bit for packaging patches: GSoC projects need to be reviewed and integrated, which may change large parts of the codebase.

eric,

if you recall there was some discussion that it was acceptable to use distutils but *only* for python 2.N (on the basis that its use is so well entrenched that it would be impossible to force python2.N applications to start using distutils2), and i agreed wholeheartedly with you that the expectation to use distutils2 would be reasonable for python3.N

just as an aside: have all python 3.N packaging scripts, for all python-dev scripts *and* all 3rd party packages world-wide, been using distutils2 by default instead of distutils?

> if you recall there was some discussion that it was acceptable to use
> distutils but *only* for python 2.N
There was discussion, yes, but it was not decided to change our decision on the freeze: msg121097

> just as an aside: have all python 3.N packaging scripts, for all
> python-dev scripts *and* all 3rd party packages world-wide, been using
> distutils2 by default instead of distutils?
We are well aware that it is going to take years for the Python world to switch to the new standard, that’s why
1) projects can have a distutils-compatible setup.py and a distutils2-compatible setup.Cfg
2) pysetup can install distutils-based projects
3) I’m working on documentation to help people use distutils2

Although mainly focussed on MinGW, I've begun adding Darwin cross support. I've also built Darwin cross compilers. The latest Linux version can be found at:

http://mingw-and-ndk.googlecode.com/files/multiarch-darwin11-cctools127.2-gcc42-5666.3-llvmgcc42-2336.1-Linux-120531.tar.xz

Here's a run-down of what's in the patch:

configure.ac:
 Set MACHDEP=darwin when using darwin-cross compilers.
 Set DELIM to ; for MACHDEP=win.
 Switch from PC/getpathp.c to Modules/getpath.c for MinGW build.
  This gives us a posix like layout as needed by autotools projects
  such as GDB.
 Add MSYSVPATH as an env. var to configure (used in getpath.c):
  MSYS make uses a 'virtual' VPATH, but getpath.c uses
  GetModuleFileNameW (replacing \ with /). This allows the user to
  define the 'actual 'real' value. Note, it should contain / not \,
  (i.e. use ecactly what is printed by "pwd -W")
 Fixes to build with MinGW64 for REPARSE_DATA_BUFFER (needs defining)

Include/fileutils.h:
 Define Py_wstat for MinGW/MinGW64.

Include/osdefs.h:
 Define SEP to / and ALTSEP to \ for MinGW.

cygwinccompiler.py:
 Fix RE_VERSION in for handling MinGW64 compiler versioning.
 Ask gcc where ld is instead of requiring it to be passed in os.environ
 
plat-generic/regen:
 Allow passing in the CC so sysroots can be determined from -print-search-dirs
 If this CC passing is detected, use $CCINSTALL/include/stddef.h as netinet/in.h doesn't come with MinGW.

Moules/faulthandler.c:
 Avoid _set_abort_behavior on MinGW64 as it's broken.

Modules/getpath.c:
 Add support for MinGW.

Modules/socketmodule.c:
 Change case (to lower) of #include <MSTcpIP.h> for MinGW/MinGW64 cross.

Parser/metagrammar.c:
 Rename _PyParser_Grammar to _PyParser_MetaGrammar to avoid multiply defined symbols.

Python/pythonrun.c:
 Translate ALTSEP (\) to SEP (/) in Py_SetProgramName

setup.py:
 Add 'msi','cabinet','rpcrt4' as libs to link against for _msi module.
 Fix an error with self.get_platform() no longer being defined (changed to host_platform)

Best regards,

Ray Donnelly.

Thanks for your work Ray, but as your patch adds a feature to distutils it cannot be accepted.

Hi Éric,

Do you mean this bit?:

diff -urN a/Lib/distutils/cygwinccompiler.py b/Lib/distutils/cygwinccompiler.py
--- a/Lib/distutils/cygwinccompiler.py	2012-05-30 07:33:00.234438631 +0100
+++ b/Lib/distutils/cygwinccompiler.py	2012-05-30 07:33:03.320855811 +0100
@@ -392,7 +392,7 @@
         return (CONFIG_H_UNCERTAIN,
                 "couldn't read '%s': %s" % (fn, exc.strerror))
 
-RE_VERSION = re.compile(b'(\d+\.\d+(\.\d+)*)')
+RE_VERSION = re.compile(b'[\D\s]*(\d+\.\d+(\.\d+)*)[\D\s]*$')

It's hardly touching distutils and only the cygwinccompiler.py part (which I doubt is used very much anyway). It there not some extensive testsuite we can run patches against that would allow distutils changes?

I guess the very limited amount of changes I've made to distutils is academic though as it depends on two patches from Roumen Petrov which touch distutils a lot more.

I thought that the distutils-is-frozen rule was only for Python 2.x and that they would be allowed for 3.x? Is this not true?

Obviously making Python cross compile properly is a very desired feature (in this day and age I'd say cross-compilation for any major software is expected to be honest with you). How can we progress this task in a way that is acceptable? Please advise.

Best regards,

Ray.

Also include mingw-w64 fixes - check for structure REPARSE_DATA_BUFFER, use lower case names of some headers . set_abort_behavior. I prefer to exclude use of parse_off_t so that functionality to be to same. Use of relict alarm function is not excluded as one of the test will fail as test require SIGALARM support.

Correction: Use of relict alarm function is not excluded even failure in one of tests - the test is run if exist alarm function but require in addition SIGALARM attribute.

Ray , about python cygwin/mingw compiler classes exists other requests - use cygwin class instead unix compiler, remove relict checks  : nocygwin and etc. but without progress. 

About merge of getpath sources - issue is 9654.

Thanks for the details. I've got more fixes but I want to get the whole thing building (at least) before I post them.

> I thought that the distutils-is-frozen rule was only for Python 2.x
> and that they would be allowed for 3.x? Is this not true?
It is not.  The normal rule is that stable versions (like 2.7) get only bug fixes while new features go only to the development version (like 3.3 before it went into beta last Tuesday).  For distutils there is a special rule that prevents new features for all versions.  The only exceptions are side effects of changes in other parts of Python (for example __pycache__ directories in 3.2 or virtualenv in 3.3).

> Obviously making Python cross compile properly is a very desired feature (in this day and age
> I'd say cross-compilation for any major software is expected to be honest with you).
Well I for one am a unix user but not a C guy; I compile Python easily and don’t fully get the benefits of cross compilation.

distutils2 is the place to add such new features.  We wanted to release it in the standard library for 3.3 but it was not ready, so it should be in 3.4.

the cross build support is now updated for 3.3. so the mingw32 patches need an update. Not sure if they will go into 3.3, because there seem to be non-trivial changes for in Lib/distutils. Other self-contained changes probably should still go into 3.3.

> Not sure if they will go into 3.3, because there seem to be non-trivial changes for in Lib/distutils.
The feature freeze applies to all branches.  Even when 3.4 starts, the same rule that has been repeatedly explained for two years will apply: no new features in distutils.  Again, neither Tarek nor I are happy about that, but it was the decision taken by python-dev due to the big number of setup.py scripts depending on undocumented behavior, bugs and quirks.

> Other self-contained changes probably should still go into 3.3.
Not sure; we are in beta now, and cross-compilation does not look like an existing feature that is being fixed, but a new feature being added despite the beta status.

Eric.
please could could you revert PEP 405 as distutils is frozen or step down as It seems to me you don't flow distutils evolution.

Hi Mattias ,
I have ready to upload patches for 3.3.30b1 but after recent changes I will postpone upload . 
Part of already uploaded patch cover cygwin native build .
For instance
a)  --enable-new-dtags is only for elf binaries.
b) use of libpython$(LDVERSION), i.e. LD...) has to be reported again in another cygwin related issue 
c) .... etc?(?) ...

10x for you efforts to review, analyze, commit patches related to cross-build.

Eric
packaging seem to me a good  tool to replace distutils remaining compatible but unfortunately is removed from 3.3 :(

> distutils2 is the place to add such new features. 

you're not getting it.  you've just told both this
mingw32 project and also the new effort by ray that
they can go fuck themselves, because their efforts
are a total waste of time (as far as python 2 is
concerned).

why? because without patches to python 2 distutils, you're condemning
their work to total irrelevance.  why? because nobody in their
right mind is going to do what you suggest - "go educate people"
about using distutils2.  that's tens of thousands of package
maintainers, and you're expecting *two* people to do that job - 
you must be off your head if you think that's even remotely
realistic!  i can't quite understand why you would even make such
a recommendation, because it's so clearly unworkable and unrealistic.

now, if python2 was in any way going to die, be totally
irrelevant or otherwise be superceded at any time in
the next ten years i would in no way be so pissed at this
freeze, and would in no way bother to comment on this.
i don't like doing this sort of thing: it makes me look
like i'm some sort of fucking idiot - but that's tough luck
on me.  someone has to tell it straight, and truthfully say
the things that everyone else doesn't have the balls to say.

what has me particularly pissed off is that the freeze happened
years *AFTER* roumen created this initial patch.

also what has me particularly pissed off is that the freeze
happened because *other people* - not roumen, and not ray -
didn't do their job, either in checking that their modifications
to distutils would have an impact on any other ports, but more
than that, neither did anyone else in the python team.

why should ray and roumen's work be made insultingly irrelevant,
just because of mistakes made by *other people*?


by complete contrast, i absolutely agree with you 100% that in
NO WAY should any modifications to distutils in python3 sources
be EVER permitted.  forcing people to use distutils2
for *new* packages, as well as packages being transitioned to python3,
and documenting that fact (and the required procedure to be followed)
prominently on the python web site, is perfectly reasonable.

the reason why it is perfectly reasonable is because the developer
who is writing the new package (or transitioning the old one) is
*actively* taking the steps - THEY are taking steps - THEY are the
ones ACTIVELY doing the work.  intiated BY THEM.  people PLURAL.
tens of THOUSANDS of people.  and those people will read documentation,
and will ask questions on the mailing list, and will generally follow
the advice and guidance of the community.

contrast this with a completely unrealistic expectation that TWO
people must DEMAND of those exact same tens of thousands of python2
developers that "ohhh, you muuust use distutils2" and you can perhaps
appreciate why i am not exactly impressed.

of course, if the Python Software Foundation is offering to fund
these two developers with a campaign to educate the entire world of
python2 package maintainers with the demand that they all convert to
distutils2, and is willing to support them by creating a write-up
and a series of articles and other supporting documentation on the
python.org web site, then that is an entirely different matter.

> The feature freeze applies to all branches.  Even when 3.4 starts, the 
> same rule that has been repeatedly explained for two years will apply: 
> no new features in distutils.  Again, neither Tarek nor I are happy
> about that, but it was the decision taken by python-dev due to the big 
> number of setup.py scripts depending on undocumented behavior, bugs and 
> quirks.

then you use monkeypatching.  split the new functionality into
a separate module - ..../distutils/mingw.py - that is called by
a clear ONE LINE modification in distutils.py that CLEARLY cannot
have IN ANY WAY not one SINGLE effect on ANY OTHER python port.

that line would be something like:

    if sys.compiler == 'mingw32': import distutils.mingw

within that distutils.mingw it would monkey-patch distutils to
add itself into the distutils code.

the same technique could be used by ray to monkey-patch the
cygwin compiler by doing this:

distutils/specialportmaintainedbyray.py:

distutils.cygwincompiler.RE_VERSION = re.compile(b'[\D\s]*(\d+\.\d+(\.\d+)*)[\D\s]*$')

of course, for this to work, support for the ports are required
(identification of the ports) down at the sys.py and os.py level.
unless you expect these port maintainers to put in special 
monkeypatching into the c-code itself (which would of course be 
conditionally compiled and would in no way affect ANY other port of
python).

this would be perfectly reasonable.  the impact on other ports would
be zero.  consideration of the one line patch would take a few
minutes.

... and for those packages that depend on undocumented behaviour?
in the unlikely event that they ever get compiled for mingw32 or
ray's port, well, the person who compiled such a package would 
encounter an error, and would wonder why it was broken, and then
would get onto a mailing list and go "i have a problem, can anyone
help me".

chances are that the majority of packages would however compile
cleanly.

there's always a way.

For me, it's all about being able to build useful software. I like Python a lot, but my goal is and has always been cross compilation of gdb with Python support. To that end, if I have to maintain some out of tree patches for 2.7.3 then so be it. For 3.x things are looking better and I'm hopeful that we'll get everything merged at some point.

For anyone who's interested in my almost-final set of patches against the released tarballs for both 2.7.3 and 3.3.0b1 they're at:

http://mingw-and-ndk.googlecode.com/files/python-273-and-330b1-patches-WIP.7z

AFAIK, the only real change I'll be making is to the build script.

Once Matthias has finished merging Roumen's cross compilation work to 3, I'll rebase my 3.3.0b1 patches against that and open an issue for each patch. If mingw cross touches too much of distutils for people's liking then hopefully we can get mac osx cross merged and that'll at least mean that the infrastructure is cross-compilation friendly.

I'm not personally interested in Cygwin (it's too slow for my liking) and I've spent a lot of time patching and building software to avoid forcing people to use it (no offence meant to Cygwin people, I can see the value of it, but IMHO native is always best) .

All of my patches target mingw-w64 (and Mac OS X cross); I'll do my best to make sure that the Cygwin port still works though (and also mingw32).

fyi, the cross build changes are now checked in. Checked with an x86_64-linux-gnu to arm-linux-gnueabi cross build.  I don't plan to add anything more for the 3.3 release besides bug fixes.

Thanks Matthias, I might wait until b2 (or rc1) before I rebase my patches, just because I'm so unfamiliar with Mercurial.

May be patch 20120729 cannot be applied as my source tree contain a number of patches related to cross build - see msg166740  from issue 3754 .

Extract of 2012-10-04 is after patch set py3k-20121004-CROSS.tgz from issue #3754 .

I'm attempting to build Python-3.3.0 with mingw64. I get a minute or two into the build and fail with this error:

./Include/pythonrun.h:178:1: warning: function declaration isn't a prototype [-W
strict-prototypes]
./Modules/getpath.c: In function 'isfile':
./Modules/getpath.c:153:5: warning: implicit declaration of function '_Py_wstat'
 [-Wimplicit-function-declaration]
./Modules/getpath.c: In function 'find_env_config_value':
./Modules/getpath.c:298:17: error: too many arguments to function 'wcstok'
In file included from c:\projects\tools\osm\dev\mapnik\custom\environment\instal
l\mingw64\bin\../lib/gcc/x86_64-w64-mingw32/4.7.2/../../../../x86_64-w64-mingw32
/include/io.h:10:0,
                 from ./PC/pyconfig.h:68,
                 from ./Include/Python.h:8,
                 from ./Modules/getpath.c:3:

It seems some people here were using this patch against 3.3.0 at one point. Did you guys ever succeed? Once I've applied the patch, i can get a little further, but the build eventually fails again. Also, conftest continually fails to run during configure.

I'm using the following settings to get the build going:

export "CFLAGS=-I/mingw/include"
export "LDFLAGS=-L/mingw/lib"
export "CPPFLAGS=-I/mingw/include" 
export "CFLAGS=$CFLAGS -I./PC -DMS_WIN64 -D__MINGW32__"


#http://bugs.python.org/issue3871
#wget http://bugs.python.org/file27474/py3k-20121004-MINGW.patch
wget http://bugs.python.org/file26572/python-py3k-20120729-MINGW.patch

#patch -t -p1 < "py3k-20121004-MINGW.patch"
patch -t -p1 < python-py3k-20120729-MINGW.patch

./configure --host=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 --prefix=/mingw

make

If I patch before configure, I get the following exception when running ./configure:

bad word @INITSYS@ in @INITSYS@ posixmodule.c
mv: cannot stat `config.c': No such file or directory

After configure, is where it gets further, until it finally fails on:

c:\projects\tools\osm\dev\mapnik\custom\environment\install\mingw64\bin\../lib/g
cc/x86_64-w64-mingw32/4.7.2/../../../../x86_64-w64-mingw32/include/io.h:311:15:
note: previous declaration of 'chmod' was here
./Modules/posixmodule.c:262:32: error: unknown type name 'uid_t'
./Modules/posixmodule.c:262:39: error: unknown type name 'gid_t'

You can replicate my environment exactly by running the following deployment scripts from my project here:

https://github.com/onepremise/MinGW-AD64S/

Just open dos in admin mode and run setup.bat.

Any help is greatly appreciated. Thanks!

Jason Huntley wrote:
....
> patch -t -p1 < python-py3k-20120729-MINGW.patch
Please run autoheader (optional) and autoconf (required .
> ./configure --host=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 --prefix=/mingw
...

Finally! I was able to get all building on mingw64, gcc 4.7.2, and python 3.3. I had to take everybody's work here a little further, and I have produced a new patch which will build python33 using the latest minw64 compiler in an msys environment.

I cloned the 3.3 branch and applied the latest patch available here, py3k-20121004-MINGW.patch. I had to make quite a few more updates to get all the modules compiling with 4.7.2 and python find all the modules in the correct paths in an msys/mingw64 environment. I've compiled another patch, which addresses all the latest issues.

For all interested, run my setup script from the following location:

https://github.com/onepremise/MinGW-AD64S/

Just open dos in admin mode and run setup.bat.

This will build all the required dependencies and finally python 3.3.

Good work Jason,

I think it would be great if everyone could collaborate better with this stuff; I'm as guilty as the next person on that score though. I might have been able to save you some effort by pointing you at either:

https://github.com/mingwandroid/crucifixion-freedom/tree/master/patches/python/3.3.0 [Python cross and MinGW compilation]

or:

https://github.com/niXman/mingw-builds/tree/master/patches/Python-3.3.0 [MinGW toolchains including Python pretty-printing GDB]

..there might be some patches in there that are useful for you. Both these projects use basically the same patches, the 'source' of them is crucifixion-freedom.

Our patching strategies are more sane (many patches as opposed to a monolithic patch - in fact, just after Roumen's huge patch we apply some patches to remove large parts of it so we can re-introduce those independent fixes via separate patches).

Both crucifixion-freedom and mingw-builds also provide patches for 2.7.3 and support building tkinter and IDLE. The unique feature of crucifixion-freedom is fairly comprehensive cross compilation capability ({Linux,Windows,Mac}*{32bit,64bit}).

Given how intrusive these patches are, getting them merged into CPython is a bit of a pipe dream (I'd love to be wrong about that), so making it as easy as possible for the interested parties to maintain (forward port), build, test and package Python releases is probably the best we can hope for. If we can make the patches small and independently reviewable then maybe some parts could be merged.

Jason Huntley wrote:
> Jason Huntley added the comment:
>
> Finally! I was able to get all building on mingw64, gcc 4.7.2, and python 3.3. I had to take everybody's work here a little further, and I have produced a new patch which will build python33 using the latest minw64 compiler in an msys environment.
>
> I cloned the 3.3 branch and applied the latest patch available here, py3k-20121004-MINGW.patch. I had to make quite a few more updates to get all the modules compiling with 4.7.2 and python find all the modules in the correct paths in an msys/mingw64 environment. I've compiled another patch, which addresses all the latest issues.
Until now I did not change Mingw compiler class, just because is 
reported in a number of separate issues. I would like to disagree with 
'hard codded' compiler flags into python code. For instance 
-mms-bitfields is default for GCC 4.7+ . Also -pthread has to be 
detected by configure if is required for host platform.
Another point is mno-cygwin - reported in issue 12641. Unfortunately one 
of PSF project directors misread error message and issue remain unresolved.

Please provide more details for change in Lib/ntpath.py .

What is reasons to add more build-in modules (time, math, ...) ?

Why is updated PC/pyconfig.h ? This is so called by python community 
"posix-build" and pyconfig.h is generated by configure script. Did I 
miss something ?

File configure.ac contain so many unrelated changes - replace tabs with 
spaces. Please post as separate diff you changes.

Why is updated Lib/plat-generic/regen ?

Currently my private repository reintroduce support _sysconfigdata.py 
but this require some changes in PC/getpathp.c . I left current 
installation from makefile as is and after this with an extra script i 
move file for location same as primary build. This is reason do not 
include more search paths in this file. Issue for unification of python 
installation across platforms may be will be resolved with 14302 and 9654.

Roumen

Finally! I was able to get all building on mingw64, gcc 4.7.2, and python 3.3. I had to take everybody's work here a little further, and I have produced a new patch which will build python33 using the latest minw64 compiler in an msys environment.

I cloned the 3.3 branch and applied the latest patch available here, py3k-20121004-MINGW.patch. I had to make quite a few more updates to get all the modules compiling with 4.7.2 and python find all the modules in the correct paths in an msys/mingw64 environment. I've compiled another patch, which addresses all the latest issues.

For all interested, run my setup script from the following location:

https://github.com/onepremise/MinGW-AD64S/

Just open dos in admin mode and run setup.bat.

This will build all the required dependencies and finally python 3.3.

Hey Guys,

I've furthered the progress of the patch I've been working on. THis patch builds Python3.3 in mingw64 & msys using gcc 4.7.2 compiler. This patch can also enable and build the following modules:

ctypes
bz2
multiprocessing
sqlite3
ssl
pyexpat
zlib

Of course, there are many others, but these were some of the problem modules I encounted previously.

Partial success running the python tests. Many of the basic tests pass, I still have many to resolve. I have one thread test failing I need to resolve as it's preventing from all the tests running. Regardless, seems like some progress.

>>I think it would be great if everyone could collaborate better with this stuff;

I agree; I would really like for us to make this a joint effort. However, I'm noticing a ton of descrepancies with these patches. Some are removing pthreads, others are adding them back. Some patches are ignoring the PC/pyconfig while others are trying to generate a custom mingw version. It seems some are also compiling and using the mingw version outside of the msys shell, which I'm not certain of the advantages there when u can just use the mscv version. I think we need to take a minute to agree on some specific goals and criteria this mingw port should be used for. Just to kick things off, here are some of the requirements which I think we should aim for.

* The build must be reproducible in the mingw msys shell, in the default environment described on the mingw64 site, also included some other good doc references:
  * https://sourceforge.net/apps/trac/mingw-w64/wiki/MSYS
  * https://sourceforge.net/apps/trac/mingw-w64/wiki/Cross%20Win32%20and%20Win64%20compiler
  * http://www.gaia-gis.it/gaia-sins/mingw64_how_to.html#env
  * These documented steps have also been automated by the sripts here https://github.com/onepremise/MinGW-AD64S
  * Currently, I have the mingw build using the PC platform specified config headers and source.
    * too many issues reside around enabling posix threads as the pthread_t identifier in mingw doesn't match existing py thread framework returning longs.
    * As a result, the patch I provided builds with the ntthread implementation.
    * In addition, PC/pyconfig.h and PC/getpath.c are being used with slight modifications for __MINGW32__
  * Initial python mingw64 deployment layout, version identifiers should follow existing scheme, "3.3" not 33:
    * should deploy libraries to /mingw/lib, libpython3.3.a, Format: libpython3.3.dll, libpython3.3.dll.a.
    * executable binaries should exist in /mingw/bin. Format: python.exe, python3.3.exe.
    * The python script library deploys to /mingw/lib/Python3.3.
      * /mingw/lib/Python3.3/lib-dynload
      * /mingw/lib/Python3.3/site-packates
    * Headers located /mingw/include/Python3.3
  * Having Python build isn't satisfactory enough to promote a patch, patches should not cause new test failures or regressions. Preferably they should resolve core compilation issues and module compilation issues, while not creating new test failures and preferrably address/fix failing tests. (python -m test)
  * covered on the dev-guilde, http://docs.python.org/devguide/
  
Please feel free to contriube, update, remove, discuss etc...
  
>>https://github.com/mingwandroid/crucifixion-freedom/tree/master/patches/python/3.3.0 [Python cross and MinGW compilation]
>>https://github.com/niXman/mingw-builds/tree/master/patches/Python-3.3.0 [MinGW toolchains including Python pretty-printing GDB]

I tried some of these patches, but they aren't very organinzed. I really need some docemntaiton to better understand what each patch does and which pairs go together and which don't. Wiki over there is blank.

>>in fact, just after Roumen's huge patch we apply some patches to remove large parts 
>>of it so we can re-introduce those independent fixes via separate patches).

I had to do the same as well. There were regressions in some of these patches causing the python tests to fail and prevent the python setup.py script from running.

>>Our patching strategies are more sane (many patches as opposed to a monolithic patch - 
>>Given how intrusive these patches are, getting them merged into CPython is a bit of a 
>>pipe dream (I'd love to be wrong about that), so making it as easy as possible 
>>for the interested parties to maintain (forward port), build, test and package 
>>Python  releases is probably the best we can hope for. If we can make the 
>>patchessmall and independently reviewable then maybe some parts could be 
>>merged.

I prefer this as well. It will make code reviewing easier. Otherwise, some kind of staging location would be useful. I'm fine using rietveld, the tracker. We just need to name the patches better, including prefix/dates or provide descriptions in the comments below. Right now it's hard to tell which patches to apply just from reading through the comments

>>I would like to disagree with 'hard codded' compiler flags into python 
>>code. For instance -mms-bitfields is default for GCC 4.7+ .

They have been removed in this patch

>>Please provide more details for change in Lib/ntpath.py .

Mingw msys was having issues processing the paths provided by ntpath.py. The paths being passed around have a seldom used format, which works in windows but not mingw, which is providing a drive without a path separator: C:my/path. This should be provided as C:/my/path. Either backslash or forward slash can be used, mingw doesn't care. Mingw just needs the path separator after the drive letter to make sense of the path.

>>What is reasons to add more build-in modules (time, math, ...) ?

python.exe setup.py build --compiler=mingw32 fails without the time and math modules enabled. Somewhere in the python script, they were being imported.

>> Why is updated PC/pyconfig.h ? This is so called by python community 

Because mingw uses much of the same existing framworks provided by system32.

>>"posix-build" and pyconfig.h is generated by configure script.

Yes i noticed this too, but as I started working with configure.ac, Makefile.pre.in, and pyconfig.h, I noticed I was basically recreating the pyconfig.h file used under PC. I also noticed others using this as well in their own ports. I imagine it would be better to get the configure scripts detecting and setting evironment varialbles correctly. However, as of right now, it's far from doing so. It was quicker to just use the existing PC file and add a few defines for MINGW32.

>>File configure.ac contain so many unrelated changes - replace tabs with 
>>spaces. Please post as separate diff you changes.

I don't know what to say about the configure.ac. The file provided by the release, has both tabs in spaces as well. Whatever the standard is, it's easy to fix.

>>Why is updated Lib/plat-generic/regen ?

Because "/usr/include/netinet/in.h" doesn't exist in the mingw64 msys environment. This script fails in mingw. You didn't notice?

>>Currently my private repository reintroduce support _sysconfigdata.py 
>>but this require some changes in PC/getpathp.c . I left current 
>>installation from makefile as is and after this with an extra script i 
>>move file for location same as primary build. 

python won't run in mys without the proper search paths being referenced in getpathp.c. The posixs getpathp.c seem to have issues well, I might have been able to get it to work if I tested more, but It was easier to just modify the PC getpathp and get everyting working.

some random comments about py3k-20121004-MINGW.patch:

- Modules/_ctypes/libffi_msvc/win32.S
  Please can you get rid of libffi_msvc and use libffi?
  afaics, libffi has support for mingw32.

- there seem to be chunks which are unrelated to mingw, like:

@@ -830,15 +926,18 @@ class PyBuildExt(build_ext):
             if have_usable_openssl:
                 # The _hashlib module wraps optimized implementations
                 # of hash functions from the OpenSSL library.
+                # NOTE: _hashlib require only OpenSSL crypto library !
                 exts.append( Extension('_hashlib', ['_hashopenssl.c'],
                                        depends = ['hashlib.h'],
                                        include_dirs = ssl_incs,
                                        library_dirs = ssl_libs,
-                                       libraries = ['ssl', 'crypto']) )
+                                       libraries = ['crypto']) )

please file separate issues.


- why setup_info.in. looks like something which could be done with get_config_var.

- why re-reading files in setup.py, and grepping these for config options?


I think the patch would benefit from splitting it up into several self-contained chunks.

did you try to do builds for windows and linux after this patch was applied?

When I say “our patches” I mean mine and Alexey Pavlov’s jointly maintained patch-set.

I hope you don’t mind that I find you saying:

"I tried some of these patches, but they aren't very organinzed. I really need some docemntaiton to better understand what each patch does and which pairs go together and which don't."

...and then continuing to present a single uber-patch to be mildly funny/ironic? You've already accepted that you should split them up and remove the needless formatting changes to make them more reviewable so I won't labour on this point.

"ctypes
 bz2
 multiprocessing
 sqlite3
 ssl
 pyexpat
 zlib"

Likewise for ours, but also curses, curses_panel, readline, TKinter, IDLE (and probably more besides) for both Win32, Win64, Mac OS X 32bit and 64bit.

The main idea behind our patches is (mostly in common with all patches) that they:
1. Are as atomic as possible in their operation (which as few interdependencies as necessary).
2. Are applied in an obvious ordering.
3. Are named sensibly according to what they do.
4. Build upon - and can reuse updates to - Roumen's uber-patch (and also clean it up a bit; let’s face it Roumen did a lot of the hard work and everyone else is trying to finish it off so we can build releases from it).
5. Enable comprehensive cross-compilation.

Given the name of each patch and the fact that they do one thing (the novel ones at least) and that by-and-large they fit on a few screens, I had hoped that they were usefully descriptive.

To explain which ones go together:
The numbering skips 5 each time for each independent bug-fix or feature - to allow space for later, related modifications. Therefore, where the numbering changes monotonically, the patches are related. This only happens twice, once to address the problem caused by Roumen providing an uber-patch and once due to a block of fixes all related to curses (ncurses and PDCurses are both supported). I assume my addressing of Roumen's uber-patch is what led you to write:

"However, I'm noticing a ton of descrepancies with these patches. Some are removing pthreads, others are adding them back."

There is logic to what you describe as a ton of discrepancies:
Roumen's patch is included in it's entirety as 0030-py3k-20121004-MINGW.patch, then I remove one feature of that patch that needed fixing (threading changes) with 0031-py3k-20121004-MINGW-removal-of-pthread-patch.patch, then I add a patch that provides my newer fix, 0032-py3k-20121004-MINGW-ntthreads.patch. The exact same thing happens with the libffi changes. The pthreads stuff needs fixing due to libwinpthread in MinGW-w64 and libffi is broken when targetting a 64bit Windows host.

My reason for doing this is so that I can directly drop updated versions from Roumen in over 0030 and it will (in theory at least) still work. He can continue to use a uber-patch if he wishes (though I don't think he does this for "new" patches). Also by taking this approach, my patch for the threading issues has no dependencies at all on Roumen’s patch, since a previous patch “handled” the dependency by removing that part of it.

Everything you said about build environment and the test-suite I agree with. My crucifixion-freedom project includes a .vbs to setup the most basic MSYS and MinGW-w64 environment: https://raw.github.com/mingwandroid/crucifixion-freedom/master/scripts/windows/BootstrapMinGW64.vbs
...this is essential to prevent spending time chasing down issues due to discrepancies between build environments.

However, we must go further and add that the patches *cannot* break any other native or cross-compilation, which - as I think Matthias is alluding to - is probably not the case with your patch. This issue is called "cross and native build of python for mingw* hosts", so that your patch possibly breaks builds on build-machine is unfortunate (especially so as a build-machine Python is required to run setup.py as part of the cross-build procedure). Even if it doesn't break building for the build-machine it breaks building the cross-MinGW-w64. Each occurrence of "/mingw" in your patch is probably a part of this problem (you can’t expect cross build systems to be be able to be rooted to /mingw, often you’ll find ~/mingw or /tmp/mingw)

You said, "It seems some are also compiling and using the mingw version outside of the msys shell, which I'm not certain of the advantages there when u can just use the mscv version", I’m not entirely sure what you mean here; if compiling MinGW Python on a Windows system with my patches, MSYS is 100% a requirement (cmd.exe will never run the configure script). I can only guess this relates to the mingw_posix2win_path stuff in your patch. Forgive me if I’m wrong, but it seems that you are conflating MSYS and MinGW when they are not related (except by frequent proximity). Without wanting to point out the obvious, MinGW-w64 is a sysroot and compiler toolchain for making native Windows programs and MSYS is a minimal Posix shell capable of running GNU Autotools. I think a result of this conflation is that a Python built using your patch is not properly usable outside of the MSYS shell (i.e. it will incorrectly transform paths), whereas using our patch-set, the built Python is fully compatible with the MSYS shell and cmd.exe (and whatever else; Wine I guess?).

You likely already know this, but there are two types programs that run on the MSYS shell, MSYS executables built with msys-gcc that link to the msys dll (which then does file path translation when you call any file IO functions) and native Windows executables (for which, when run under MSYS, MSYS’s bash.exe transforms the input arguments before calling the executable when it detects that msys dll is *not* used by that executable). A MinGW/MSYS environment consists of a mix of both, but programs built *with* it should be native Windows executables only (unless you are involved in hacking on the MSYS/MSYS2 projects that is). Your patch makes something in-between. I think this confusion is the reason why you wrote "u can just use the mscv version", please read again the name of this issue. I fail to see how adding a restriction so that MinGW Python can only work properly under MSYS can be argued as anything other than a bug (if you want to build an MSYS Python then that’s a different issue altogether - you’ll end up wrestling with GCC 3.3). Also, with regards to MSVC, given a choice, for philosophical reasons I'll always use software built predominantly with FLOSS software. For sure, as you say, our patches also allow building MinGW Python outside of the MSYS shell, specifically, you can use a Linux or Mac OS X bourne-compatible shell (good luck trying to get recent MinGW cross-compilers for Darwin though). Again, this is a feature and not a problem.

IMHO, updating these patches to track the latest Python is a pointless goal. The only reason I can see to do this would be to make it easy for a reviewer to merge it with the CPython mercurial repository, and realistically, these patches/patch-sets are not going to be merged wholesale. As I said before, the only hope is that bit-by-bit, some small fixes get merged and the patch-set is reduced over time and made more manageable [1]. Instead, I think it makes more sense to target the latest released versions from each branch as that way we’re not chasing a moving target (in particular, we’d only have to update it every-so-often, and if we enforced a disciplined usage of the test-suite then regressions caused by our patches would not be intermixed with ones caused by changes to CPython repository).

To recap, the features of our patch-set are:
1. Readable and reviewable.
2. Doesn't break builds for Cygwin or (likely) any other platform.
3. Fixes cross compilation (for me this is hugely important).
4. Can build 64bit MinGW Python.
5. Works outside of MSYS without any path assumptions or hacks.
6. Attempts to minimize impact to distutils (monkey patching instead of direct modification where possible - this is the most contentious part of Python, patches-wise).
7. Can co-exist with Windows Python due to different dll naming.
8. Can be installed to any prefix, using a Posix installation layout rather than the Windows one (“$prefix/bin”, “$prefix/include”, “$prefix/lib” and “$prefix/share”)
9. Can subsequently be relocated to anywhere the user wishes.
10. Is well tested. Python's built with the 2.7.x version of this patch-set have been in use by many people for well over a year now - to provide pretty-printer support for GDB, without any reported issues.
11. As well as cross/MSYS compiling to/on Windows, these patches allow Darwin Python to be cross-compiled.

I hope you will consider collaborating. If so then I will set aside some time to add descriptions to each of the patch files; otherwise, if you just split your patches up into a series of atomic fixes instead then I’ll examine those patches to see if there’s anything I can use.

As usual, for anyone who wants to get involved in this effort, you can send patches and/or suggestions to me at mingw.android at gmail.com or reply to this issue.

[1] http://hg.python.org/cpython/rev/c0370730b364

> IMHO, updating these patches to track the latest Python is a pointless goal.

sorry, no. it's the *only* way to get these patches upstream. The mingw patches will never see the light of the 3.3 branch. So the best thing to do is to actively submit the patches for trunk.

If you do want to maintain your patches, then split your set of patches into those which are upstream (and then pack-ported), and which are not yet upstream.

> However, we must go further and add that the patches *cannot* break any
> other native or cross-compilation, which - as I think Matthias is
> alluding to - is probably not the case with your patch. This issue is
> called "cross and native build of python for mingw* hosts",

no, I just didn't check. And I probably won't check. However it's more likely that your patches are accepted if e.g. it doesn't break the normal Windows build and an unrelated build, e.g. for a Linux target.

Maybe you could even separate the patches into a patch supporting a native mingw build and a cmingw ross build?

Re: basing the patches against the latest master branch or targeting released versions, I wasn't clear enough about my thinking.

For sure, when trying to get any patches merged, the submitted patch must be re-based (forward ported) and tested against the master branch. 

But from the perspective of having a project that people can reliably expect to build a working MinGW-w64 Python from (until that elusive fully-merged-day comes), having patches based on the latest released versions seems to me to be the best thing to do (for that goal of course)

I'll keep forward porting these patches to each new release, and at those times I'll also submit some of them as issues to bugs.python.org as that's the best way for me to manage my limited spare time.

Do you know what the situation is with packaging/distutils2? I think our next good merge opportunity for the bulk of these patches will be when we can target that instead of distutils.

> Maybe you could even separate the patches into a patch supporting a native mingw build and a cmingw ross build?

I'm already doing that:
https://github.com/mingwandroid/crucifixion-freedom/tree/master/patches/python/3.3.0

0000 and 0005 are now merged with upstream. 0000 you merged today, 0005 was merged also (mostly via Roumen's patches which 2/3 of my patch duplicated). At this stage, the framework for cross compiling Python is mostly merged; other than 0065-cross-dont-add-multiarch-paths-if-cross-compiling.patch

The stumbling block for me is that there's no working example of cross-compilation in CPython so it will surely rot. This was what I tried to fix with my next patch, 0010-cross-darwin-feature.patch. I went for darwin instead of MinGW as the amount of changes needed is tiny but it was met with some resistance and hasn't seen any activity for 3 months:

http://bugs.python.org/issue16291

I should also clarify, by cross compilation I'm specifically excluding GNU-Linux/GNU-Linux combinations such as x86-GNU-Linux/arm-GNU-Linux as they're not as exhaustive a test-case as cross-compiling between different OSes.

> Maybe you could even separate the patches into a patch supporting a native mingw build and a cmingw ross build?

Not sure if that was directed at me as my set of patches are logically split up I think. Ones with 'mingw' in the name relate to mingw (either cross or native), ones with 'msys' in the name relate to building mingw natively and ones with 'cross' in the name relate to general cross compilation matters; ones with 'hack' in the name generally need improving, rewriting or removing.

Matthias Klose wrote:
>
> Matthias Klose added the comment:
>
> some random comments about py3k-20121004-MINGW.patch:
>
> - Modules/_ctypes/libffi_msvc/win32.S
>    Please can you get rid of libffi_msvc and use libffi?
>    afaics, libffi has support for mingw32.

No python still use custom hack for MSVC.  win32.S include this hack and 
could be compiled with gcc.

> - there seem to be chunks which are unrelated to mingw, like:
>
> @@ -830,15 +926,18 @@ class PyBuildExt(build_ext):
>               if have_usable_openssl:
>                   # The _hashlib module wraps optimized implementations
>                   # of hash functions from the OpenSSL library.
> +                # NOTE: _hashlib require only OpenSSL crypto library !
>                   exts.append( Extension('_hashlib', ['_hashopenssl.c'],
>                                          depends = ['hashlib.h'],
>                                          include_dirs = ssl_incs,
>                                          library_dirs = ssl_libs,
> -                                       libraries = ['ssl', 'crypto']) )
> +                                       libraries = ['crypto']) )
>
> please file separate issues.
>
>
> - why setup_info.in. looks like something which could be done with get_config_var.

Yes and no.  get_config_var will use makefile variables (indirectly from 
sysconfigdata).
If you add a make macro(variable) then it will be installed.
Setup info is designed to communicate from configure to setup.py without 
to expose data to installation.

> - why re-reading files in setup.py, and grepping these for config options?

For historical reasons. Before other commits to start to look info some 
generated by autotool files.

>
> I think the patch would benefit from splitting it up into several self-contained chunks.

May be.
I don't know how to group split. If is based on functionality patch 
order will be important as more then one patch will update "near lines" -
for instance pyport.h is difficult to split :(.
cygwinccompiler.py could be rewritten as separate issue, already opened.

> did you try to do builds for windows and linux after this patch was applied?

Title is correct. Hosts include native linux, cygwin, mingw and cross 
linux->mingw, linux->android ( not published ). All from one and the 
same source tree.

Roumen

This discussion got very long and it's not clear (to me at least) what the status of the patch is and whether it has been accepted for inclusion.
Maybe it makes sense to bring this up to python-dev mailing list instead of keep stressing this thread?

Roumen, I think it would be really great if you could split py3k-20121004-MINGW.patch up into separate bits.

The pthread stuff and libffi stuff being two obvious candidates for atomic parts.

As patch 0005-CROSS-revert-issue13150-i.e.-python-solution-with-_s.patch (CROSS-revert issue13150, i.e. python solution with _sysconfigdata.py instead Makefile) from issue3754 is now obsolete by issue 13547,  14774,  16342, 15298 it is save to switch  sysconfigdata with proposed changes in PC/getpathp.c .

the last time this was brought up on python-dev the opinions of the primary python developers was made very very clear: anything that is not written by them is treated with extreme hostile and predudicial contempt.

what i mean by that is that the view of the primary python developers is that when they make decisions to decrease their workload, other people are *NOT* welcome to question those decisions.  [i questioned their decisions; all bugreports were terminated with excuses without good technical justification, and when i questioned those, access to the bugtracker was terminated].

roumen's work should have been encouraged and welcomed when it was first initiated - FIVE years ago.

the best bet therefore is to go over the heads of the primary python developers and to approach the Python Software Foundation's Board of Directors.  they are required to enact the PSF Charter in the best interests of the Python Programming Language.  what the existing primary python developers can or cannot cope with is of secondary consequence for the Board.

the main concern of the primary python developers was that they would be landed with yet another platform to support, and, as they are struggling to cope with the existing ones, they made - and make - absolutely any excuse that they can, regardless of actual merit, to ensure that no other platforms are added.

roumen, my opinion is that you have demonstrated over the past five years that you are committed to ensuring that python gets mingw support, and that you yourself are the best person to become the maintainer of python-mingw.  you are, already, de-facto, its maintainer.

answering some other points:

* python-mingw actually requires an entire new platform (built in to sys and os at a fundamental level).  neither the platform rules for cygwin, nor the platform rules for gcc, *nor* the platform rules for win32 correctly apply.  the MSVC platform rules should also be obviously understood to be useless.  mingw uses gcc, but it is gcc for win32: it therefore falls through *all* the existing platforms and requires its own separate class in distutils.

* the autoconf-generated pyconfig.h is so insanely slow to generate under mingw (any configure script running under MSYS is insanely slow) that it is virtually impossible to do any development.  also, the hard-coded config header file for win32 is also useless: firstly mingw doesn't have all the features of win32 (the headers are reverse-engineered and so are incomplete).  secondly, the hard-coded config header file for win32 has MSVC-specific details in it that make it useless.

* distutils was frozen because several immature developers made improperly tested commits that caused massive disruption.  rather than put in proper review procedures, the primary python developers decided to terminate all possibility of ever adding in a new platform - ever - and began distutils2.  this situation has to be reversed.  mingw *requires* that distutils have support for the mingw compiler.  a way to achieve this is to add separate files into distutils (which cannot be questioned as to their effect on existing files in distutils, because they are separate and therefore have zero effect), and then, once those files have been added and accepted, create a *ONE* line patch to distutils which pulls in the new mingw distutils functionality.  a one-line patch cannot really be argued with, esp. if it is only an "include".

roumen and others: i recommend that you approach the Python Software Foundation and ask that this project be sponsored for 3 months and/or until it is completed and the patches are in.

Please do not post to this thread.
Follow python bugs list for result of patch-split. I will try to post parts of this patch to already opened issues or to open new one if necessary.

As basis will be used py3k-20121004-MINGW.patch with following main changes:
- prefer posix installation scheme
- use of system libffi

Lets keep issue open as reference until is reached usable level.

Great Roumen, please split into new issues then make a new overall tracking issue, then this one can be left for historians and archaeologists.

For Posix installation scheme I've got:

https://github.com/niXman/mingw-builds/blob/master/patches/Python-3.3.0/0050-mingw-sysconfig-like-posix.patch

It needs some changes to allow for normal NT scheme when not using MinGW though.

For system libffi there's:

https://github.com/niXman/mingw-builds/blob/master/patches/Python-3.3.0/0035-mingw-system-libffi.patch

When you make your new patches I will base a new set of patches on them.

> Ray Donnelly added the comment:
> Next time there's a release of Python 3, I'll rebase my patches against that.

sorry, this is the wrong attitude, if you want mingw support to go upstream. fetch tip/trunk, re-apply your patches, and submit them.

now closing/rejecting this issue. See
http://mail.python.org/pipermail/python-dev/2013-January/123774.html
for the discussion.

For protocol - as requested patches attached to this issue are split ans posted as separate issues.

Please follow those listed below:
issue18654 - modernize mingw&cygwin compiler classes (new)
issue17605 - mingw-meta: build interpreter core
issue18653 - mingw-meta: build core modules
issue19245 - mingw-meta: install (new)

Test cases are not covered. Patches will be posted later, if above are resolved.