Author jhuntley
Recipients BreamoreBoy, LRN, Ray.Donnelly, WhiteTiger, alesko, amaury.forgeotdarc, davidfraser, doko, eric.araujo, georg.brandl, giampaolo.rodola, jhuntley, kalev, lkcl, rpetrov, rschoon.old, schmir, scott.tsai, tarek, tshepang
Date 2013-01-11.20:55:09
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1357937723.44.0.0670920729491.issue3871@psf.upfronthosting.co.za>
In-reply-to
Content
Hey Guys,

I've furthered the progress of the patch I've been working on. THis patch builds Python3.3 in mingw64 & msys using gcc 4.7.2 compiler. This patch can also enable and build the following modules:

ctypes
bz2
multiprocessing
sqlite3
ssl
pyexpat
zlib

Of course, there are many others, but these were some of the problem modules I encounted previously.

Partial success running the python tests. Many of the basic tests pass, I still have many to resolve. I have one thread test failing I need to resolve as it's preventing from all the tests running. Regardless, seems like some progress.

>>I think it would be great if everyone could collaborate better with this stuff;

I agree; I would really like for us to make this a joint effort. However, I'm noticing a ton of descrepancies with these patches. Some are removing pthreads, others are adding them back. Some patches are ignoring the PC/pyconfig while others are trying to generate a custom mingw version. It seems some are also compiling and using the mingw version outside of the msys shell, which I'm not certain of the advantages there when u can just use the mscv version. I think we need to take a minute to agree on some specific goals and criteria this mingw port should be used for. Just to kick things off, here are some of the requirements which I think we should aim for.

* The build must be reproducible in the mingw msys shell, in the default environment described on the mingw64 site, also included some other good doc references:
  * https://sourceforge.net/apps/trac/mingw-w64/wiki/MSYS
  * https://sourceforge.net/apps/trac/mingw-w64/wiki/Cross%20Win32%20and%20Win64%20compiler
  * http://www.gaia-gis.it/gaia-sins/mingw64_how_to.html#env
  * These documented steps have also been automated by the sripts here https://github.com/onepremise/MinGW-AD64S
  * Currently, I have the mingw build using the PC platform specified config headers and source.
    * too many issues reside around enabling posix threads as the pthread_t identifier in mingw doesn't match existing py thread framework returning longs.
    * As a result, the patch I provided builds with the ntthread implementation.
    * In addition, PC/pyconfig.h and PC/getpath.c are being used with slight modifications for __MINGW32__
  * Initial python mingw64 deployment layout, version identifiers should follow existing scheme, "3.3" not 33:
    * should deploy libraries to /mingw/lib, libpython3.3.a, Format: libpython3.3.dll, libpython3.3.dll.a.
    * executable binaries should exist in /mingw/bin. Format: python.exe, python3.3.exe.
    * The python script library deploys to /mingw/lib/Python3.3.
      * /mingw/lib/Python3.3/lib-dynload
      * /mingw/lib/Python3.3/site-packates
    * Headers located /mingw/include/Python3.3
  * Having Python build isn't satisfactory enough to promote a patch, patches should not cause new test failures or regressions. Preferably they should resolve core compilation issues and module compilation issues, while not creating new test failures and preferrably address/fix failing tests. (python -m test)
  * covered on the dev-guilde, http://docs.python.org/devguide/
  
Please feel free to contriube, update, remove, discuss etc...
  
>>https://github.com/mingwandroid/crucifixion-freedom/tree/master/patches/python/3.3.0 [Python cross and MinGW compilation]
>>https://github.com/niXman/mingw-builds/tree/master/patches/Python-3.3.0 [MinGW toolchains including Python pretty-printing GDB]

I tried some of these patches, but they aren't very organinzed. I really need some docemntaiton to better understand what each patch does and which pairs go together and which don't. Wiki over there is blank.

>>in fact, just after Roumen's huge patch we apply some patches to remove large parts 
>>of it so we can re-introduce those independent fixes via separate patches).

I had to do the same as well. There were regressions in some of these patches causing the python tests to fail and prevent the python setup.py script from running.

>>Our patching strategies are more sane (many patches as opposed to a monolithic patch - 
>>Given how intrusive these patches are, getting them merged into CPython is a bit of a 
>>pipe dream (I'd love to be wrong about that), so making it as easy as possible 
>>for the interested parties to maintain (forward port), build, test and package 
>>Python  releases is probably the best we can hope for. If we can make the 
>>patchessmall and independently reviewable then maybe some parts could be 
>>merged.

I prefer this as well. It will make code reviewing easier. Otherwise, some kind of staging location would be useful. I'm fine using rietveld, the tracker. We just need to name the patches better, including prefix/dates or provide descriptions in the comments below. Right now it's hard to tell which patches to apply just from reading through the comments

>>I would like to disagree with 'hard codded' compiler flags into python 
>>code. For instance -mms-bitfields is default for GCC 4.7+ .

They have been removed in this patch

>>Please provide more details for change in Lib/ntpath.py .

Mingw msys was having issues processing the paths provided by ntpath.py. The paths being passed around have a seldom used format, which works in windows but not mingw, which is providing a drive without a path separator: C:my/path. This should be provided as C:/my/path. Either backslash or forward slash can be used, mingw doesn't care. Mingw just needs the path separator after the drive letter to make sense of the path.

>>What is reasons to add more build-in modules (time, math, ...) ?

python.exe setup.py build --compiler=mingw32 fails without the time and math modules enabled. Somewhere in the python script, they were being imported.

>> Why is updated PC/pyconfig.h ? This is so called by python community 

Because mingw uses much of the same existing framworks provided by system32.

>>"posix-build" and pyconfig.h is generated by configure script.

Yes i noticed this too, but as I started working with configure.ac, Makefile.pre.in, and pyconfig.h, I noticed I was basically recreating the pyconfig.h file used under PC. I also noticed others using this as well in their own ports. I imagine it would be better to get the configure scripts detecting and setting evironment varialbles correctly. However, as of right now, it's far from doing so. It was quicker to just use the existing PC file and add a few defines for MINGW32.

>>File configure.ac contain so many unrelated changes - replace tabs with 
>>spaces. Please post as separate diff you changes.

I don't know what to say about the configure.ac. The file provided by the release, has both tabs in spaces as well. Whatever the standard is, it's easy to fix.

>>Why is updated Lib/plat-generic/regen ?

Because "/usr/include/netinet/in.h" doesn't exist in the mingw64 msys environment. This script fails in mingw. You didn't notice?

>>Currently my private repository reintroduce support _sysconfigdata.py 
>>but this require some changes in PC/getpathp.c . I left current 
>>installation from makefile as is and after this with an extra script i 
>>move file for location same as primary build. 

python won't run in mys without the proper search paths being referenced in getpathp.c. The posixs getpathp.c seem to have issues well, I might have been able to get it to work if I tested more, but It was easier to just modify the PC getpathp and get everyting working.
History
Date User Action Args
2013-01-11 20:55:27jhuntleysetrecipients: + jhuntley, georg.brandl, doko, lkcl, amaury.forgeotdarc, davidfraser, giampaolo.rodola, schmir, scott.tsai, tarek, eric.araujo, rpetrov, rschoon.old, WhiteTiger, BreamoreBoy, LRN, alesko, tshepang, kalev, Ray.Donnelly
2013-01-11 20:55:23jhuntleysetmessageid: <1357937723.44.0.0670920729491.issue3871@psf.upfronthosting.co.za>
2013-01-11 20:55:23jhuntleylinkissue3871 messages
2013-01-11 20:55:11jhuntleycreate