classification
Title: cross-compilation support for python build
Type: enhancement Stage: patch review
Components: Build, Distutils2 Versions: Python 3.3
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: tarek Nosy List: Arfrever, BreamoreBoy, David.Kern, Greg.Hellings, WhiteTiger, alexis, eric.araujo, kalev, marqvar, rpetrov, tarek, wrobell
Priority: normal Keywords: needs review, patch

Created on 2008-09-01 21:49 by rpetrov, last changed 2011-11-09 16:44 by marqvar.

Files
File name Uploaded Description Edit
python-trunk-20100516-CROSS.patch rpetrov, 2010-05-16 12:09
python-trunk-20100626-CROSS.patch rpetrov, 2010-06-26 19:43
python-trunk-20100824-CROSS.patch rpetrov, 2010-08-24 11:22
python-2.7-20101112-CROSS.patch rpetrov, 2010-11-12 21:02
python-py3k-20101112-CROSS.patch rpetrov, 2010-11-12 21:03
python-2.7-20110126-CROSS.patch rpetrov, 2011-01-26 21:02
python-py3k-20110126-CROSS.patch rpetrov, 2011-01-26 21:05
python-py3k-20110407-CROSS.patch rpetrov, 2011-04-07 20:26 review
python-py3k-20110520-CROSS.patch rpetrov, 2011-05-19 22:07 review
python-py3k-20110825-CROSS.patch rpetrov, 2011-08-24 21:31 review
Messages (20)
msg72299 - (view) Author: Roumen Petrov (rpetrov) Date: 2008-09-01 21:49
This is minimal patch that add basic cross-compilation possibilities for
python build (configure script).

The patch add macro AC_CANONICAL_HOST.
This macro require files config.guess, config.sub. The patch don't
include them. You may obtain them from GNU automake tarbal.

As result of macro new variable $host ("host triplet":=cpu-verdor-os) is
used to detect so called "host system".

Since this is basic patch, detection of build system in native builds
based on $ac_sys_system and/or $ac_sys_release isn't replaced. This
detection isn't appropriate for cross-compilation environment as contain
values for "build system" and has to be replaces in addition by future
patches.


Also the patch posted in http://bugs.python.org/issue3718 (about
environment variable MACHDEP) isn't required for native builds, but will
help in case of cross-compilation.
msg77241 - (view) Author: Roumen Petrov (rpetrov) Date: 2008-12-07 18:55
The support for *-*mingw* host is enhanced to native msys environment.
This include check for device file /dev/ptmx. The MSYS report that file
exist. As the check is skipped on mingw* host os now cugwin environment
can be used too.

Also check libintl is skipped for mingw as the native build don't use it.
msg98223 - (view) Author: Roumen Petrov (rpetrov) Date: 2010-01-24 13:31
system python has to be at least revision 77704 (trunk)
msg105859 - (view) Author: Roumen Petrov (rpetrov) Date: 2010-05-16 12:09
As part of issue 8510 (update to autoconf2.65) configure script is modernized and most of  updates from patches attached to this issue now are in repository.

Starting from 16 may new patches will include in addition minimal updates of setup.py and Makefile.pre.in (moved from issue 3871) : 
- build modules with system python (from 3871).
- configure ctypes with same arguments as top configure (new)
msg108744 - (view) Author: Roumen Petrov (rpetrov) Date: 2010-06-26 19:43
- consistent platform detection in all methods from setup.py;
- readline libraries detected by configure script are used for setup.py : this could help to detect is more platform independent way if readline is linked with ncurses{w} library;
- ignore "friendly" search library paths : this is useless in case of cross-compilation . Also it is reported that those paths are not so "friendly" even in native builds. Note that with "friendly search library paths" python library search based on file suffix will find libraries system on build system if host system lack them.
msg110685 - (view) Author: Mark Lawrence (BreamoreBoy) Date: 2010-07-18 20:35
Could someone with knowledge of the Python build systems please comment on this, thanks.
msg114776 - (view) Author: Roumen Petrov (rpetrov) Date: 2010-08-24 11:22
It seems to me I forgot to upload version after updates in Lib/sysconfig.py from issue 7880.
I could not understand what Florent Xicluna (flox). I could not reproduce it. To me update abspath -> realpath is bogus and with reverted update of Lib/sysconfig.py test_sysconfig pass.

This patch is restored previous behaviour and now symlinked system python into cross build tree will behave as before i.e. as expected ;)
msg127140 - (view) Author: Roumen Petrov (rpetrov) Date: 2011-01-26 21:02
fixed patch failure on Parser/pgen.stamp
msg132102 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-03-25 15:26
I see some “fix for issue #NNN is bogus” in your patch: would you open separate bug reports for those?  A diff file is not a very useful way to report bugs or express opinions <wink>.
msg133252 - (view) Author: Roumen Petrov (rpetrov) Date: 2011-04-07 20:26
Uhh after some pseudo multiarch improvements previous patch fail. So new one is uploaded.
Also with this version cross-build won't build pgen$(EXEEXT).
msg136371 - (view) Author: Greg Hellings (Greg.Hellings) Date: 2011-05-20 13:15
Current patch errors with the following message:
gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -L/manual/lib -L/binary/lib -L/manual/lib -L/binary/lib  Parser/acceler.o Parser/grammar1.o Pars
er/listnode.o Parser/node.o Parser/parser.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o
Python/dynamic_annotations.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/parsetok_pgen.o Parser/pgenmain
.o -lintl -lpthread  -o Parser/pgen.exe
./Grammar/Grammar ./Include/graminit.h ./Python/graminit.c
./Grammar/Grammar: line 18: single_input:: command not found
./Grammar/Grammar: line 18: simple_stmt: command not found
./Grammar/Grammar: line 18: compound_stmt: command not found
./Grammar/Grammar: line 19: syntax error near unexpected token `NEWLINE'
./Grammar/Grammar: line 19: `file_input: (NEWLINE | stmt)* ENDMARKER'
make: *** [Parser/pgen.stamp] Error 2
msg136772 - (view) Author: Roumen Petrov (rpetrov) Date: 2011-05-24 17:07
Greg, ensure correct configure script first as run commands autoheader and autoconf.
Updates to configure script are not in patch . The patch include updates to source configure.in and autoconf command will update "configure".
msg136885 - (view) Author: wrobell (wrobell) Date: 2011-05-25 18:49
Would it be possible to get list of steps required to test this patch?

1. hg clone
2. copy config.sub and config.guess?
3. autoreconf
4. configure...
5. make

What else?
msg136887 - (view) Author: Roumen Petrov (rpetrov) Date: 2011-05-25 19:42
Usually this is not a question for bug-tracking system . configure .. 
--host=.... will enter into cross-compilation mode. You could find more 
details into respective manual.

Roumen
msg136911 - (view) Author: wrobell (wrobell) Date: 2011-05-25 22:06
well, looking at the first comment, there is required more than simple read of the manual (i.e. missing config.guess/config.sub).

is anything else missing or required?
msg136962 - (view) Author: wrobell (wrobell) Date: 2011-05-26 13:54
as well, by default one experiences

checking for /dev/ptmx... not set
configure: error: set ac_cv_file__dev_ptmx to yes/no in your CONFIG_SITE file when cross compiling


the workaround:

1. create config.site file with the following lines

     ac_cv_file__dev_ptmx=no
     ac_cv_file__dev_ptc=no

2. run configure script as: CONFIG_SITE=config.site ./configure --host...

would be it ok if i add appropriate options for /dev/ptmx and /dev/ptc to this patch so we do not have to play with config.site file?
msg136963 - (view) Author: wrobell (wrobell) Date: 2011-05-26 13:55
next question. when starting compilation i am getting

In file included from Include/Python.h:50,
                 from ./Modules/python.c:3:
Include/pyport.h:257:13: error: #error "This platform's pyconfig.h needs to define PY_FORMAT_LONG_LONG"
make: *** [Modules/python.o] Error 1

how to deal with above?
msg136987 - (view) Author: wrobell (wrobell) Date: 2011-05-26 17:45
looking into configure.in the above fails due to following check

  AC_MSG_CHECKING(for %lld and %llu printf() format support)

the check compiles and tries to _run_ a bit of software to determine lld/llu support.

that of course fails (we are cross compiling). this is similar problem to ptmx/ptc problem (we do not know if host have support for ptmx/ptc).

i wonder if the following would be acceptable

1. if in cross compilation mode, the configure.in could load config-cross.site.
2. the config-cross.site would define variables for ptmx/ptc, long long format and other checks which consist of running code snippets
3. appropriate checks (i.e. ptmx/ptc, lld/llu) would not be run in cross compilation mode. variables from config-cross.site would be used instead.
4. it would be a developer responsibility to set appropriate values for his platform in config-cross.site file.

does above sound like a solution?
msg137001 - (view) Author: Roumen Petrov (rpetrov) Date: 2011-05-26 19:50
At least one is really would like to cross-compile.

worber, the config site has to look like (sample for linux i?86, i.e. intel 32 bit, as host platform)
=================================
ac_cv_little_endian_double=yes
ac_cv_broken_sem_getvalue=no
ac_cv_computed_gotos=yes
ac_cv_buggy_getaddrinfo=no
ac_cv_working_tzset=yes
#next line require fix typo in configure.in
ac_cv_pthread=yes
ac_cv_pthread_system_supported=yes
ac_cv_tanh_preserves_zero_sign=yes

ac_cv_have_long_long_format=yes
ac_cv_file__dev_ptmx=yes
ac_cv_file__dev_ptc=no
=================================

Also with following patch into configure.in
=======================================
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -1372,7 +1372,7 @@
 # so we need to run a program to see whether it really made the
 # function available.
 AC_MSG_CHECKING(whether $CC accepts -pthread)
-AC_CACHE_VAL(ac_cv_thread,
+AC_CACHE_VAL(ac_cv_pthread,
 [ac_save_cc="$CC"
 CC="$CC -pthread"
 AC_RUN_IFELSE([AC_LANG_SOURCE([[
=======================================
msg142972 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-25 16:05
Copy-pasting of a previous comment: I see some “fix for issue #NNN is bogus” in your patch: would you open separate bug reports for those?  A diff file is not a very useful way to report bugs or express opinions <wink>.

Copy-pasting of many previous comments: distutils does not get new features (cross-compilation is a new feature), please focus on packaging.
History
Date User Action Args
2011-11-09 16:44:49marqvarsetnosy: + marqvar
2011-08-25 16:05:42eric.araujosetnosy: + alexis, tarek
messages: + msg142972

assignee: tarek
components: + Distutils2
2011-08-24 21:31:25rpetrovsetfiles: + python-py3k-20110825-CROSS.patch
2011-08-18 09:40:20kalevsetnosy: + kalev
2011-05-26 19:50:19rpetrovsetmessages: + msg137001
2011-05-26 17:45:50wrobellsetmessages: + msg136987
2011-05-26 13:55:58wrobellsetmessages: + msg136963
2011-05-26 13:54:38wrobellsetmessages: + msg136962
2011-05-25 22:17:52victorpolucenosetnosy: - victorpoluceno
2011-05-25 22:06:43wrobellsetmessages: + msg136911
2011-05-25 19:42:23rpetrovsetmessages: + msg136887
2011-05-25 18:49:53wrobellsetmessages: + msg136885
2011-05-24 17:07:25rpetrovsetmessages: + msg136772
2011-05-20 15:11:51eric.araujolinkissue3871 dependencies
2011-05-20 13:15:26Greg.Hellingssetmessages: + msg136371
2011-05-20 10:08:25wrobellsetnosy: + wrobell
2011-05-20 05:40:33Greg.Hellingssetnosy: + Greg.Hellings
2011-05-19 22:07:23rpetrovsetfiles: + python-py3k-20110520-CROSS.patch
2011-05-17 01:58:38WhiteTigersetnosy: + WhiteTiger
2011-05-07 01:34:17David.Kernsetnosy: + David.Kern
2011-04-07 20:26:47rpetrovsetfiles: + python-py3k-20110407-CROSS.patch

messages: + msg133252
2011-03-25 15:26:43eric.araujosetmessages: + msg132102
2011-03-25 15:25:27eric.araujosetnosy: + eric.araujo

versions: + Python 3.3, - Python 3.2
2011-01-26 21:05:26rpetrovsetfiles: + python-py3k-20110126-CROSS.patch
nosy: rpetrov, Arfrever, victorpoluceno, BreamoreBoy
2011-01-26 21:02:37rpetrovsetfiles: + python-2.7-20110126-CROSS.patch
nosy: rpetrov, Arfrever, victorpoluceno, BreamoreBoy
messages: + msg127140
2010-11-12 21:03:57rpetrovsetfiles: + python-py3k-20101112-CROSS.patch
2010-11-12 21:02:45rpetrovsetfiles: + python-2.7-20101112-CROSS.patch
2010-08-24 14:50:19Arfreversetnosy: + Arfrever
2010-08-24 11:22:23rpetrovsetfiles: + python-trunk-20100824-CROSS.patch

messages: + msg114776
2010-07-18 20:37:47BreamoreBoysetversions: + Python 3.2, - Python 2.7
2010-07-18 20:35:55BreamoreBoysetnosy: + BreamoreBoy
messages: + msg110685
2010-06-26 19:43:30rpetrovsetfiles: + python-trunk-20100626-CROSS.patch

messages: + msg108744
2010-05-16 12:11:23rpetrovsetfiles: - python-trunk-20100124-CROSS.patch
2010-05-16 12:11:15rpetrovsetfiles: - python-trunk-20100103-CROSS.patch
2010-05-16 12:11:10rpetrovsetfiles: - python-trunk-20091206-CROSS.patch
2010-05-16 12:11:03rpetrovsetfiles: - python-trunk-20091129-CROSS.patch
2010-05-16 12:10:55rpetrovsetfiles: - python-trunk-20091104-CROSS.patch
2010-05-16 12:10:47rpetrovsetfiles: - python-trunk-20090722-CROSS.patch
2010-05-16 12:10:00rpetrovsetfiles: + python-trunk-20100516-CROSS.patch

messages: + msg105859
title: minimal cross-compilation support for configure -> cross-compilation support for python build
2010-01-24 16:12:56brian.curtinsetpriority: normal
keywords: + needs review
stage: patch review
2010-01-24 13:31:09rpetrovsetfiles: + python-trunk-20100124-CROSS.patch

messages: + msg98223
2010-01-03 11:24:42rpetrovsetfiles: + python-trunk-20100103-CROSS.patch
versions: + Python 2.7
2009-12-06 16:37:44rpetrovsetfiles: + python-trunk-20091206-CROSS.patch
2009-11-29 17:43:41rpetrovsetfiles: + python-trunk-20091129-CROSS.patch
2009-11-04 22:04:31rpetrovsetfiles: + python-trunk-20091104-CROSS.patch
2009-07-22 20:08:53rpetrovsetnosy: rpetrov, victorpoluceno
type: enhancement
components: + Build
2009-07-22 20:08:29rpetrovsetfiles: - python-trunk-20090612-CROSS.patch
2009-07-22 20:08:22rpetrovsetfiles: - python-trunk-20090415-CROSS.patch
2009-07-22 20:08:12rpetrovsetfiles: - python-trunk.patch-CROSS-20090110
2009-07-22 20:07:34rpetrovsetfiles: + python-trunk-20090722-CROSS.patch
2009-06-12 19:40:50rpetrovsetfiles: + python-trunk-20090612-CROSS.patch
2009-04-17 16:36:57victorpolucenosetnosy: + victorpoluceno
2009-04-15 20:42:10rpetrovsetfiles: - python-trunk-CROSS.patch
2009-04-15 20:41:54rpetrovsetfiles: - python-trunk-CROSS.patch
2009-04-15 20:41:46rpetrovsetfiles: + python-trunk-20090415-CROSS.patch
2009-01-10 18:41:04rpetrovsetfiles: + python-trunk.patch-CROSS-20090110
2008-12-07 18:55:52rpetrovsetfiles: + python-trunk-CROSS.patch
messages: + msg77241
2008-09-30 17:54:09rpetrovsetfiles: - python-trunk-CROSS.patch
2008-09-30 17:53:55rpetrovsetfiles: + python-trunk-CROSS.patch
2008-09-01 21:49:18rpetrovcreate