classification
Title: don't use host python, use host search paths for host compiler
Type: Stage:
Components: Cross-Build Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: doko Nosy List: Arfrever, barry, doko, eric.araujo, ezio.melotti, georg.brandl, ncoghlan, python-dev, skrah, tarek
Priority: release blocker Keywords: needs review, patch

Created on 2012-03-16 00:14 by doko, last changed 2012-08-08 10:18 by doko. This issue is now closed.

Files
File name Uploaded Description Edit
no-host-python-for-build.diff doko, 2012-03-16 00:14 no-host-python-for-build.diff review
no-host-python-and-host-gcc.diff doko, 2012-06-30 13:28 don't use host python, use host search paths for host compiler review
Messages (26)
msg155976 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-03-16 00:14
for various tasks, the just built python for the host is used. this patch searches for a suitable python for the build machine and uses it for the build.
msg156044 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-03-16 14:47
+	case $$MAKEFLAGS in *s*) quiet=-s; esac; \
Should be quiet=-q.  make -s → setup.py -q
msg164380 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-06-30 13:28
the updated patch addresses:

 - fix typo mentioned in msg156044

 - don't use the host python during build and installation

 - for include files and libraries, use the search directories
   provided by gcc, and only fall back to hard coded
   directories for native builds.

 - fix the dpkg-architecture call for cross builds

 - pass the --host and --build arguments to the libffi build

 - fix the pydebug check, taken from issue #3871.

The patch introduces two new variables, which are set in the environment for cross builds:

 - _PROJECT_BASE is used by Lib/sysconfig.py to point to the
   current source, while using the python3 interpreter of the
   build machine, so that the Makefile and the pyconfig.h are
   read from the current sources.

 - _HOST_PLATFORM is used to override the get_platform calls
   in Lib/sysconfig.py, Lib/distutils/utils.py and setup.py.
   Yes, distutils is frozen, however this change shouldn't
   break anything else. It only relies on one environment
   variable _HOST_PLATFORM, so if this is an issue, then the
   name can be obfuscated even more.

tested with a cross build for arm-linux-gnueabihf on x86_64-linux-gnu, and a native build.
msg164397 - (view) Author: Roundup Robot (python-dev) Date: 2012-06-30 16:33
New changeset b285ec79e5de by Matthias Klose in branch 'default':
previous patch needs patch from issue #14330
http://hg.python.org/cpython/rev/b285ec79e5de
msg164406 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-06-30 18:22
Patch looks okay as it contains no important changes to distutils, the compilers or any big part; it’s the sort of changes that have been accepted to fix Mac issue or support multiarch on Debian.  I would prefer safer names for environment variables though, for example PYTHON_BUILD_HOST_PLATFORM instead of just _HOST_PLATFORM (unless this is not specific to the Python build process but used by all cross-compiling builds).
msg164408 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-06-30 18:33
PYTHON_BUILD_HOST_PLATFORM is confusing/misleading. I'll use _PYTHON_HOST_PLATFORM (with the leading underscore to mark it somehow internal).
msg164410 - (view) Author: Roundup Robot (python-dev) Date: 2012-06-30 18:44
New changeset cfbe51e66749 by doko in branch 'default':
- Issue #14330: For cross builds, don't use host python, use host search paths
http://hg.python.org/cpython/rev/cfbe51e66749
msg164411 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-06-30 18:48
applied
msg164414 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-06-30 19:09
+if "_PROJECT_BASE" in os.environ:

Can you use _PYTHON_PROJECT_BASE here?  Thanks.

(BTW it would be nice if you could mark the bugs for which you did commits as closed+fixed+committed.)
msg164424 - (view) Author: Roundup Robot (python-dev) Date: 2012-06-30 20:38
New changeset 86ac09f118be by doko in branch 'default':
- Issue #14330: For cross builds, don't use host python, use host search paths
http://hg.python.org/cpython/rev/86ac09f118be
msg164425 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-06-30 20:39
now renamed, add added the news entry
msg164432 - (view) Author: Roundup Robot (python-dev) Date: 2012-06-30 22:40
New changeset 7955d769fdf5 by doko in branch 'default':
- Issue #14330: Don't derive the include and library search dirs
http://hg.python.org/cpython/rev/7955d769fdf5
msg166444 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2012-07-26 00:08
The last changeset (7955d769fdf5) caused some problem on my machine.
Before the changeset I was getting:

Python build finished, but the necessary bits to build these modules were not found:
_bz2               _curses            _curses_panel   
_dbm               _gdbm              _lzma           
_tkinter                                              
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

After 7955d769fdf5 I get:

Python build finished, but the necessary bits to build these modules were not found:
_bz2               _curses            _curses_panel   
_dbm               _gdbm              _lzma           
_sqlite3           _ssl               _tkinter        
readline           zlib                               
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

I'm on an ubuntu 12.04 machine.
msg166542 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-07-27 05:18
I would really like to release beta2 this weekend.  Is it possible to get this resolved by then?
msg166571 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-07-27 14:42
just back in DE today, but still travelling. Won't be able to look at this before late Monday.
msg166658 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-07-28 13:44
Everything is still building happily for me on Fedora, so I suggest looking at the Debian/Ubuntu multiarch support (which is the origin of the dpkg-architecture call).

Background in http://bugs.python.org/issue11715
msg167290 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-08-03 05:40
Matthias, any comments?
msg167436 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-08-04 20:17
Matthias: ping.  I don't want to hold up beta2 indefinitely.  In order to resolve this blocker, we will revert the batch of cross-compiling patches in a few days if there is no progress here.
msg167612 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-08-07 10:10
these are all extensions, which use headers and libraries installed in multiarch paths, which I think are not found in this case. If the dpkg-dev package isn't installed, please install it and recheck. So this issue should be closed, maybe with the ma.diff patch applies as in issue #15426.
msg167613 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-08-07 10:12
and please make sure that other build dependencies are installed as well:

 sudo apt-get build-dep python3.2 (on 12.04/precise)

 sudo apt-get build-dep python3.3 (on 12.10/quantal)
msg167615 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-08-07 10:16
afaics, msg166444 doesn't have to do anything with the cross build issue, but a missing build dependency (here: dpkg-dev).
msg167625 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-08-07 17:12
Ezio?
msg167626 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-08-07 17:12
Installing dpkg-dev indeed resolved the issue for me on Debian Wheezy,
but msg166444 said that the problem appeared in 7955d769fdf5. 

So was dpkg-dev already an official dependency before 7955d769fdf5
or not?
msg167627 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-08-07 17:24
With ma.diff from #11715 dpkg-dev is indeed not required (checked
on Wheezy).
msg167643 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2012-08-07 21:30
On another Ubuntu 12.04 machine with dpkg-dev installed I didn't have any issue.  Once I removed dpkg-dev I was able to reproduce the issue again.
Applying ma.diff from #11715 solves the issue, with and without dpkg-dev installed.
msg167681 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-08-08 10:18
committed the ma.diff from #11715, msg167680.
History
Date User Action Args
2012-08-08 10:18:23dokosetstatus: open -> closed
resolution: fixed
messages: + msg167681
2012-08-07 21:30:35ezio.melottisetmessages: + msg167643
2012-08-07 17:24:13skrahsetmessages: + msg167627
2012-08-07 17:12:51skrahsetnosy: + skrah
messages: + msg167626
2012-08-07 17:12:20georg.brandlsetmessages: + msg167625
2012-08-07 10:16:20dokosetmessages: + msg167615
2012-08-07 10:12:32dokosetmessages: + msg167613
2012-08-07 10:10:34dokosetmessages: + msg167612
2012-08-04 20:17:38georg.brandlsetmessages: + msg167436
2012-08-03 05:40:23georg.brandlsetmessages: + msg167290
2012-07-28 13:44:16ncoghlansetnosy: + barry, ncoghlan
resolution: fixed -> (no value)
messages: + msg166658
2012-07-27 14:42:50dokosetmessages: + msg166571
2012-07-27 05:18:59georg.brandlsetnosy: + georg.brandl
messages: + msg166542
2012-07-26 05:58:23georg.brandlsetpriority: normal -> release blocker
2012-07-26 00:08:23ezio.melottisetstatus: closed -> open
nosy: + ezio.melotti
messages: + msg166444

2012-06-30 22:40:48python-devsetmessages: + msg164432
2012-06-30 20:39:37dokosetresolution: fixed
2012-06-30 20:39:03dokosetmessages: + msg164425
2012-06-30 20:38:24python-devsetmessages: + msg164424
2012-06-30 19:09:47eric.araujosetmessages: + msg164414
2012-06-30 18:48:11dokosetstatus: open -> closed

messages: + msg164411
2012-06-30 18:44:57python-devsetmessages: + msg164410
2012-06-30 18:33:43dokosetmessages: + msg164408
2012-06-30 18:22:41eric.araujosetmessages: + msg164406
2012-06-30 16:33:30python-devsetnosy: + python-dev
messages: + msg164397
2012-06-30 13:29:32dokosetnosy: + tarek
2012-06-30 13:28:46dokosetfiles: + no-host-python-and-host-gcc.diff

messages: + msg164380
title: do not use the host python for cross builds -> don't use host python, use host search paths for host compiler
2012-03-16 14:47:48eric.araujosetnosy: + eric.araujo
messages: + msg156044
2012-03-16 00:28:27Arfreversetnosy: + Arfrever
2012-03-16 00:14:07dokocreate