classification
Title: Building Python on multiarch Debian and Ubuntu
Type: behavior Stage:
Components: Build Versions: Python 3.3, Python 3.2, Python 3.1, Python 2.7, Python 2.6, Python 2.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: barry Nosy List: barry, christian.heimes, doko, eric.araujo, ezio.melotti, jcea, loewis, ncoghlan, python-dev, rosslagerwall, sandro.tosi, skrah
Priority: high Keywords: patch

Created on 2011-03-29 21:48 by barry, last changed 2012-09-21 12:05 by doko. This issue is now closed.

Files
File name Uploaded Description Edit
d504ca7e0fe5.diff barry, 2011-03-29 21:56 review
find_executable.patch skrah, 2011-04-07 06:30 review
ma.diff doko, 2012-08-07 10:05 review
Repositories containing patches
http://hg.python.org/users/barry#multiarch
Messages (28)
msg132536 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-03-29 21:48
Ubuntu 11.04 introduces new directories for libraries and headers to support multiple architectures on a single machine.  E.g. 64bit and 32bit on a 64bit Ubuntu.  Here are the specs:

https://wiki.ubuntu.com/MultiarchSpec
http://wiki.debian.org/ReleaseGoals/MultiArch

Unlike issue 1294959 this bug simply covers building Python and its stdlib extension modules on such systems.  For example, libsqlite3.so is no longer in /usr/lib but instead in /usr/lib/x86_64-linux-gnu on Ubuntu 11.04.  This means that a number of extension modules which depend on these 3rd party shared libraries simply won't build, because setup.py is too naive about the paths it searches.

The fix is fairly simple; you have to call out to dpkg-architecture to get the value of the platform's architecture and append that to /usr/lib and /usr/include for setup.py's search paths.  See the attached branch and patch for a candidate fix.

Because this does not introduce a new feature, it simply fixes the build process for changes to already supported platforms, I would like to apply this to Python 2.5 through 3.3.
msg132714 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-03-31 22:08
Python 2.5 is not open for bug fixes anymore, so this can't be applied to this branch. I suggest that Python 2.6 is closed for bug fixes as well.
msg132717 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-03-31 22:52
On Mar 31, 2011, at 10:08 PM, Martin v. Löwis wrote:

>Martin v. Löwis <martin@v.loewis.de> added the comment:
>
>Python 2.5 is not open for bug fixes anymore, so this can't be applied to
>this branch. I suggest that Python 2.6 is closed for bug fixes as well.

Although I'd still like to apply it to 2.5, I defer to you as RM.  However,
if/when I ever need to do a 2.6 release I'll need to build and test it and I
don't want to have to keep an old version of Ubuntu around to do that.

So how about if I apply it to 2.6, 2.7, 3.1 - 3.3?

(I've also addressed the comment you made in the review.)
msg132718 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2011-03-31 22:59
If Ubuntu stops supporting to build old Python releases, I rather consider this a serious bug in Debian, not one in Python.

If the issue is merely that certain extension modules fail to build, I see no real reason to do anything about it. Users affected by this issue can still edit Modules/Setup.
msg132744 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-04-01 15:18
"As I see it, the patch is uncontroversial for 3.3, 3.2, and 2.7.  And it definitely will not be applied to 3.0.  That leaves 2.5, 2.6, and 3.1.  If you really care one way or the other, please register your vote in the tracker."

2.5: +0
2.6: +1
3.1: +1
msg132745 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-04-01 15:30
I’m not opposed to the change.
msg132747 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-04-01 15:55
2.5: -1
2.6: -0
3.1: +0

As I see it, a large part of the "security fixes only" rule is for the benefit of folks auditing those security fixes, as it means there's very little noise in the branch to confuse the matter.
msg133155 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-04-06 19:25
Since I do automated module testing against all Python versions,
my vote would be:

2.5: +1
2.6: +1
3.1: +1


This with the caveat that of course Martin has to decide for 2.5.
I'm just indicating my preference.
msg133165 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-06 20:35
New changeset 7582a78f573b by Barry Warsaw in branch '3.1':
Issue 11715: Build extension modules on multiarch Debian and Ubuntu by
http://hg.python.org/cpython/rev/7582a78f573b

New changeset 867937dd2279 by Barry Warsaw in branch '3.2':
Issue 11715: Merge multiarch fix from 3.1 branch.
http://hg.python.org/cpython/rev/867937dd2279

New changeset 3f00611c3daf by Barry Warsaw in branch 'default':
Issue 11715: Merge multiarch fix from 3.1 branch.
http://hg.python.org/cpython/rev/3f00611c3daf
msg133194 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-04-07 06:30
The FreeBSD and Solaris bots are failing:

dpkg-architecture: not found
error: build/temp.freebsd-8.2-RELEASE-amd64-3.3-pydebug/multiarch: No such file or directory
[62607 refs]
*** Error code 1


find_executable.patch should solve the problem.
msg133219 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-04-07 14:32
Stefan, thanks for the patch.  The problem is that on FreeBSD and Solaris, if the command fails, I/O redirection does not create the file, whereas on Linux and OS X it does.  So I was going to wrap the os.remove() of the temp file in a try/except.  But I like your patch better because it avoids the dpkg-architecture call in the first place on systems that don't have it, albeit at the cost of traversing $PATH.
msg133220 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-07 14:48
New changeset c8738114b962 by Barry Warsaw in branch '3.1':
Refinement by Stefan Krah (see issue 11715, msg133194) to exit early if the
http://hg.python.org/cpython/rev/c8738114b962

New changeset 3d7c9b38fbfd by Barry Warsaw in branch '3.2':
Refinement by Stefan Krah (see issue 11715, msg133194) to exit early if the
http://hg.python.org/cpython/rev/3d7c9b38fbfd

New changeset bbfc65d05588 by Barry Warsaw in branch 'default':
Refinement by Stefan Krah (see issue 11715, msg133194) to exit early if the
http://hg.python.org/cpython/rev/bbfc65d05588
msg133222 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-07 15:28
New changeset bd0f73a9538e by Barry Warsaw in branch '2.7':
Backport for Python 2.7 of issue 11715 support for building Python on
http://hg.python.org/cpython/rev/bd0f73a9538e
msg142304 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-08-18 07:09
I'm not sure this is 100% fixed. After dist-upgrading the Kubuntu VM on my netbook and updating to the latest Py3k code, I got a lot of test errors, even after a make distclean and ./configure.

The errors went away after manually tweaking LDFLAGS as Christian describes here:
http://lipyrary.blogspot.com/2011/05/how-to-compile-python-on-ubuntu-1104.html
msg142763 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-08-22 22:18
On Aug 18, 2011, at 07:09 AM, Nick Coghlan wrote:

>I'm not sure this is 100% fixed. After dist-upgrading the Kubuntu VM on my
>netbook and updating to the latest Py3k code, I got a lot of test errors,
>even after a make distclean and ./configure.

Hi Nick.  Would this be Kubuntu 11.04 or some other release?
msg143892 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2011-09-12 11:48
Hey Nick and Barry,

the fix in http://hg.python.org/cpython/rev/bd0f73a9538e isn't sufficient. You have added /usr/lib/MULTIARCH and /usr/include/MULTIARCH but you forgot to add /lib/MULTIARCH. On my system zlib is installed at /lib/x86_64-linux-gnu/libz.so.
msg143893 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2011-09-12 11:55
Update:

It turns out that zlib1g-dev adds a symlink from /usr/lib/x86_64-linux-gnu/libz.so to /lib/x86_64-linux-gnu/libz.so.1 .

$ locate libz.
/lib/x86_64-linux-gnu/libz.so.1
/lib/x86_64-linux-gnu/libz.so.1.2.3.4
/usr/lib/x86_64-linux-gnu/libz.a
/usr/lib/x86_64-linux-gnu/libz.so

Perhaps this symlink is missing on Nick's installation. It might be a wise idea to add /lib/MULTIARCH to the library search paths, too.
msg143899 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-09-12 12:34
It wouldn't surprise me at all if the laptop's links were a little off - I started with a Kubuntu image off VMWare's site quite some time ago, then dist-upgraded it through a couple of releases as they came out.
msg143906 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-09-12 15:35
On Sep 12, 2011, at 12:34 PM, Nick Coghlan wrote:

>It wouldn't surprise me at all if the laptop's links were a little off - I
>started with a Kubuntu image off VMWare's site quite some time ago, then
>dist-upgraded it through a couple of releases as they came out.

I'll try to get a VM up with the latest Oneiric Kubuntu image and see what
happens.
msg145561 - (view) Author: Ross Lagerwall (rosslagerwall) (Python committer) Date: 2011-10-14 20:10
I see this requires dpkg-architecture which isn't always available. While it isn't hard to install it, it isn't very clear that this is the cause of the nis and crypt modules failing to build on a fresh install of 11.10.

What would be nice is if there were some way of determining this information without dpkg-architecture such as reading it from /etc/ld.so.conf.d/x86_64-linux-gnu.conf.
msg167611 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-08-07 10:05
about searching /lib/<multiarch>: adding this directory won't help. all .a and .so files are installed in /usr/lib or /usr/lib/<multiarch>.

about the missing dpkg-architecture: see the attached ma.diff patch. the Debian/Ubuntu system compilers add an option -print-multiarch, which can be used to get the multiarch name without having the dpkg-dev package installed.
msg167680 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-08 10:16
New changeset 5966c206654b by doko in branch 'default':
- Issue #11715: Fix multiarch detection without having Debian development
http://hg.python.org/cpython/rev/5966c206654b
msg169908 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2012-09-06 09:41
Today I got the same problem on 3.2 too.
I made a distclean and recompiled and got:

Python build finished, but the necessary bits to build these modules were not found:
_curses            _curses_panel      _dbm
_gdbm              _sqlite3           _ssl
_tkinter           bz2                readline

I tried to apply ma.diff and now I get:

Python build finished, but the necessary bits to build these modules were not found:
_curses            _curses_panel      _dbm            
_gdbm              _tkinter           bz2
msg169914 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2012-09-06 12:22
2.7 is affected too:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _curses            _curses_panel   
_sqlite3           _ssl               _tkinter        
bsddb185           bz2                dbm             
gdbm               readline           sunaudiodev     

And after applying ma.diff:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _curses            _curses_panel   
_tkinter           bsddb185           bz2             
dbm                gdbm               sunaudiodev
msg169945 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-09-06 22:31
I think that the ma.diff can safely go to the 2.7 and 3.2 branches.

For the other extensions which are not built you are probably missing the build dependencies (try: apt-get build-dep python2.7 python3.2).
msg169946 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2012-09-06 22:33
The other missing extensions are not a problem, as long as the one that I need and already have (e.g. readline, _ssl) are built correctly.
msg170881 - (view) Author: Roundup Robot (python-dev) Date: 2012-09-21 12:04
New changeset 53fa224b95f4 by doko in branch '2.7':
- Issue #11715: Fix multiarch detection without having Debian development
http://hg.python.org/cpython/rev/53fa224b95f4

New changeset 78aba41a8105 by doko in branch '3.2':
- Issue #11715: Fix multiarch detection without having Debian development
http://hg.python.org/cpython/rev/78aba41a8105
msg170882 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-09-21 12:05
fixed for 2.7 and 3.2 as well.
History
Date User Action Args
2012-09-21 12:05:54dokosetstatus: open -> closed

messages: + msg170882
2012-09-21 12:04:55python-devsetmessages: + msg170881
2012-09-21 08:37:24ezio.melottisetmessages: - msg169961
2012-09-21 08:37:21ezio.melottisetmessages: - msg169959
2012-09-21 08:37:17ezio.melottisetmessages: - msg169954
2012-09-07 00:39:15christian.heimessetmessages: + msg169961
2012-09-07 00:27:24python-devsetmessages: + msg169959
2012-09-06 23:19:04python-devsetmessages: + msg169954
2012-09-06 22:33:53ezio.melottisetmessages: + msg169946
2012-09-06 22:31:55dokosetmessages: + msg169945
2012-09-06 12:22:48ezio.melottisetmessages: + msg169914
2012-09-06 09:41:47ezio.melottisetstatus: closed -> open
nosy: + ezio.melotti
messages: + msg169908

2012-08-08 10:18:44dokosetstatus: open -> closed
resolution: fixed
2012-08-08 10:16:17python-devsetmessages: + msg167680
2012-08-07 10:05:32dokosetstatus: closed -> open
files: + ma.diff

nosy: + doko
messages: + msg167611

resolution: fixed -> (no value)
2011-10-14 20:10:42rosslagerwallsetnosy: + rosslagerwall
messages: + msg145561
2011-09-12 15:35:45barrysetmessages: + msg143906
2011-09-12 12:34:05ncoghlansetmessages: + msg143899
2011-09-12 11:55:24christian.heimessetmessages: + msg143893
2011-09-12 11:48:28christian.heimessetnosy: + christian.heimes
messages: + msg143892
2011-08-22 22:18:28barrysetmessages: + msg142763
2011-08-18 07:09:48ncoghlansetmessages: + msg142304
2011-05-27 14:51:50barrylinkissue12194 superseder
2011-04-07 15:32:05barrysetstatus: open -> closed
resolution: fixed
2011-04-07 15:28:51python-devsetmessages: + msg133222
2011-04-07 14:48:42python-devsetmessages: + msg133220
2011-04-07 14:32:52barrysetmessages: + msg133219
2011-04-07 06:30:45skrahsetfiles: + find_executable.patch

messages: + msg133194
2011-04-06 20:35:37python-devsetnosy: + python-dev
messages: + msg133165
2011-04-06 19:25:41skrahsetnosy: + skrah
messages: + msg133155
2011-04-06 18:39:14barrysetassignee: barry
2011-04-01 18:11:50sandro.tosisetnosy: + sandro.tosi
2011-04-01 15:55:31ncoghlansetnosy: + ncoghlan
messages: + msg132747
2011-04-01 15:30:14eric.araujosetnosy: + eric.araujo
messages: + msg132745
2011-04-01 15:18:59barrysetmessages: + msg132744
2011-03-31 22:59:38loewissetmessages: + msg132718
2011-03-31 22:52:58barrysetmessages: + msg132717
2011-03-31 22:08:50loewissetnosy: + loewis
messages: + msg132714
2011-03-30 11:32:49jceasetnosy: + jcea
2011-03-29 21:56:12barrysetfiles: + d504ca7e0fe5.diff
2011-03-29 21:53:35barrysetfiles: - a9b05b89ea39.diff
2011-03-29 21:52:46barrysetfiles: + a9b05b89ea39.diff
keywords: + patch
2011-03-29 21:48:17barrycreate