Building Python on multiarch Debian and Ubuntu #55924

warsaw · 2011-03-29T21:48:17Z

BPO	11715
Nosy	@loewis, @warsaw, @doko42, @jcea, @ncoghlan, @tiran, @ezio-melotti, @merwok, @skrah, @sandrotosi
Files	d504ca7e0fe5.diff find_executable.patch ma.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/warsaw'
closed_at = <Date 2012-09-21.12:05:54.682>
created_at = <Date 2011-03-29.21:48:17.378>
labels = ['type-bug', 'build']
title = 'Building Python on multiarch Debian and Ubuntu'
updated_at = <Date 2012-09-21.12:05:54.681>
user = 'https://github.com/warsaw'

bugs.python.org fields:

activity = <Date 2012-09-21.12:05:54.681>
actor = 'doko'
assignee = 'barry'
closed = True
closed_date = <Date 2012-09-21.12:05:54.682>
closer = 'doko'
components = ['Build']
creation = <Date 2011-03-29.21:48:17.378>
creator = 'barry'
dependencies = []
files = ['21465', '21558', '26719']
hgrepos = ['13']
issue_num = 11715
keywords = ['patch']
message_count = 28.0
messages = ['132536', '132714', '132717', '132718', '132744', '132745', '132747', '133155', '133165', '133194', '133219', '133220', '133222', '142304', '142763', '143892', '143893', '143899', '143906', '145561', '167611', '167680', '169908', '169914', '169945', '169946', '170881', '170882']
nosy_count = 12.0
nosy_names = ['loewis', 'barry', 'doko', 'jcea', 'ncoghlan', 'christian.heimes', 'ezio.melotti', 'eric.araujo', 'skrah', 'sandro.tosi', 'rosslagerwall', 'python-dev']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11715'
versions = ['Python 2.6', 'Python 2.5', 'Python 3.1', 'Python 2.7', 'Python 3.2', 'Python 3.3']

warsaw · 2011-03-29T21:48:17Z

Ubuntu 11.04 introduces new directories for libraries and headers to support multiple architectures on a single machine. E.g. 64bit and 32bit on a 64bit Ubuntu. Here are the specs:

https://wiki.ubuntu.com/MultiarchSpec
http://wiki.debian.org/ReleaseGoals/MultiArch

Unlike bpo-1294959 this bug simply covers building Python and its stdlib extension modules on such systems. For example, libsqlite3.so is no longer in /usr/lib but instead in /usr/lib/x86_64-linux-gnu on Ubuntu 11.04. This means that a number of extension modules which depend on these 3rd party shared libraries simply won't build, because setup.py is too naive about the paths it searches.

The fix is fairly simple; you have to call out to dpkg-architecture to get the value of the platform's architecture and append that to /usr/lib and /usr/include for setup.py's search paths. See the attached branch and patch for a candidate fix.

Because this does not introduce a new feature, it simply fixes the build process for changes to already supported platforms, I would like to apply this to Python 2.5 through 3.3.

loewis · 2011-03-31T22:08:51Z

Python 2.5 is not open for bug fixes anymore, so this can't be applied to this branch. I suggest that Python 2.6 is closed for bug fixes as well.

warsaw · 2011-03-31T22:52:58Z

On Mar 31, 2011, at 10:08 PM, Martin v. Löwis wrote:

Martin v. Löwis <martin@v.loewis.de> added the comment:

Python 2.5 is not open for bug fixes anymore, so this can't be applied to
this branch. I suggest that Python 2.6 is closed for bug fixes as well.

Although I'd still like to apply it to 2.5, I defer to you as RM. However,
if/when I ever need to do a 2.6 release I'll need to build and test it and I
don't want to have to keep an old version of Ubuntu around to do that.

So how about if I apply it to 2.6, 2.7, 3.1 - 3.3?

(I've also addressed the comment you made in the review.)

loewis · 2011-03-31T22:59:39Z

If Ubuntu stops supporting to build old Python releases, I rather consider this a serious bug in Debian, not one in Python.

If the issue is merely that certain extension modules fail to build, I see no real reason to do anything about it. Users affected by this issue can still edit Modules/Setup.

warsaw · 2011-04-01T15:19:00Z

"As I see it, the patch is uncontroversial for 3.3, 3.2, and 2.7. And it definitely will not be applied to 3.0. That leaves 2.5, 2.6, and 3.1. If you really care one way or the other, please register your vote in the tracker."

2.5: +0
2.6: +1
3.1: +1

merwok · 2011-04-01T15:30:15Z

I’m not opposed to the change.

ncoghlan · 2011-04-01T15:55:32Z

2.5: -1
2.6: -0
3.1: +0

As I see it, a large part of the "security fixes only" rule is for the benefit of folks auditing those security fixes, as it means there's very little noise in the branch to confuse the matter.

skrah · 2011-04-06T19:25:41Z

Since I do automated module testing against all Python versions,
my vote would be:

2.5: +1
2.6: +1
3.1: +1

This with the caveat that of course Martin has to decide for 2.5.
I'm just indicating my preference.

python-dev · 2011-04-06T20:35:37Z

New changeset 7582a78f573b by Barry Warsaw in branch '3.1':
bpo-11715: Build extension modules on multiarch Debian and Ubuntu by
http://hg.python.org/cpython/rev/7582a78f573b

New changeset 867937dd2279 by Barry Warsaw in branch '3.2':
bpo-11715: Merge multiarch fix from 3.1 branch.
http://hg.python.org/cpython/rev/867937dd2279

New changeset 3f00611c3daf by Barry Warsaw in branch 'default':
bpo-11715: Merge multiarch fix from 3.1 branch.
http://hg.python.org/cpython/rev/3f00611c3daf

skrah · 2011-04-07T06:30:45Z

The FreeBSD and Solaris bots are failing:

dpkg-architecture: not found
error: build/temp.freebsd-8.2-RELEASE-amd64-3.3-pydebug/multiarch: No such file or directory
[62607 refs]
*** Error code 1

find_executable.patch should solve the problem.

warsaw · 2011-04-07T14:32:52Z

Stefan, thanks for the patch. The problem is that on FreeBSD and Solaris, if the command fails, I/O redirection does not create the file, whereas on Linux and OS X it does. So I was going to wrap the os.remove() of the temp file in a try/except. But I like your patch better because it avoids the dpkg-architecture call in the first place on systems that don't have it, albeit at the cost of traversing $PATH.

python-dev · 2011-04-07T14:48:42Z

New changeset c8738114b962 by Barry Warsaw in branch '3.1':
Refinement by Stefan Krah (see bpo-11715, msg133194) to exit early if the
http://hg.python.org/cpython/rev/c8738114b962

New changeset 3d7c9b38fbfd by Barry Warsaw in branch '3.2':
Refinement by Stefan Krah (see bpo-11715, msg133194) to exit early if the
http://hg.python.org/cpython/rev/3d7c9b38fbfd

New changeset bbfc65d05588 by Barry Warsaw in branch 'default':
Refinement by Stefan Krah (see bpo-11715, msg133194) to exit early if the
http://hg.python.org/cpython/rev/bbfc65d05588

python-dev · 2011-04-07T15:28:51Z

New changeset bd0f73a9538e by Barry Warsaw in branch '2.7':
Backport for Python 2.7 of bpo-11715 support for building Python on
http://hg.python.org/cpython/rev/bd0f73a9538e

ncoghlan · 2011-08-18T07:09:48Z

I'm not sure this is 100% fixed. After dist-upgrading the Kubuntu VM on my netbook and updating to the latest Py3k code, I got a lot of test errors, even after a make distclean and ./configure.

The errors went away after manually tweaking LDFLAGS as Christian describes here:
http://lipyrary.blogspot.com/2011/05/how-to-compile-python-on-ubuntu-1104.html

warsaw · 2011-08-22T22:18:29Z

On Aug 18, 2011, at 07:09 AM, Nick Coghlan wrote:

I'm not sure this is 100% fixed. After dist-upgrading the Kubuntu VM on my
netbook and updating to the latest Py3k code, I got a lot of test errors,
even after a make distclean and ./configure.

Hi Nick. Would this be Kubuntu 11.04 or some other release?

tiran · 2011-09-12T11:48:28Z

Hey Nick and Barry,

the fix in http://hg.python.org/cpython/rev/bd0f73a9538e isn't sufficient. You have added /usr/lib/MULTIARCH and /usr/include/MULTIARCH but you forgot to add /lib/MULTIARCH. On my system zlib is installed at /lib/x86_64-linux-gnu/libz.so.

tiran · 2011-09-12T11:55:24Z

Update:

It turns out that zlib1g-dev adds a symlink from /usr/lib/x86_64-linux-gnu/libz.so to /lib/x86_64-linux-gnu/libz.so.1 .

$ locate libz.
/lib/x86_64-linux-gnu/libz.so.1
/lib/x86_64-linux-gnu/libz.so.1.2.3.4
/usr/lib/x86_64-linux-gnu/libz.a
/usr/lib/x86_64-linux-gnu/libz.so

Perhaps this symlink is missing on Nick's installation. It might be a wise idea to add /lib/MULTIARCH to the library search paths, too.

ncoghlan · 2011-09-12T12:34:06Z

It wouldn't surprise me at all if the laptop's links were a little off - I started with a Kubuntu image off VMWare's site quite some time ago, then dist-upgraded it through a couple of releases as they came out.

warsaw · 2011-09-12T15:35:46Z

On Sep 12, 2011, at 12:34 PM, Nick Coghlan wrote:

It wouldn't surprise me at all if the laptop's links were a little off - I
started with a Kubuntu image off VMWare's site quite some time ago, then
dist-upgraded it through a couple of releases as they came out.

I'll try to get a VM up with the latest Oneiric Kubuntu image and see what
happens.

rosslagerwall · 2011-10-14T20:10:42Z

I see this requires dpkg-architecture which isn't always available. While it isn't hard to install it, it isn't very clear that this is the cause of the nis and crypt modules failing to build on a fresh install of 11.10.

What would be nice is if there were some way of determining this information without dpkg-architecture such as reading it from /etc/ld.so.conf.d/x86_64-linux-gnu.conf.

doko42 · 2012-08-07T10:05:30Z

about searching /lib/<multiarch>: adding this directory won't help. all .a and .so files are installed in /usr/lib or /usr/lib/<multiarch>.

about the missing dpkg-architecture: see the attached ma.diff patch. the Debian/Ubuntu system compilers add an option -print-multiarch, which can be used to get the multiarch name without having the dpkg-dev package installed.

python-dev · 2012-08-08T10:16:17Z

New changeset 5966c206654b by doko in branch 'default':

Issue bpo-11715: Fix multiarch detection without having Debian development
http://hg.python.org/cpython/rev/5966c206654b

ezio-melotti · 2012-09-06T09:41:47Z

Today I got the same problem on 3.2 too.
I made a distclean and recompiled and got:

Python build finished, but the necessary bits to build these modules were not found:
_curses _curses_panel _dbm
_gdbm _sqlite3 _ssl
_tkinter bz2 readline

I tried to apply ma.diff and now I get:

Python build finished, but the necessary bits to build these modules were not found:
_curses _curses_panel _dbm
_gdbm _tkinter bz2

ezio-melotti · 2012-09-06T12:22:49Z

2.7 is affected too:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb _curses _curses_panel
_sqlite3 _ssl _tkinter
bsddb185 bz2 dbm
gdbm readline sunaudiodev

And after applying ma.diff:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb _curses _curses_panel
_tkinter bsddb185 bz2
dbm gdbm sunaudiodev

doko42 · 2012-09-06T22:31:55Z

I think that the ma.diff can safely go to the 2.7 and 3.2 branches.

For the other extensions which are not built you are probably missing the build dependencies (try: apt-get build-dep python2.7 python3.2).

ezio-melotti · 2012-09-06T22:33:54Z

The other missing extensions are not a problem, as long as the one that I need and already have (e.g. readline, _ssl) are built correctly.

python-dev · 2012-09-21T12:04:55Z

New changeset 53fa224b95f4 by doko in branch '2.7':

Issue bpo-11715: Fix multiarch detection without having Debian development
http://hg.python.org/cpython/rev/53fa224b95f4

New changeset 78aba41a8105 by doko in branch '3.2':

Issue bpo-11715: Fix multiarch detection without having Debian development
http://hg.python.org/cpython/rev/78aba41a8105

doko42 · 2012-09-21T12:05:55Z

fixed for 2.7 and 3.2 as well.

warsaw added build The build process and cross-build type-bug An unexpected behavior, bug, or error labels Mar 29, 2011

warsaw self-assigned this Apr 6, 2011

warsaw closed this as completed Apr 7, 2011

doko42 reopened this Aug 7, 2012

doko42 closed this as completed Aug 8, 2012

ezio-melotti reopened this Sep 6, 2012

doko42 closed this as completed Sep 21, 2012

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Building Python on multiarch Debian and Ubuntu #55924

Building Python on multiarch Debian and Ubuntu #55924

warsaw commented Mar 29, 2011

warsaw commented Mar 29, 2011

loewis mannequin commented Mar 31, 2011

warsaw commented Mar 31, 2011

loewis mannequin commented Mar 31, 2011

warsaw commented Apr 1, 2011

merwok commented Apr 1, 2011

ncoghlan commented Apr 1, 2011

skrah mannequin commented Apr 6, 2011

python-dev mannequin commented Apr 6, 2011

skrah mannequin commented Apr 7, 2011

warsaw commented Apr 7, 2011

python-dev mannequin commented Apr 7, 2011

python-dev mannequin commented Apr 7, 2011

ncoghlan commented Aug 18, 2011

warsaw commented Aug 22, 2011

tiran commented Sep 12, 2011

tiran commented Sep 12, 2011

ncoghlan commented Sep 12, 2011

warsaw commented Sep 12, 2011

rosslagerwall mannequin commented Oct 14, 2011

doko42 commented Aug 7, 2012

python-dev mannequin commented Aug 8, 2012

ezio-melotti commented Sep 6, 2012

ezio-melotti commented Sep 6, 2012

doko42 commented Sep 6, 2012

ezio-melotti commented Sep 6, 2012

python-dev mannequin commented Sep 21, 2012

doko42 commented Sep 21, 2012

Building Python on multiarch Debian and Ubuntu #55924

Building Python on multiarch Debian and Ubuntu #55924

Comments

warsaw commented Mar 29, 2011

warsaw commented Mar 29, 2011

loewis mannequin commented Mar 31, 2011

warsaw commented Mar 31, 2011

loewis mannequin commented Mar 31, 2011

warsaw commented Apr 1, 2011

merwok commented Apr 1, 2011

ncoghlan commented Apr 1, 2011

skrah mannequin commented Apr 6, 2011

python-dev mannequin commented Apr 6, 2011

skrah mannequin commented Apr 7, 2011

warsaw commented Apr 7, 2011

python-dev mannequin commented Apr 7, 2011

python-dev mannequin commented Apr 7, 2011

ncoghlan commented Aug 18, 2011

warsaw commented Aug 22, 2011

tiran commented Sep 12, 2011

tiran commented Sep 12, 2011

ncoghlan commented Sep 12, 2011

warsaw commented Sep 12, 2011

rosslagerwall mannequin commented Oct 14, 2011

doko42 commented Aug 7, 2012

python-dev mannequin commented Aug 8, 2012

ezio-melotti commented Sep 6, 2012

ezio-melotti commented Sep 6, 2012

doko42 commented Sep 6, 2012

ezio-melotti commented Sep 6, 2012

python-dev mannequin commented Sep 21, 2012

doko42 commented Sep 21, 2012