Ubuntu 11.04 introduces new directories for libraries and headers to support multiple architectures on a single machine.  E.g. 64bit and 32bit on a 64bit Ubuntu.  Here are the specs:

https://wiki.ubuntu.com/MultiarchSpec
http://wiki.debian.org/ReleaseGoals/MultiArch

Unlike issue 1294959 this bug simply covers building Python and its stdlib extension modules on such systems.  For example, libsqlite3.so is no longer in /usr/lib but instead in /usr/lib/x86_64-linux-gnu on Ubuntu 11.04.  This means that a number of extension modules which depend on these 3rd party shared libraries simply won't build, because setup.py is too naive about the paths it searches.

The fix is fairly simple; you have to call out to dpkg-architecture to get the value of the platform's architecture and append that to /usr/lib and /usr/include for setup.py's search paths.  See the attached branch and patch for a candidate fix.

Because this does not introduce a new feature, it simply fixes the build process for changes to already supported platforms, I would like to apply this to Python 2.5 through 3.3.

Python 2.5 is not open for bug fixes anymore, so this can't be applied to this branch. I suggest that Python 2.6 is closed for bug fixes as well.

On Mar 31, 2011, at 10:08 PM, Martin v. Löwis wrote:

>Martin v. Löwis <martin@v.loewis.de> added the comment:
>
>Python 2.5 is not open for bug fixes anymore, so this can't be applied to
>this branch. I suggest that Python 2.6 is closed for bug fixes as well.

Although I'd still like to apply it to 2.5, I defer to you as RM.  However,
if/when I ever need to do a 2.6 release I'll need to build and test it and I
don't want to have to keep an old version of Ubuntu around to do that.

So how about if I apply it to 2.6, 2.7, 3.1 - 3.3?

(I've also addressed the comment you made in the review.)

If Ubuntu stops supporting to build old Python releases, I rather consider this a serious bug in Debian, not one in Python.

If the issue is merely that certain extension modules fail to build, I see no real reason to do anything about it. Users affected by this issue can still edit Modules/Setup.

"As I see it, the patch is uncontroversial for 3.3, 3.2, and 2.7.  And it definitely will not be applied to 3.0.  That leaves 2.5, 2.6, and 3.1.  If you really care one way or the other, please register your vote in the tracker."

2.5: +0
2.6: +1
3.1: +1

I’m not opposed to the change.

2.5: -1
2.6: -0
3.1: +0

As I see it, a large part of the "security fixes only" rule is for the benefit of folks auditing those security fixes, as it means there's very little noise in the branch to confuse the matter.

Since I do automated module testing against all Python versions,
my vote would be:

2.5: +1
2.6: +1
3.1: +1


This with the caveat that of course Martin has to decide for 2.5.
I'm just indicating my preference.

New changeset 7582a78f573b by Barry Warsaw in branch '3.1':
Issue 11715: Build extension modules on multiarch Debian and Ubuntu by
http://hg.python.org/cpython/rev/7582a78f573b

New changeset 867937dd2279 by Barry Warsaw in branch '3.2':
Issue 11715: Merge multiarch fix from 3.1 branch.
http://hg.python.org/cpython/rev/867937dd2279

New changeset 3f00611c3daf by Barry Warsaw in branch 'default':
Issue 11715: Merge multiarch fix from 3.1 branch.
http://hg.python.org/cpython/rev/3f00611c3daf

The FreeBSD and Solaris bots are failing:

dpkg-architecture: not found
error: build/temp.freebsd-8.2-RELEASE-amd64-3.3-pydebug/multiarch: No such file or directory
[62607 refs]
*** Error code 1


find_executable.patch should solve the problem.

Stefan, thanks for the patch.  The problem is that on FreeBSD and Solaris, if the command fails, I/O redirection does not create the file, whereas on Linux and OS X it does.  So I was going to wrap the os.remove() of the temp file in a try/except.  But I like your patch better because it avoids the dpkg-architecture call in the first place on systems that don't have it, albeit at the cost of traversing $PATH.

New changeset c8738114b962 by Barry Warsaw in branch '3.1':
Refinement by Stefan Krah (see issue 11715, msg133194) to exit early if the
http://hg.python.org/cpython/rev/c8738114b962

New changeset 3d7c9b38fbfd by Barry Warsaw in branch '3.2':
Refinement by Stefan Krah (see issue 11715, msg133194) to exit early if the
http://hg.python.org/cpython/rev/3d7c9b38fbfd

New changeset bbfc65d05588 by Barry Warsaw in branch 'default':
Refinement by Stefan Krah (see issue 11715, msg133194) to exit early if the
http://hg.python.org/cpython/rev/bbfc65d05588

New changeset bd0f73a9538e by Barry Warsaw in branch '2.7':
Backport for Python 2.7 of issue 11715 support for building Python on
http://hg.python.org/cpython/rev/bd0f73a9538e

I'm not sure this is 100% fixed. After dist-upgrading the Kubuntu VM on my netbook and updating to the latest Py3k code, I got a lot of test errors, even after a make distclean and ./configure.

The errors went away after manually tweaking LDFLAGS as Christian describes here:
http://lipyrary.blogspot.com/2011/05/how-to-compile-python-on-ubuntu-1104.html

On Aug 18, 2011, at 07:09 AM, Nick Coghlan wrote:

>I'm not sure this is 100% fixed. After dist-upgrading the Kubuntu VM on my
>netbook and updating to the latest Py3k code, I got a lot of test errors,
>even after a make distclean and ./configure.

Hi Nick.  Would this be Kubuntu 11.04 or some other release?

Hey Nick and Barry,

the fix in http://hg.python.org/cpython/rev/bd0f73a9538e isn't sufficient. You have added /usr/lib/MULTIARCH and /usr/include/MULTIARCH but you forgot to add /lib/MULTIARCH. On my system zlib is installed at /lib/x86_64-linux-gnu/libz.so.

Update:

It turns out that zlib1g-dev adds a symlink from /usr/lib/x86_64-linux-gnu/libz.so to /lib/x86_64-linux-gnu/libz.so.1 .

$ locate libz.
/lib/x86_64-linux-gnu/libz.so.1
/lib/x86_64-linux-gnu/libz.so.1.2.3.4
/usr/lib/x86_64-linux-gnu/libz.a
/usr/lib/x86_64-linux-gnu/libz.so

Perhaps this symlink is missing on Nick's installation. It might be a wise idea to add /lib/MULTIARCH to the library search paths, too.

It wouldn't surprise me at all if the laptop's links were a little off - I started with a Kubuntu image off VMWare's site quite some time ago, then dist-upgraded it through a couple of releases as they came out.

On Sep 12, 2011, at 12:34 PM, Nick Coghlan wrote:

>It wouldn't surprise me at all if the laptop's links were a little off - I
>started with a Kubuntu image off VMWare's site quite some time ago, then
>dist-upgraded it through a couple of releases as they came out.

I'll try to get a VM up with the latest Oneiric Kubuntu image and see what
happens.

I see this requires dpkg-architecture which isn't always available. While it isn't hard to install it, it isn't very clear that this is the cause of the nis and crypt modules failing to build on a fresh install of 11.10.

What would be nice is if there were some way of determining this information without dpkg-architecture such as reading it from /etc/ld.so.conf.d/x86_64-linux-gnu.conf.

about searching /lib/<multiarch>: adding this directory won't help. all .a and .so files are installed in /usr/lib or /usr/lib/<multiarch>.

about the missing dpkg-architecture: see the attached ma.diff patch. the Debian/Ubuntu system compilers add an option -print-multiarch, which can be used to get the multiarch name without having the dpkg-dev package installed.

New changeset 5966c206654b by doko in branch 'default':
- Issue #11715: Fix multiarch detection without having Debian development
http://hg.python.org/cpython/rev/5966c206654b

Today I got the same problem on 3.2 too.
I made a distclean and recompiled and got:

Python build finished, but the necessary bits to build these modules were not found:
_curses            _curses_panel      _dbm
_gdbm              _sqlite3           _ssl
_tkinter           bz2                readline

I tried to apply ma.diff and now I get:

Python build finished, but the necessary bits to build these modules were not found:
_curses            _curses_panel      _dbm            
_gdbm              _tkinter           bz2

2.7 is affected too:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _curses            _curses_panel   
_sqlite3           _ssl               _tkinter        
bsddb185           bz2                dbm             
gdbm               readline           sunaudiodev     

And after applying ma.diff:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _curses            _curses_panel   
_tkinter           bsddb185           bz2             
dbm                gdbm               sunaudiodev

I think that the ma.diff can safely go to the 2.7 and 3.2 branches.

For the other extensions which are not built you are probably missing the build dependencies (try: apt-get build-dep python2.7 python3.2).

The other missing extensions are not a problem, as long as the one that I need and already have (e.g. readline, _ssl) are built correctly.

New changeset 53fa224b95f4 by doko in branch '2.7':
- Issue #11715: Fix multiarch detection without having Debian development
http://hg.python.org/cpython/rev/53fa224b95f4

New changeset 78aba41a8105 by doko in branch '3.2':
- Issue #11715: Fix multiarch detection without having Debian development
http://hg.python.org/cpython/rev/78aba41a8105

fixed for 2.7 and 3.2 as well.