This is something that's becoming more and more of a
problem as more people get systems that run both 32-bit
and 64-bit code.  There are patches (926209 858809) in
the tracker to resolve this issue, I don't know exactly
what state they are in.  They seem fairly old, one is
closed and one is still open.

The Fedora RPMs include the following patches for lib64
(attached).

Any thoughts on what we need to do to allow building
lib64 as a part of the standard release?  Or do we just
want to wait for these terrible transition days to end
and rely on 32+64 packagers to deal with it?

Here's a short run-down of the situation:  Some systems
can run both 32 and 64 bit software.  On these systems
you *CAN* install a native 64-bit tool-chain, but some
people set up both 32 and 64 bit software on these
systems for compatibility.  You can do 32+64 in a
couple of ways, one is to set up a "chroot" environment
for the secondary (typically 32-bit) set of libraries
and programs.  The other is to run the system with
64-bit libraries in /usr/lib64 and 32-bit in /usr/lib
(because most software that needs compatibility would
be running against /usr/lib, not /usr/lib32).

It's a kludge to be sure, but if you need any legacy
software that you do not have source for, this is a way
of getting 64-bits while still retaining the ability to
run 32-bit  This is not a problem for systems which
only run 64-bit code, nor is it a problem for these
32+64 systems which are running only 64-bit
distributions on them.

Thoughts?
Sean

The problem with 64 bit builds should be discussed before we release
2.6. Another bug day topic.

Can someone update the priority so that this is looked at before the 2.6 
release?

Placing the entire library tree in /usr/lib64 is wasteful on dual 
32/64bit installation, but placing just the C modules there is contrary 
to python import logic and may cause problems to relative imports.

I have suggested what I believed was a workable solution: have 64-bit 
python search lib64-dynload subdirectories instead of lib-dynload.

See http://mail.python.org/pipermail/python-dev/2007-April/072653.html

Currently $(prefix)/pythonX.Y/lib-dynload is inserted in the sys.path, 
but I think it would be better to handle this inside the importer in a 
way similar to how the importer looks for both foo.so and foomodule.so 
when importing foo. This would allow submodules and user modules treated  
the same way.

Similar problem report: http://bugs.python.org/issue1019715

3rd party C modules are put in site-packages,
so just having importer of 64-bit python look at lib64-dynload is not
enough for solving this.

To work around this problem, I did some hacks on my local Python to look
at lib and lib64. It worked, but just as belopolsky said, this is
wasteful and ugly.

for completenes, here's a patch that's in use in SUSE. it's advantage
over Fedora's is that it works on both 32bit and 64bit installs

both patches assume that everybody uses lib64 for 64bit libs, which is
not true for Debian/Ubuntu. Even the FHS doesn't mandate the use of lib64.

well in our patch, at least, the directory is governed by sys.lib which
is defined through configure.
i don't understand the configure language well enough, but i'd assume
that making it parametrized isn't too hard?

I think this is duplicate of issue858809.

If I understand correctly, using lib32 or lib64 is a kludge. Debian
and Ubuntu want to come up with a better way to do this:
http://wiki.debian.org/ReleaseGoals/MultiArch
https://wiki.ubuntu.com/MultiarchSpec

Kind regards.

thanks for the info; however, it still means that python needs to learn
to live in places other than "/usr/lib"

jan matejek wrote:
> 
> jan matejek <jmatejek@suse.cz> added the comment:
> 
> thanks for the info; however, it still means that python needs to learn
> to live in places other than "/usr/lib"

The main problem is that Python's configuration system is not
geared up to having the lib directories for platform dependent
and platform independent parts use different names.

It currently only supports using different path *prefixes* for
such setups (--prefix and --exec-prefix), e.g. /usr and /usr64
would work just fine. It doesn't follow --libdir.

Note that Tarek is currently working on a cleanup of the
installation schemes (see distutils/commands/install.py,
distutils/sysconfig.py and site.py) which will then also
be used by site.py to setup sys.path.

A new modules will unite all these settings, so this should
be the target of any patches regarding installation and
path lookup schemes.

BTW: The "sys.lib" setting mentioned on the tracker is not standard.
The naming also doesn't look right, since the name of the "lib"
directory path component is an OS feature, not a system one that
you configure. os.lib_dir would be more appropriate.

Adding Tarek to the ticket.

Please note another aspect of this problem will bite all Python developers on Ubuntu 11.04.  With the introduction of multiarch, not all stdlib Python extension modules can be built out of the box, as seen here:

https://bugs.launchpad.net/ubuntu/+source/db4.8/+bug/738213/comments/13

Ubuntu's source package was hacked to make things work, by calling out to dpkg-architecture and adding the resulting directories to library_dirs and include_dirs search paths.  That patch would obviously have to be modified at the very least to be robust on non-Debian/Ubuntu platforms.

I'm not sure what the right solution is for upstream.

Here's a fix that works for me on Ubuntu 11.04.

I should note that I'd love to backport this to Python 3.2, 3.1, 2.7 and 2.6 since none of them can build entirely now on multiarch systems.  Since it only affects search order in the build process, one could argue that it's not a new feature :).

Barry: does it allow to install Python into /usr/lib/whateverarch, or is it just a partial fix for something slightly unrelated to this issue?

A proper fix is to introduce sys.libdir, which would be controllable by --libdir=${value} option of `configure`. If --libdir=${value} is not passed, then sys.libdir would default to sys.prefix + "/lib".
sysconfig, distutils etc. would have to use sys.libdir.

On Mar 29, 2011, at 07:28 PM, Antoine Pitrou wrote:

>
>Antoine Pitrou <pitrou@free.fr> added the comment:
>
>Barry: does it allow to install Python into /usr/lib/whateverarch, or is it
>just a partial fix for something slightly unrelated to this issue?

Antoine, you're right that the problem and fix I'm talking about is probably
different than the original bug report.  I really should create a new issue,
since mine isn't about building multiarch for Python, but instead just being
able to build Python on a multiarch system.  My problem is thankfully much
simpler.

I retract my patch for this bug because the issue described here is actually different than the one I want to fix.  See issue 11715 for the problem of building Python on multiarch Debian and Ubuntu (e.g. Ubuntu 11.04).

On Mar 29, 2011, at 07:38 PM, Arfrever Frehtes Taifersar Arahesis wrote:

>A proper fix is to introduce sys.libdir, which would be controllable by
>--libdir=${value} option of `configure`. If --libdir=${value} is not passed,
>then sys.libdir would default to sys.prefix + "/lib".  sysconfig, distutils
>etc. would have to use sys.libdir.

Please note that I'm not interested (right now <wink>) in building Python
multiarch, but building Python *on* multiarch.  So I think simply adding the
right search paths to setup.py is the right way to go, and I've opened a
separate issue for it.

heh, that's easy, just add the multiarch id to the extension name ;-)

On 29.03.2011 21:28, Antoine Pitrou wrote:
> > Barry: does it allow to install Python into /usr/lib/whateverarch,
no, it looks for headers and libraries in more directories.  But really, this
whole testing for paths is wrong. Just use the compiler to search for headers
and libraries, no need to check these on your own.

> > or is it just a partial fix for something slightly unrelated to this issue?

IMO, unrelated to the original report.

On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:

>no, it looks for headers and libraries in more directories.  But really, this
>whole testing for paths is wrong. Just use the compiler to search for headers
>and libraries, no need to check these on your own.

You're probably right about that, but reimplementing Python's build system is
out of scope for right now. ;)  For one thing, doing so wouldn't allow me to
backport to older Pythons, which I really want to do.

On Mar 29, 2011, at 10:11 PM, Matthias Klose wrote:

>heh, that's easy, just add the multiarch id to the extension name ;-)

Clever! :)

I currently think that sys.libdir should be only basename of libdir (e.g. "lib" or "lib64") to allow to easily use it with something else than sys.prefix.

On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:
> no, it looks for headers and libraries in more directories.  But really, this
> whole testing for paths is wrong. Just use the compiler to search for headers
> and libraries, no need to check these on your own.

Do all compilers provide this info, including Windows ones?  If so, that would be a nice feature for distutils2.

Arfrever Frehtes Taifersar Arahesis (Arfrever) on 2012-05-12 19:00
> I currently think that sys.libdir should be only basename of libdir (e.g.
> "lib" or "lib64") to allow to easily use it with something else than sys.prefix.

With the new sysconfig module I don’t think we need to clutter sys with another attribute.

Éric Araujo wrote:
> 
> Éric Araujo <merwok@netwok.org> added the comment:
> 
> On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:
>> no, it looks for headers and libraries in more directories.  But really, this
>> whole testing for paths is wrong. Just use the compiler to search for headers
>> and libraries, no need to check these on your own.
> 
> Do all compilers provide this info, including Windows ones?  If so, that would be a nice feature for distutils2.

This only works for a handful of system library paths, not the extra
ones that you may need to search for local installations of
libraries and which you have to inform the compiler about :-)

Many gcc installations, for example, don't include the /usr/local
or /opt/local dir trees in the search. On Windows, you have to
run the correct vc*.bat files to have the paths setup and optional
software rarely adds the correct paths to LIB and INCLUDE.

The compiler also won't help with the problem Sean originally
pointed to: building software on systems that can run both
32-bit and 64-bit and finding the right set of libs to
link at.

Another problem is finding the paths to the right version of a
library (both include files and corresponding libraries).

While it would be great to have a system tool take care of setting
things up correctly, I don't know of any such tool, so searching
paths and inspecting files using REs appears to be the only way
to build a general purpose detection scheme.

mxSetup.py (included in egenix-mx-base) uses such a scheme, distutils
has one too.