This is something that's becoming more and more of a
problem as more people get systems that run both 32-bit
and 64-bit code. There are patches (926209 858809) in
the tracker to resolve this issue, I don't know exactly
what state they are in. They seem fairly old, one is
closed and one is still open.

The Fedora RPMs include the following patches for lib64
(attached).

Any thoughts on what we need to do to allow building
lib64 as a part of the standard release? Or do we just
want to wait for these terrible transition days to end
and rely on 32+64 packagers to deal with it?

Here's a short run-down of the situation: Some systems
can run both 32 and 64 bit software. On these systems
you *CAN* install a native 64-bit tool-chain, but some
people set up both 32 and 64 bit software on these
systems for compatibility. You can do 32+64 in a
couple of ways, one is to set up a "chroot" environment
for the secondary (typically 32-bit) set of libraries
and programs. The other is to run the system with
64-bit libraries in /usr/lib64 and 32-bit in /usr/lib
(because most software that needs compatibility would
be running against /usr/lib, not /usr/lib32).

It's a kludge to be sure, but if you need any legacy
software that you do not have source for, this is a way
of getting 64-bits while still retaining the ability to
run 32-bit This is not a problem for systems which
only run 64-bit code, nor is it a problem for these
32+64 systems which are running only 64-bit
distributions on them.

Thoughts?
Sean

The problem with 64 bit builds should be discussed before we release
2.6. Another bug day topic.

Can someone update the priority so that this is looked at before the 2.6
release?

Placing the entire library tree in /usr/lib64 is wasteful on dual
32/64bit installation, but placing just the C modules there is contrary
to python import logic and may cause problems to relative imports.

I have suggested what I believed was a workable solution: have 64-bit
python search lib64-dynload subdirectories instead of lib-dynload.

See http://mail.python.org/pipermail/python-dev/2007-April/072653.html

Currently $(prefix)/pythonX.Y/lib-dynload is inserted in the sys.path,
but I think it would be better to handle this inside the importer in a
way similar to how the importer looks for both foo.so and foomodule.so
when importing foo. This would allow submodules and user modules treated
the same way.

Similar problem report: http://bugs.python.org/issue1019715

3rd party C modules are put in site-packages,
so just having importer of 64-bit python look at lib64-dynload is not
enough for solving this.

To work around this problem, I did some hacks on my local Python to look
at lib and lib64. It worked, but just as belopolsky said, this is
wasteful and ugly.

for completenes, here's a patch that's in use in SUSE. it's advantage
over Fedora's is that it works on both 32bit and 64bit installs

both patches assume that everybody uses lib64 for 64bit libs, which is
not true for Debian/Ubuntu. Even the FHS doesn't mandate the use of lib64.

well in our patch, at least, the directory is governed by sys.lib which
is defined through configure.
i don't understand the configure language well enough, but i'd assume
that making it parametrized isn't too hard?

I think this is duplicate of bpo-858809.

If I understand correctly, using lib32 or lib64 is a kludge. Debian
and Ubuntu want to come up with a better way to do this:
http://wiki.debian.org/ReleaseGoals/MultiArch
https://wiki.ubuntu.com/MultiarchSpec

Kind regards.

thanks for the info; however, it still means that python needs to learn
to live in places other than "/usr/lib"

jan matejek wrote:

jan matejek <jmatejek@suse.cz> added the comment:

thanks for the info; however, it still means that python needs to learn
to live in places other than "/usr/lib"

The main problem is that Python's configuration system is not
geared up to having the lib directories for platform dependent
and platform independent parts use different names.

It currently only supports using different path *prefixes* for
such setups (--prefix and --exec-prefix), e.g. /usr and /usr64
would work just fine. It doesn't follow --libdir.

Note that Tarek is currently working on a cleanup of the
installation schemes (see distutils/commands/install.py,
distutils/sysconfig.py and site.py) which will then also
be used by site.py to setup sys.path.

A new modules will unite all these settings, so this should
be the target of any patches regarding installation and
path lookup schemes.

BTW: The "sys.lib" setting mentioned on the tracker is not standard.
The naming also doesn't look right, since the name of the "lib"
directory path component is an OS feature, not a system one that
you configure. os.lib_dir would be more appropriate.

Adding Tarek to the ticket.

Please note another aspect of this problem will bite all Python developers on Ubuntu 11.04. With the introduction of multiarch, not all stdlib Python extension modules can be built out of the box, as seen here:

https://bugs.launchpad.net/ubuntu/+source/db4.8/+bug/738213/comments/13

Ubuntu's source package was hacked to make things work, by calling out to dpkg-architecture and adding the resulting directories to library_dirs and include_dirs search paths. That patch would obviously have to be modified at the very least to be robust on non-Debian/Ubuntu platforms.

I'm not sure what the right solution is for upstream.

Here's a fix that works for me on Ubuntu 11.04.

I should note that I'd love to backport this to Python 3.2, 3.1, 2.7 and 2.6 since none of them can build entirely now on multiarch systems. Since it only affects search order in the build process, one could argue that it's not a new feature :).

Barry: does it allow to install Python into /usr/lib/whateverarch, or is it just a partial fix for something slightly unrelated to this issue?

A proper fix is to introduce sys.libdir, which would be controllable by --libdir=${value} option of configure. If --libdir=${value} is not passed, then sys.libdir would default to sys.prefix + "/lib".
sysconfig, distutils etc. would have to use sys.libdir.

On Mar 29, 2011, at 07:28 PM, Antoine Pitrou wrote:

Antoine Pitrou <pitrou@free.fr> added the comment:

Barry: does it allow to install Python into /usr/lib/whateverarch, or is it
just a partial fix for something slightly unrelated to this issue?

Antoine, you're right that the problem and fix I'm talking about is probably
different than the original bug report. I really should create a new issue,
since mine isn't about building multiarch for Python, but instead just being
able to build Python on a multiarch system. My problem is thankfully much
simpler.

I retract my patch for this bug because the issue described here is actually different than the one I want to fix. See bpo-11715 for the problem of building Python on multiarch Debian and Ubuntu (e.g. Ubuntu 11.04).

On Mar 29, 2011, at 07:38 PM, Arfrever Frehtes Taifersar Arahesis wrote:

A proper fix is to introduce sys.libdir, which would be controllable by
--libdir=${value} option of configure. If --libdir=${value} is not passed,
then sys.libdir would default to sys.prefix + "/lib". sysconfig, distutils
etc. would have to use sys.libdir.

Please note that I'm not interested (right now <wink>) in building Python
multiarch, but building Python *on* multiarch. So I think simply adding the
right search paths to setup.py is the right way to go, and I've opened a
separate issue for it.

heh, that's easy, just add the multiarch id to the extension name ;-)

On 29.03.2011 21:28, Antoine Pitrou wrote:

> Barry: does it allow to install Python into /usr/lib/whateverarch,
no, it looks for headers and libraries in more directories. But really, this
whole testing for paths is wrong. Just use the compiler to search for headers
and libraries, no need to check these on your own.

> or is it just a partial fix for something slightly unrelated to this issue?

IMO, unrelated to the original report.

On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:

no, it looks for headers and libraries in more directories. But really, this
whole testing for paths is wrong. Just use the compiler to search for headers
and libraries, no need to check these on your own.

You're probably right about that, but reimplementing Python's build system is
out of scope for right now. ;) For one thing, doing so wouldn't allow me to
backport to older Pythons, which I really want to do.

On Mar 29, 2011, at 10:11 PM, Matthias Klose wrote:

heh, that's easy, just add the multiarch id to the extension name ;-)

Clever! :)

lib64_tests-3.py: Updated script to handle different ABI flags (release vs debug mode).

New changeset 8510f43 by Victor Stinner in branch 'master':
bpo-1294959: Add sys.platlibdir attribute (GH-18381)
8510f43

I chose to exclude setup.py changes from my commit 8510f43, whereas the Fedora downstream patch has a few changes:
https://src.fedoraproject.org/rpms/python39/blob/master/f/00102-lib64.patch

See also bpo-14791: "setup.py only adds /prefix/lib, not /prefix/lib64".

diff --git a/setup.py b/setup.py
index 51e67fe4a5..bafa0bf99a 100644
--- a/setup.py
+++ b/setup.py
@@ -649,7 +649,7 @@ class PyBuildExt(build_ext):
         # directories (i.e. '.' and 'Include') must be first.  See issue
         # 10520.
         if not CROSS_COMPILING:
-            add_dir_to_list(self.compiler.library_dirs, '/usr/local/lib')
+            add_dir_to_list(self.compiler.library_dirs, '/usr/local/lib64')
             add_dir_to_list(self.compiler.include_dirs, '/usr/local/include')
         # only change this for cross builds for 3.3, issues on Mageia
         if CROSS_COMPILING:
@@ -955,11 +955,11 @@ class PyBuildExt(build_ext):
             elif curses_library:
                 readline_libs.append(curses_library)
             elif self.compiler.find_library_file(self.lib_dirs +
-                                                     ['/usr/lib/termcap'],
+                                                     ['/usr/lib64/termcap'],
                                                      'termcap'):
                 readline_libs.append('termcap')
             self.add(Extension('readline', ['readline.c'],
-                               library_dirs=['/usr/lib/termcap'],
+                               library_dirs=['/usr/lib64/termcap'],
                                extra_link_args=readline_extra_link_args,
                                libraries=readline_libs))
         else:

Another related issue is bpo-34058: "Default Python 3.7 install broken on openSUSE Leap 42.3: $PYTHONHOME/lib64/python3.7/lib-dynload/ not linked to $PYTHONHOME/lib/python3.7/lib-dynload/".

Another related issue is bpo-34058: "Default Python 3.7 install broken on openSUSE Leap 42.3: $PYTHONHOME/lib64/python3.7/lib-dynload/ not linked to $PYTHONHOME/lib/python3.7/lib-dynload/".

I marked bpo-34058 as a duplicate of this issue.

New changeset 3c29675 by Raúl Cumplido in branch 'master':
bpo-1294959: Fix typo for new attribute platlibdir. (GH-18960)
3c29675

Miro Hrončok ran rpmdiff to compare the Fedora with the old downstream patches and with the new upstream commit 8510f43: compare the Python package with and without setup.py changes.

https://src.fedoraproject.org/rpms/python39/pull-request/31#comment-39392

In short, there is no difference. Moreover, Miro Hrončok, Matej Cepl and me don't know the rationale of these setup.py changes.

I propose to drop them and wait until someone complains. If someone complains, we would know the rationale and so have a good reason to have these setup.py changes.

For readline: Fedora libreadline is already linked to libtinfo and so setup.py doesn't go up to self.compiler.find_library_file(self.lib_dirs + ['/usr/lib/termcap'], 'termcap') code path. This code is deadcode on Fedora.

For /usr/local/lib vs /usr/local/lib64: I'm not sure why /usr/local/lib is used in the first place. Fedora installs libraries in /usr, not in /usr/local. Maybe it's there for users who install dependencies without Fedora: by installed them manually in /usr/local. Again, since it's unclear how /usr/local is supposed to be used, I prefer to leave it unchanged.

I closed the PR 18917.

Morevoer, if it becomes an issue in Fedora, we can start with a downstream patch again, to see how it goes, before going directly upstream.

I close this very old issue (15 years old!). Thank you *very much* to everybody who was involved in helping to get it fixed!

If I forgot someone, comment this issue or open a new one.

There is one last minor issue, but there is already a dedicated issue: bpo-24871 "freeze.py doesn't work on x86_64 Linux out of the box".

Does this mean that platforms using /usr/lib64 for shared libraries are now forced to install Python into /usr/lib64/python*?

Does this mean that platforms using /usr/lib64 for shared libraries are now forced to install Python into /usr/lib64/python*?

Not at all. This means that it is possible to do so. It remains optional.

Not at all. This means that it is possible to do so. It remains optional.

...but then sys.platlibdir is going to incorrectly list 'lib', isn't it? According to https://docs.python.org/3.9/library/sys.html#sys.platlibdir it's used 'to build the path of platform-specific dynamic libraries and the path of the standard library'. This means that people relying on it for the former will get 32-bit libraries instead.

If you don't use the new --with-platlibdir configuration option, Python 3.9 behaves exactly as Python 3.8: there is zero change.

Python 3.9 only builds a different sys.path if sys.platlibdir is different than "lib".

Can we clarify the wording to clearly indicate it's to be used only for Python modules/extensions and not system dynamic libs?

Can we clarify the wording to clearly indicate it's to be used only for Python modules/extensions and not system dynamic libs?

Sure. Can you please propose a different wording? English is not my first language.

I created bpo-40684: make install doesn't respect configure --with-platlibdir=lib64.

Can you please propose a different wording? English is not my first language.

Mine neither but I'll try. How about:

'Name of the platform-specific library directory. It is used to build the path of the standard library and C extension modules of the standard library.'

Can you propose a PR with your doc enhancement? I suggest to start with updating https://docs.python.org/dev/library/sys.html#sys.platlibdir documentation.

Ok, it seems that I misunderstood it at first. Judging by the code, it also affects extensions installed into site-packages, so I've tried to make that clear and add one more example to the bullet list.

New changeset 242d956 by Michał Górny in branch 'master':
bpo-1294959: Try to clarify the meaning of platlibdir (GH-20332)
242d956

New changeset a5936ad by Miss Islington (bot) in branch '3.9':
bpo-1294959: Try to clarify the meaning of platlibdir (GH-20332)
a5936ad

Thank you!

Note: https://docs.python.org/3.9/install/index.html probably needs some update wrt this change.

Oh, it seems like one of the changes merged in this issue broke the following use case if platlibdir is different than "lib":

export PYTHONHOME=<Prefix>
export PYTHONPATH=<Prefix>/lib/python3.9
python3.9

See bpo-40854 "Allow overriding sys.platlibdir: add PYTHONPLATLIBDIR env var".

I marked bpo-40505 as a duplicate of this issue.

Follow-up: bpo-44860 is removing platlibdir from posix_user scheme