This is something that's becoming more and more of a
problem as more people get systems that run both 32-bit
and 64-bit code.  There are patches (926209 858809) in
the tracker to resolve this issue, I don't know exactly
what state they are in.  They seem fairly old, one is
closed and one is still open.

The Fedora RPMs include the following patches for lib64
(attached).

Any thoughts on what we need to do to allow building
lib64 as a part of the standard release?  Or do we just
want to wait for these terrible transition days to end
and rely on 32+64 packagers to deal with it?

Here's a short run-down of the situation:  Some systems
can run both 32 and 64 bit software.  On these systems
you *CAN* install a native 64-bit tool-chain, but some
people set up both 32 and 64 bit software on these
systems for compatibility.  You can do 32+64 in a
couple of ways, one is to set up a "chroot" environment
for the secondary (typically 32-bit) set of libraries
and programs.  The other is to run the system with
64-bit libraries in /usr/lib64 and 32-bit in /usr/lib
(because most software that needs compatibility would
be running against /usr/lib, not /usr/lib32).

It's a kludge to be sure, but if you need any legacy
software that you do not have source for, this is a way
of getting 64-bits while still retaining the ability to
run 32-bit  This is not a problem for systems which
only run 64-bit code, nor is it a problem for these
32+64 systems which are running only 64-bit
distributions on them.

Thoughts?
Sean

The problem with 64 bit builds should be discussed before we release
2.6. Another bug day topic.

Can someone update the priority so that this is looked at before the 2.6 
release?

Placing the entire library tree in /usr/lib64 is wasteful on dual 
32/64bit installation, but placing just the C modules there is contrary 
to python import logic and may cause problems to relative imports.

I have suggested what I believed was a workable solution: have 64-bit 
python search lib64-dynload subdirectories instead of lib-dynload.

See http://mail.python.org/pipermail/python-dev/2007-April/072653.html

Currently $(prefix)/pythonX.Y/lib-dynload is inserted in the sys.path, 
but I think it would be better to handle this inside the importer in a 
way similar to how the importer looks for both foo.so and foomodule.so 
when importing foo. This would allow submodules and user modules treated  
the same way.

Similar problem report: http://bugs.python.org/issue1019715

3rd party C modules are put in site-packages,
so just having importer of 64-bit python look at lib64-dynload is not
enough for solving this.

To work around this problem, I did some hacks on my local Python to look
at lib and lib64. It worked, but just as belopolsky said, this is
wasteful and ugly.

for completenes, here's a patch that's in use in SUSE. it's advantage
over Fedora's is that it works on both 32bit and 64bit installs

both patches assume that everybody uses lib64 for 64bit libs, which is
not true for Debian/Ubuntu. Even the FHS doesn't mandate the use of lib64.

well in our patch, at least, the directory is governed by sys.lib which
is defined through configure.
i don't understand the configure language well enough, but i'd assume
that making it parametrized isn't too hard?

I think this is duplicate of issue858809.

If I understand correctly, using lib32 or lib64 is a kludge. Debian
and Ubuntu want to come up with a better way to do this:
http://wiki.debian.org/ReleaseGoals/MultiArch
https://wiki.ubuntu.com/MultiarchSpec

Kind regards.

thanks for the info; however, it still means that python needs to learn
to live in places other than "/usr/lib"

jan matejek wrote:
> 
> jan matejek <jmatejek@suse.cz> added the comment:
> 
> thanks for the info; however, it still means that python needs to learn
> to live in places other than "/usr/lib"

The main problem is that Python's configuration system is not
geared up to having the lib directories for platform dependent
and platform independent parts use different names.

It currently only supports using different path *prefixes* for
such setups (--prefix and --exec-prefix), e.g. /usr and /usr64
would work just fine. It doesn't follow --libdir.

Note that Tarek is currently working on a cleanup of the
installation schemes (see distutils/commands/install.py,
distutils/sysconfig.py and site.py) which will then also
be used by site.py to setup sys.path.

A new modules will unite all these settings, so this should
be the target of any patches regarding installation and
path lookup schemes.

BTW: The "sys.lib" setting mentioned on the tracker is not standard.
The naming also doesn't look right, since the name of the "lib"
directory path component is an OS feature, not a system one that
you configure. os.lib_dir would be more appropriate.

Adding Tarek to the ticket.

Please note another aspect of this problem will bite all Python developers on Ubuntu 11.04.  With the introduction of multiarch, not all stdlib Python extension modules can be built out of the box, as seen here:

https://bugs.launchpad.net/ubuntu/+source/db4.8/+bug/738213/comments/13

Ubuntu's source package was hacked to make things work, by calling out to dpkg-architecture and adding the resulting directories to library_dirs and include_dirs search paths.  That patch would obviously have to be modified at the very least to be robust on non-Debian/Ubuntu platforms.

I'm not sure what the right solution is for upstream.

Here's a fix that works for me on Ubuntu 11.04.

I should note that I'd love to backport this to Python 3.2, 3.1, 2.7 and 2.6 since none of them can build entirely now on multiarch systems.  Since it only affects search order in the build process, one could argue that it's not a new feature :).

Barry: does it allow to install Python into /usr/lib/whateverarch, or is it just a partial fix for something slightly unrelated to this issue?

A proper fix is to introduce sys.libdir, which would be controllable by --libdir=${value} option of `configure`. If --libdir=${value} is not passed, then sys.libdir would default to sys.prefix + "/lib".
sysconfig, distutils etc. would have to use sys.libdir.

On Mar 29, 2011, at 07:28 PM, Antoine Pitrou wrote:

>
>Antoine Pitrou <pitrou@free.fr> added the comment:
>
>Barry: does it allow to install Python into /usr/lib/whateverarch, or is it
>just a partial fix for something slightly unrelated to this issue?

Antoine, you're right that the problem and fix I'm talking about is probably
different than the original bug report.  I really should create a new issue,
since mine isn't about building multiarch for Python, but instead just being
able to build Python on a multiarch system.  My problem is thankfully much
simpler.

I retract my patch for this bug because the issue described here is actually different than the one I want to fix.  See issue 11715 for the problem of building Python on multiarch Debian and Ubuntu (e.g. Ubuntu 11.04).

On Mar 29, 2011, at 07:38 PM, Arfrever Frehtes Taifersar Arahesis wrote:

>A proper fix is to introduce sys.libdir, which would be controllable by
>--libdir=${value} option of `configure`. If --libdir=${value} is not passed,
>then sys.libdir would default to sys.prefix + "/lib".  sysconfig, distutils
>etc. would have to use sys.libdir.

Please note that I'm not interested (right now <wink>) in building Python
multiarch, but building Python *on* multiarch.  So I think simply adding the
right search paths to setup.py is the right way to go, and I've opened a
separate issue for it.

heh, that's easy, just add the multiarch id to the extension name ;-)

On 29.03.2011 21:28, Antoine Pitrou wrote:
> > Barry: does it allow to install Python into /usr/lib/whateverarch,
no, it looks for headers and libraries in more directories.  But really, this
whole testing for paths is wrong. Just use the compiler to search for headers
and libraries, no need to check these on your own.

> > or is it just a partial fix for something slightly unrelated to this issue?

IMO, unrelated to the original report.

On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:

>no, it looks for headers and libraries in more directories.  But really, this
>whole testing for paths is wrong. Just use the compiler to search for headers
>and libraries, no need to check these on your own.

You're probably right about that, but reimplementing Python's build system is
out of scope for right now. ;)  For one thing, doing so wouldn't allow me to
backport to older Pythons, which I really want to do.

On Mar 29, 2011, at 10:11 PM, Matthias Klose wrote:

>heh, that's easy, just add the multiarch id to the extension name ;-)

Clever! :)

I currently think that sys.libdir should be only basename of libdir (e.g. "lib" or "lib64") to allow to easily use it with something else than sys.prefix.

On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:
> no, it looks for headers and libraries in more directories.  But really, this
> whole testing for paths is wrong. Just use the compiler to search for headers
> and libraries, no need to check these on your own.

Do all compilers provide this info, including Windows ones?  If so, that would be a nice feature for distutils2.

Arfrever Frehtes Taifersar Arahesis (Arfrever) on 2012-05-12 19:00
> I currently think that sys.libdir should be only basename of libdir (e.g.
> "lib" or "lib64") to allow to easily use it with something else than sys.prefix.

With the new sysconfig module I don’t think we need to clutter sys with another attribute.

Éric Araujo wrote:
> 
> Éric Araujo <merwok@netwok.org> added the comment:
> 
> On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:
>> no, it looks for headers and libraries in more directories.  But really, this
>> whole testing for paths is wrong. Just use the compiler to search for headers
>> and libraries, no need to check these on your own.
> 
> Do all compilers provide this info, including Windows ones?  If so, that would be a nice feature for distutils2.

This only works for a handful of system library paths, not the extra
ones that you may need to search for local installations of
libraries and which you have to inform the compiler about :-)

Many gcc installations, for example, don't include the /usr/local
or /opt/local dir trees in the search. On Windows, you have to
run the correct vc*.bat files to have the paths setup and optional
software rarely adds the correct paths to LIB and INCLUDE.

The compiler also won't help with the problem Sean originally
pointed to: building software on systems that can run both
32-bit and 64-bit and finding the right set of libs to
link at.

Another problem is finding the paths to the right version of a
library (both include files and corresponding libraries).

While it would be great to have a system tool take care of setting
things up correctly, I don't know of any such tool, so searching
paths and inspecting files using REs appears to be the only way
to build a general purpose detection scheme.

mxSetup.py (included in egenix-mx-base) uses such a scheme, distutils
has one too.

Is this issue still relevant?

Yes. We still have a patch for two things:
1. fix setup.py script to look for libraries in correct (lib64) prefixes, and
2. fix values returned from sysconfig, to reflect that python resides in lib64

"$prefix/lib" is hardcoded in many places. Lib64 is probably not going away anytime soon, so it would be nice if this was solved once and for all ;)
The good thing is that with sysconfig, we don't have to do much beyond teaching sysconfig about the right values.

To reiterate, our current solution is to introduce "sys.lib" (and "sys.arch", but that is never used anymore) that is either "lib" or "lib64", and use this in place of the string "lib" wherever appropriate. We find the value for sys.lib through configure magic.

the patch in msg202343 is wrong, hardcoding lib64 on Debian/Ubuntu. At least the configure check should check for lib64 as a directory and not a symlink, and only then default to lib64.

two other issues with the patch:

 - I would like to see any new OS-dependent locations in the sysconfig module, not the sys module.

 - Please don't depend on uname for the autoconf check, but on the gnu host triplet.

 - Please don't add another `arch' attribute to sys. We already have enough of these in sysconfig.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dne 7.11.2013 16:56, Matthias Klose napsal(a):
> 
> Matthias Klose added the comment:
> 
> the patch in msg202343 is wrong, hardcoding lib64 on Debian/Ubuntu.

This patch is provided for reference only - it works for us at SUSE. I'll be
happy to spend some time improving it for general usage, if this has any
chance of being commited.

> At least the configure check should check for lib64 as a directory and not
> a symlink, and only then default to lib64.

Maybe this should be detected differently altogether. Perhaps by working with
LIBDIR, which is known to configure?

> 
> two other issues with the patch:
> 
> - I would like to see any new OS-dependent locations in the sysconfig
> module, not the sys module.

how would you propose to put the value into sysconfig in the first place? It
seems to rely heavily on existing attributes from sys.

> 
> - Please don't depend on uname for the autoconf check, but on the gnu host
> triplet.
> 
> - Please don't add another `arch' attribute to sys. We already have enough
> of these in sysconfig.

We don't use this one anymore, and i'm not entirely sure that we ever did. I
am happy to drop it.

> 
> ----------
> 
> _______________________________________ Python tracker
> <report@bugs.python.org> <http://bugs.python.org/issue1294959> 
> _______________________________________
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSe7pdAAoJEIskb84DCy7LvwcP/2n74K2XDsRu7K6OV9S4SzDa
v7vpDVhAgTBQlHglY+wavUQU2WLBlGyVEk2xHDV8WdI4zU7rAbn7XAW5URxznctq
t/Ptvt0IsDAqONrF8ezg8/eTUkcP3nV2Hk90RNe0gliDH6uc0wekKUZzVaTObO1L
3vM8XfEtTQstmK1VxQVpYolUPZm8n7Fe8NEPA6A8bu8CU736cg+wWdbDrr6Mjowo
OuO4b56J1P3BIQkBcOLe3mH20Bv8O03P9iNADwYHUOayvgthFWCmoDzh0Y1dQa9/
ynT+G9BuYyXOli6Yr15W0L8OFU+nwxByK81lEClz6UonCvoStnEWnXIN3JYW15Yb
rNwb5HKNmKB16yx/RuV3WCvlKbg6ziMlfWGW6qTA1g0P0ivU+sRVQXv5gI8NHcQ9
/4jmaUh7Dr1T4KHujI57Z99kLQHvSlHEM3v4aT96IZNaPghkA+e7TjhMdHmvYtQz
YoY75FIy0xVStdXzw8zbM2LVlKp8vxncrjYbuzJYgG1jiYwmF6gDoztXRVP/zwei
PypIiui4QaQc32V5dCwpQYpAvpgCVHm6sGSO0HbLWSUK71M8f1YU2BPwvglHb6jh
N9tmYfmahvQSyIFOZdb4C6HLlzEezENdYYIf7oMW07z8SOOLU+8eKP13dp6NWINP
HCSU34LLYTfwnQ+27aMk
=raBO
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Nov 07, 2013, at 04:05 PM, jan matejek wrote:

>> - I would like to see any new OS-dependent locations in the sysconfig
>> module, not the sys module.
>
>how would you propose to put the value into sysconfig in the first place? It
>seems to rely heavily on existing attributes from sys.

Actually, I think it mostly parses these values from the installed Makefile.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.15 (GNU/Linux)

iQIcBAEBCAAGBQJSe8T6AAoJEBJutWOnSwa/6oAQAJM3IksGydXk9CEp7rbSrHea
DvYUccmyon1xrmu5RjWnz6ZbJdWlFwx8ouFpbzmZvfAF8E2m0HliNzW+/w28sik5
F37p5/7hScQ2x/AcmdnVrzDotMkvcvMILsCDIhSy/nPIWkI4hAuRGLdPJmgiE3HE
b3hjQDCgsVDrp8arJioepx6xWSMpE1dQ/KsL6DDuNl3rUNVR1E1jTYEqY12+fOjc
Mcgz6lVndyZreY958iQE/UR7MKoW5lcLnin44PtMYrizwWv82kgwhkqU5tubnTBD
m9tD16uCz7xvATM5XI8nmXeLcLMSMfUdaG+4ny//cIjDKYIC6XXoZvCgE7iSEws1
pEIQCmrs6mpk6d83Yz/XDXc4OqjqC+tUPY1TGNqAI/nm416uoKUuO/f1eU397EV+
2RqJqev8Ho8Sgk7skFJGwcCfTO4yDR40+0wm3u2BiM9bTcnGiJaC7z2TAp9eb4Qs
jo/cmYi3BbqPu9Xx3P4oX11NHmjTPBUcZjqsJa8w8q3lf9r5haE5EqlLaNgDnGtL
efu7OMom2yQHXdwIJ2efmefjoby812uNFSbTiMvDxZTVCCUyBczBT/Q7gu/4S9Ks
Mv3oY1bkq6qAXKKOwzKoblzHJ6VW+A3Rn15Lh6Tb2kj1pTbdS9fFJASWr6CprjWi
XbCdez4dMhd+PGwgxHs6
=/r/D
-----END PGP SIGNATURE-----

On Nov 07, 2013, at 03:40 PM, jan matejek wrote:

>To reiterate, our current solution is to introduce "sys.lib" (and "sys.arch",
>but that is never used anymore) that is either "lib" or "lib64", and use this
>in place of the string "lib" wherever appropriate. We find the value for
>sys.lib through configure magic.

PEP 421 added sys.implementation, which contains provisions for
implementation-specific additions.  So a better place to put these
non-standard values is sys.implementation._lib and sys.implementation._arch,
either instead of or in addition to sysconfig variables.

I disagree about sys.implementation. It's useless and wrong for cross builds.  Please use sysconfig instead. What sysconfig is maybe missing is a set of variables which you can rely on.

configure.ac should accept an option, which allows to set any custom libdir.

Examples (architecture: libdir) in Gentoo:
x32:       libx32
mips o32:  lib
mips n32:  lib32
mips n64:  lib64

On Nov 07, 2013, at 04:56 PM, Matthias Klose wrote:

>I disagree about sys.implementation. It's useless and wrong for cross builds.
>Please use sysconfig instead. What sysconfig is maybe missing is a set of
>variables which you can rely on.

Agreed that sysconfig is a better place for more general values.  My point was
that if OpenSUSE wants to carry deltas that are specific only to its platform,
then sys.implementation._<whatever> is the standard place to put it.

Is there agreement on what needs doing here?  I'd like to see this into 3.5 before it reaches its 10th birthday :)

Also see #15631

at this again, when porting SUSE patches to 3.6.0 :) ( :( )

Last time there was a discussion, Barry suggested using sysconfig variables to find the proper libdir. Trouble is, to fill out the variables, sysconfig itself uses two sources:
a) compiled-in information from the binary, and
b) Makefile items
Makefile seems an obvious location for storing the libdir info, except, well, to FIND the makefile in the first place, we are using sysconfig variables.... you see where this is heading.

So, given that sysconfig is the correct Primary Source for libdir info, we have three options to get it there:
1. compile it into the binary. sys.implementation.platlibdir seems as good as any -- i'm not completely convinced that this is a property of the _implementation_, but, well.
2. modify sysconfig.py at build time
3. guess it from the environment

And options (2) and (3) seem wrong, so i'm going with (1) for now

The downstream patch we currently use in Fedora [0].

[0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/plain/00102-lib64.patch

Attached is a patch that I'd like to propose for inclusion.

It introduces a new configure option "--with-custom-platlibdir=<name>", which defaults to `basename $libdir`. This is converted to makefile variable "platlibdir", which is used in getpath.c to generate value of the lib_python variable.

sysconfig and distutils.sysconfig retrieve the variable from makefile (i learned about existence of _sysconfigdata so that is where the variable now lives) to correctly substitute in posix_prefix (and unix_prefix for distutils install) scheme.

Sysconfig is then used in pydoc and trace module, instead of locally calculating paths. Perhaps distutils.sysconfig should also use more of sysconfig instead of duplicating the functionality?

As it stands, the python stdlib is installed into /usr/$platlibdir, because I changed SCRIPTDIR to $libdir instead of $prefix/lib. Maybe this should be also separately configurable?
In any case, third-party modules get installed either into /usr/lib or /usr/$platlibdir depending on whether they are arch-independent or not. Hence the final modification in site.py that adds /usr/lib to search path on systems where $platlibdir != "lib".

This caused test_sysconfig to fail because it checks that posix_user and posix_prefix schemes generate "similar" paths. I have left the posix_user scheme alone, so all its paths are based on "lib", where the posix_prefix scheme uses $platlibdir. Maybe posix_user scheme should be modified as well?
Anyway, for now, I have accounted for the changes in the test.

What now?
(please don't make me write a PEP please don't make me write a PEP please don't make......)

I don't think we need a PEP for this, however I would like to name the new macro libsubdir, or sublibdir, to make it clear it's just the libdir component. Or make the value absolute, like it already is for libdir ...

On Jan 18, 2017, at 03:57 PM, Matthias Klose wrote:

>I don't think we need a PEP for this

Correct.  Generally we don't need PEPs for build system changes.

Pinging here. Is there some way to push the issue forward?

the patch looks ok. was it tested on a Debian or Ubuntu system to produce the same layout with this patch and without the new configure option?

I closed bpo-858809 as a duplicate of this issue.

I marked bpo-15631 as a duplicate of this issue.

I marked bpo-34032 as a duplicate of this issue. It has an open PR written by Matěj Cepl (OpenSUSE): PR 8068. It looks similar to PR 3698.

See also bpo-34058 "Default Python 3.7 install broken on openSUSE Leap 42.3: $PYTHONHOME/lib64/python3.7/lib-dynload/ not linked to $PYTHONHOME/lib/python3.7/lib-dynload/": OpenSUSE and Centos are mentioned as being impacted.

See also bpo-14791: "setup.py only adds /prefix/lib, not /prefix/lib64".

I marked bpo-18092 ("Python 2.7.5 installation broken on OpenSuse 12.2") as a duplicate of this issue.

See also bpo-21016: "trace: $prefix and $exec_prefix improperly replaced on Fedora".

New changeset 4fac7ed43ebf1771a8fe86fdfe7b9991f3be78cd by Victor Stinner in branch 'master':
bpo-21016: pydoc and trace use sysconfig (GH-18476)
https://github.com/python/cpython/commit/4fac7ed43ebf1771a8fe86fdfe7b9991f3be78cd

New changeset ca133e53fafdec1aa77613fcb7558deed959383f by Miss Islington (bot) in branch '3.7':
bpo-21016: pydoc and trace use sysconfig (GH-18476)
https://github.com/python/cpython/commit/ca133e53fafdec1aa77613fcb7558deed959383f

New changeset ac6f4d2db703c0ff88e496bcb7b7fe55cf2ac458 by Miss Islington (bot) in branch '3.8':
bpo-21016: pydoc and trace use sysconfig (GH-18476)
https://github.com/python/cpython/commit/ac6f4d2db703c0ff88e496bcb7b7fe55cf2ac458

Summary:

* This issue is mostly about pushing a 16-years old patch from Fedora and SuSE to Python upstream. Other Linux distributions use different solutions to support multiple architectures in the same filesystem. For example, Debian and Ubuntu uses "Multiarch" which continues to use /usr/lib/ for 32-bit and 64-bit x86 architecture (x86 and x86-64).

* I pushed the non controversial part of PR 3698: fix pydoc and trace modules to use sysconfig to get the stdlib path

* CI tests pass on PR 18381.

* Matthias Klose asked to not add another sys attribute, but Jan Matějek explained that the option value is needed to import the sysconfig module. Moreover, the option value is needed in the site module and we are trying to avoid "import sysconfig" in site since it would slowdown Python startup time.

* I chose "./configure --with-python-libdir" and "sys.python_libdir" names. PR 3698 uses "./configure --with-custom-platsubdir" and sysconfig.get_config_var('platsubdir') names. Other proposed names: sys.lib, os.lib_dir. It was proposed to drop "custom" from "./configure --with-custom-platsubdir" option name.

Python use many paths. The sysconfig module is a good starting point to discover them all in the _INSTALL_SCHEMES directories. PR 18381 changes the following paths:

* posix_prefix: stdlib, platstdlib, platlib
* posix_user: stdlib, platstdlib, platlib


Examples on Fedora:

* pure Python: /usr/lib64/python3.7/os.py
* C extension: /usr/lib64/python3.7/lib-dynload/_multiprocessing.cpython-37m-x86_64-linux-gnu.so

Examples on SuSE:

* pure Python: /usr/lib64/python3.7/os.py

Ubuntu 16.04:

* pure Python: /usr/lib/python3.5/os.py
* C extension: /usr/lib/python3.5/lib-dynload/_multiprocessing.cpython-35m-x86_64-linux-gnu.so

Default with prefix=/usr:

* posix_prefix:

  * stdlib: '/usr/lib/python3.9'
  * platstdlib: '/usr/lib/python3.9'
  * platlib: '/usr/lib/python3.9/site-packages'
 
* posix_user:

  * stdlib: '~/.local/lib/python3.9'
  * platstdlib: '~/.local/lib/python3.9'
  * platlib: '~/.local/lib/python3.9/site-packages'

Customized with prefix=/usr and python_libdir=lib64, PR 18381:

* posix_prefix:

  * stdlib: '/usr/lib64/python3.9'
  * platstdlib: '/usr/lib64/python3.9'
  * platlib: '/usr/lib64/python3.9/site-packages'

* posix_user:

  * stdlib: '~/.local/lib64/python3.9'
  * platstdlib: '~/.local/lib64/python3.9'
  * platlib: '~/.local/lib64/python3.9/site-packages'

> posix_user:
> 
> * stdlib: '~/.local/lib64/python3.9'
> * platstdlib: '~/.local/lib64/python3.9'
> * platlib: '~/.local/lib64/python3.9/site-packages'

I am still not sure about the this one. I know that I have mentioned a possibility of multiplatform $HOME on NFS, but it seems too far fetched idea to break the current not-completely-standard.

Also note that the extension modules have archful suffix, so they will not collide in $HOME, unless they are installed as name.so.

> posix_user:
> 
> * stdlib: '~/.local/lib64/python3.9'
> * platstdlib: '~/.local/lib64/python3.9'
> * platlib: '~/.local/lib64/python3.9/site-packages'

Honestly, I have no idea how posix_user is used.

~/.local/lib64/ directory doesn't exist on my work machine, whereas I'm installed various Python modules and applications.

In practice, ~/.local/lib/ is used on Fedora, even if posix_user scheme uses ~/.local/lib64/. Examples:

* pure Python: ~/.local/lib/python3.7/site-packages/pip/__main__.py
* C extension: ~/.local/lib/python3.7/site-packages/typed_ast/_ast3.cpython-37m-x86_64-linux-gnu.so

So changing posix_user should have no impact on end users.


I also tested a legacy "python3 setup.py install" to install a C extension, it also lands into ~/.local/lib:

~/.local/lib/python3.7/site-packages/test_cext-0.0.0-py3.7-linux-x86_64.egg/test_cext.cpython-37m-x86_64-linux-gnu.so

The setup.py uses setuptools.


Miro:
> Also note that the extension modules have archful suffix, so they will not collide in $HOME, unless they are installed as name.so.

Yeah, it seems like pip install adopted Debian multiarch naming: "x86_64-linux-gnu" triplet in "_ast3.cpython-37m-x86_64-linux-gnu.so" filename.

> > Also note that the extension modules have archful suffix, so they will not collide in $HOME, unless they are installed as name.so.
> 
> Yeah, it seems like pip install adopted Debian multiarch naming: "x86_64-linux-gnu" triplet in "_ast3.cpython-37m-x86_64-linux-gnu.so" filename.

For reference: issue22980

SuSE currently uses this patch for lib64 in the Python 3 package:
https://build.opensuse.org/package/view_file/devel:languages:python:Factory/python3/F00102-lib64.patch?expand=1

Ooops, SuSE has 2 patches.

SuSE currently uses these patches for lib64 in the Python 3 package:

(1) https://build.opensuse.org/package/view_file/devel:languages:python:Factory/python3/F00102-lib64.patch?expand=1

(2) https://build.opensuse.org/package/view_file/devel:languages:python:Factory/python3/SUSE-FEDORA-multilib.patch?expand=1

lib64_tests.py: test suite hardcoded for Fedora, SuSE and PR 18381 with --with-python-libdir=lib64. I wrote it to help me to understand differences between Fedora, SuSE and PR 18381.

On Fedora, the test fails because we have an additional patch on distutils to install packages in /usr/local instead of /usr when using pip:

======================================================================
FAIL: test_distutils_install (__main__.Lib64Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "lib64_tests.py", line 81, in test_distutils_install
    self.assertEqual(cmd.install_platlib, PLATLIB)
AssertionError: '/usr/local/lib64/python3.7/site-packages' != '/usr/lib64/python3.7/site-packages'
- /usr/local/lib64/python3.7/site-packages
?     ------
+ /usr/lib64/python3.7/site-packages

Links on Debian Multiarch:

* https://wiki.debian.org/Multiarch/HOWTO
* https://lwn.net/Articles/711199/

Why "sys.python_libdir"?  Isn't that too public? A lot of similar path-related info is in sysconfig; why shouldn't it be there?
How does "sys.python_libdir" work on other platforms?

> Why "sys.python_libdir"?  Isn't that too public? A lot of similar path-related info is in sysconfig; why shouldn't it be there?

I replied in a previous comment:

"Matthias Klose asked to not add another sys attribute, but Jan Matějek explained that the option value is needed to import the sysconfig module. Moreover, the option value is needed in the site module and we are trying to avoid "import sysconfig" in site since it would slowdown Python startup time."


> How does "sys.python_libdir" work on other platforms?

On Windows, the variable is equal to "lib". It's equal to "lib" on all platforms, except if you use configure to override its value.

Does it reply to your question?

See also bpo-24871: "freeze.py doesn't work on x86_64 Linux out of the box".

> On 64-bit Linux freeze.py uses lib instead of lib64 when constructing
> path to Makefile etc. Using sysconfig fixes this issue.

lib64_tests-2.py: Updated test for PR 18381. It now uses sys.platlibdir and it tests also Makefile LIBPL variable.

lib64_tests-3.py: Updated script to handle different ABI flags (release vs debug mode).

New changeset 8510f430781118d9b603c3a2f06945d6ebc5fe42 by Victor Stinner in branch 'master':
bpo-1294959: Add sys.platlibdir attribute (GH-18381)
https://github.com/python/cpython/commit/8510f430781118d9b603c3a2f06945d6ebc5fe42

I chose to exclude setup.py changes from my commit 8510f430781118d9b603c3a2f06945d6ebc5fe42, whereas the Fedora downstream patch has a few changes:
https://src.fedoraproject.org/rpms/python39/blob/master/f/00102-lib64.patch

See also bpo-14791: "setup.py only adds /prefix/lib, not /prefix/lib64".

diff --git a/setup.py b/setup.py
index 51e67fe4a5..bafa0bf99a 100644
--- a/setup.py
+++ b/setup.py
@@ -649,7 +649,7 @@ class PyBuildExt(build_ext):
         # directories (i.e. '.' and 'Include') must be first.  See issue
         # 10520.
         if not CROSS_COMPILING:
-            add_dir_to_list(self.compiler.library_dirs, '/usr/local/lib')
+            add_dir_to_list(self.compiler.library_dirs, '/usr/local/lib64')
             add_dir_to_list(self.compiler.include_dirs, '/usr/local/include')
         # only change this for cross builds for 3.3, issues on Mageia
         if CROSS_COMPILING:
@@ -955,11 +955,11 @@ class PyBuildExt(build_ext):
             elif curses_library:
                 readline_libs.append(curses_library)
             elif self.compiler.find_library_file(self.lib_dirs +
-                                                     ['/usr/lib/termcap'],
+                                                     ['/usr/lib64/termcap'],
                                                      'termcap'):
                 readline_libs.append('termcap')
             self.add(Extension('readline', ['readline.c'],
-                               library_dirs=['/usr/lib/termcap'],
+                               library_dirs=['/usr/lib64/termcap'],
                                extra_link_args=readline_extra_link_args,
                                libraries=readline_libs))
         else:

Another related issue is bpo-34058: "Default Python 3.7 install broken on openSUSE Leap 42.3: $PYTHONHOME/lib64/python3.7/lib-dynload/ not linked to $PYTHONHOME/lib/python3.7/lib-dynload/".

> Another related issue is bpo-34058: "Default Python 3.7 install broken on openSUSE Leap 42.3: $PYTHONHOME/lib64/python3.7/lib-dynload/ not linked to $PYTHONHOME/lib/python3.7/lib-dynload/".

I marked bpo-34058 as a duplicate of this issue.

New changeset 3c29675d8736cb1860f342d872ae41e342c6d383 by Raúl Cumplido in branch 'master':
bpo-1294959: Fix typo for new attribute platlibdir. (GH-18960)
https://github.com/python/cpython/commit/3c29675d8736cb1860f342d872ae41e342c6d383

Miro Hrončok ran rpmdiff to compare the Fedora with the old downstream patches and with the new upstream commit 8510f430781118d9b603c3a2f06945d6ebc5fe42: compare the Python package with and without setup.py changes.

https://src.fedoraproject.org/rpms/python39/pull-request/31#comment-39392

In short, there is no difference. Moreover, Miro Hrončok, Matej Cepl and me don't know the rationale of these setup.py changes.

I propose to drop them and wait until someone complains. If someone complains, we would know the rationale and so have a good reason to have these setup.py changes.

For readline: Fedora libreadline is already linked to libtinfo and so setup.py doesn't go up to `self.compiler.find_library_file(self.lib_dirs + ['/usr/lib/termcap'], 'termcap')` code path. This code is deadcode on Fedora.

For /usr/local/lib vs /usr/local/lib64: I'm not sure why /usr/local/lib is used in the first place. Fedora installs libraries in /usr, not in /usr/local. Maybe it's there for users who install dependencies without Fedora: by installed them manually in /usr/local. Again, since it's unclear how /usr/local is supposed to be used, I prefer to leave it unchanged.

I closed the PR 18917.

Morevoer, if it becomes an issue in Fedora, we can start with a downstream patch again, to see how it goes, before going directly upstream.

I close this very old issue (15 years old!). Thank you *very much* to everybody who was involved in helping to get it fixed!

If I forgot someone, comment this issue or open a new one.

There is one last minor issue, but there is already a dedicated issue: bpo-24871 "freeze.py doesn't work on x86_64 Linux out of the box".

Does this mean that platforms using /usr/lib64 for shared libraries are now forced to install Python into /usr/lib64/python*?

> Does this mean that platforms using /usr/lib64 for shared libraries are now forced to install Python into /usr/lib64/python*?

Not at all. This means that it is possible to do so. It remains optional.

> Not at all. This means that it is possible to do so. It remains optional.

...but then sys.platlibdir is going to incorrectly list 'lib', isn't it?  According to https://docs.python.org/3.9/library/sys.html#sys.platlibdir it's used 'to build the path of platform-specific dynamic libraries and the path of the standard library'.  This means that people relying on it for the former will get 32-bit libraries instead.

If you don't use the new --with-platlibdir configuration option, Python 3.9 behaves exactly as Python 3.8: there is zero change.

Python 3.9 only builds a different sys.path if sys.platlibdir is different than "lib".

Can we clarify the wording to clearly indicate it's to be used only for Python modules/extensions and not system dynamic libs?

> Can we clarify the wording to clearly indicate it's to be used only for Python modules/extensions and not system dynamic libs?

Sure. Can you please propose a different wording? English is not my first language.

I created bpo-40684: make install doesn't respect configure --with-platlibdir=lib64.

> Can you please propose a different wording? English is not my first language.

Mine neither but I'll try.  How about:

'Name of the platform-specific library directory. It is used to build the path of the standard library and C extension modules of the standard library.'

Can you propose a PR with your doc enhancement? I suggest to start with updating https://docs.python.org/dev/library/sys.html#sys.platlibdir documentation.

Ok, it seems that I misunderstood it at first.  Judging by the code, it also affects extensions installed into site-packages, so I've tried to make that clear and add one more example to the bullet list.

New changeset 242d95659b6b4ff4fb54b58a30454dafa311d4e9 by Michał Górny in branch 'master':
bpo-1294959: Try to clarify the meaning of platlibdir (GH-20332)
https://github.com/python/cpython/commit/242d95659b6b4ff4fb54b58a30454dafa311d4e9

New changeset a5936ad6323b2cda712726a25d9301f766146cff by Miss Islington (bot) in branch '3.9':
bpo-1294959: Try to clarify the meaning of platlibdir (GH-20332)
https://github.com/python/cpython/commit/a5936ad6323b2cda712726a25d9301f766146cff

Thank you!

Note: https://docs.python.org/3.9/install/index.html probably needs some update wrt this change.

Oh, it seems like one of the changes merged in this issue broke the following use case if platlibdir is different than "lib":

    export PYTHONHOME=<Prefix>
    export PYTHONPATH=<Prefix>/lib/python3.9
    python3.9

See bpo-40854 "Allow overriding sys.platlibdir: add PYTHONPLATLIBDIR env var".

I marked bpo-40505 as a duplicate of this issue.

Follow-up: #44860 is removing platlibdir from posix_user scheme