classification
Title: Problems with /usr/lib64 builds.
Type: behavior Stage: patch review
Components: Build Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, akitada, andybuckley, barry, belopolsky, carlos.velasco, catalin.iacob, christian.heimes, cstratak, doko, eitan.adler, eric.araujo, hroncok, ivazquez, jafo, jcea, lemburg, matejcik, mcepl, miss-islington, petr.viktorin, pillarsdotnet, piotr.dobrogost, tarek, vstinner
Priority: normal Keywords: patch

Created on 2005-09-19 03:05 by jafo, last changed 2020-02-20 15:53 by petr.viktorin.

Files
File name Uploaded Description Edit
python-2.3.4-lib64-regex.patch jafo, 2005-09-19 03:05 One of the Fedora patches.
python-2.4.1-lib64.patch jafo, 2005-09-19 03:06 Another of the Fedora patches.
japanese-codecs-lib64.patch jafo, 2005-09-19 03:06 The third obviously lib64 Fedora patch.
Python-2.6.2-multilib.patch matejcik, 2009-08-14 16:27 SUSE patch for the lib64 issue
Python-3.3.0b2-multilib.patch matejcik, 2013-11-07 15:40 SUSE patch in python3
python-3.6.0-multilib-new.patch matejcik, 2017-01-16 11:20
lib64_tests.py vstinner, 2020-02-12 17:50
Pull Requests
URL Status Linked Edit
PR 3698 closed matejcik, 2017-09-22 16:45
PR 11755 open cstratak, 2019-02-04 15:09
PR 18381 open vstinner, 2020-02-06 17:17
PR 18476 merged vstinner, 2020-02-12 09:14
PR 18482 merged miss-islington, 2020-02-12 12:02
PR 18483 merged miss-islington, 2020-02-12 12:02
Repositories containing patches
http://hg.python.org/users/barry#multiarch
Messages (68)
msg26314 - (view) Author: Sean Reifschneider (jafo) * (Python committer) Date: 2005-09-19 03:05
This is something that's becoming more and more of a
problem as more people get systems that run both 32-bit
and 64-bit code.  There are patches (926209 858809) in
the tracker to resolve this issue, I don't know exactly
what state they are in.  They seem fairly old, one is
closed and one is still open.

The Fedora RPMs include the following patches for lib64
(attached).

Any thoughts on what we need to do to allow building
lib64 as a part of the standard release?  Or do we just
want to wait for these terrible transition days to end
and rely on 32+64 packagers to deal with it?

Here's a short run-down of the situation:  Some systems
can run both 32 and 64 bit software.  On these systems
you *CAN* install a native 64-bit tool-chain, but some
people set up both 32 and 64 bit software on these
systems for compatibility.  You can do 32+64 in a
couple of ways, one is to set up a "chroot" environment
for the secondary (typically 32-bit) set of libraries
and programs.  The other is to run the system with
64-bit libraries in /usr/lib64 and 32-bit in /usr/lib
(because most software that needs compatibility would
be running against /usr/lib, not /usr/lib32).

It's a kludge to be sure, but if you need any legacy
software that you do not have source for, this is a way
of getting 64-bits while still retaining the ability to
run 32-bit  This is not a problem for systems which
only run 64-bit code, nor is it a problem for these
32+64 systems which are running only 64-bit
distributions on them.

Thoughts?
Sean
msg59310 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-01-05 19:33
The problem with 64 bit builds should be discussed before we release
2.6. Another bug day topic.
msg64592 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-03-27 14:11
Can someone update the priority so that this is looked at before the 2.6 
release?
msg64593 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-03-27 14:31
Placing the entire library tree in /usr/lib64 is wasteful on dual 
32/64bit installation, but placing just the C modules there is contrary 
to python import logic and may cause problems to relative imports.

I have suggested what I believed was a workable solution: have 64-bit 
python search lib64-dynload subdirectories instead of lib-dynload.

See http://mail.python.org/pipermail/python-dev/2007-April/072653.html

Currently $(prefix)/pythonX.Y/lib-dynload is inserted in the sys.path, 
but I think it would be better to handle this inside the importer in a 
way similar to how the importer looks for both foo.so and foomodule.so 
when importing foo. This would allow submodules and user modules treated  
the same way.
msg81099 - (view) Author: Akira Kitada (akitada) * Date: 2009-02-03 21:23
Similar problem report: http://bugs.python.org/issue1019715
msg82929 - (view) Author: Akira Kitada (akitada) * Date: 2009-02-28 17:32
3rd party C modules are put in site-packages,
so just having importer of 64-bit python look at lib64-dynload is not
enough for solving this.

To work around this problem, I did some hacks on my local Python to look
at lib and lib64. It worked, but just as belopolsky said, this is
wasteful and ugly.
msg91559 - (view) Author: jan matejek (matejcik) * Date: 2009-08-14 16:27
for completenes, here's a patch that's in use in SUSE. it's advantage
over Fedora's is that it works on both 32bit and 64bit installs
msg91769 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2009-08-20 11:01
both patches assume that everybody uses lib64 for 64bit libs, which is
not true for Debian/Ubuntu. Even the FHS doesn't mandate the use of lib64.
msg91781 - (view) Author: jan matejek (matejcik) * Date: 2009-08-20 18:29
well in our patch, at least, the directory is governed by sys.lib which
is defined through configure.
i don't understand the configure language well enough, but i'd assume
that making it parametrized isn't too hard?
msg92541 - (view) Author: Akira Kitada (akitada) * Date: 2009-09-12 12:46
I think this is duplicate of issue858809.
msg94941 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2009-11-05 19:23
If I understand correctly, using lib32 or lib64 is a kludge. Debian
and Ubuntu want to come up with a better way to do this:
http://wiki.debian.org/ReleaseGoals/MultiArch
https://wiki.ubuntu.com/MultiarchSpec

Kind regards.
msg94982 - (view) Author: jan matejek (matejcik) * Date: 2009-11-06 16:26
thanks for the info; however, it still means that python needs to learn
to live in places other than "/usr/lib"
msg94986 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2009-11-06 17:27
jan matejek wrote:
> 
> jan matejek <jmatejek@suse.cz> added the comment:
> 
> thanks for the info; however, it still means that python needs to learn
> to live in places other than "/usr/lib"

The main problem is that Python's configuration system is not
geared up to having the lib directories for platform dependent
and platform independent parts use different names.

It currently only supports using different path *prefixes* for
such setups (--prefix and --exec-prefix), e.g. /usr and /usr64
would work just fine. It doesn't follow --libdir.

Note that Tarek is currently working on a cleanup of the
installation schemes (see distutils/commands/install.py,
distutils/sysconfig.py and site.py) which will then also
be used by site.py to setup sys.path.

A new modules will unite all these settings, so this should
be the target of any patches regarding installation and
path lookup schemes.

BTW: The "sys.lib" setting mentioned on the tracker is not standard.
The naming also doesn't look right, since the name of the "lib"
directory path component is an OS feature, not a system one that
you configure. os.lib_dir would be more appropriate.
msg94989 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2009-11-06 17:41
Adding Tarek to the ticket.
msg132424 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-03-28 21:29
Please note another aspect of this problem will bite all Python developers on Ubuntu 11.04.  With the introduction of multiarch, not all stdlib Python extension modules can be built out of the box, as seen here:

https://bugs.launchpad.net/ubuntu/+source/db4.8/+bug/738213/comments/13

Ubuntu's source package was hacked to make things work, by calling out to dpkg-architecture and adding the resulting directories to library_dirs and include_dirs search paths.  That patch would obviously have to be modified at the very least to be robust on non-Debian/Ubuntu platforms.

I'm not sure what the right solution is for upstream.
msg132503 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-03-29 19:11
Here's a fix that works for me on Ubuntu 11.04.
msg132505 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-03-29 19:19
I should note that I'd love to backport this to Python 3.2, 3.1, 2.7 and 2.6 since none of them can build entirely now on multiarch systems.  Since it only affects search order in the build process, one could argue that it's not a new feature :).
msg132506 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-03-29 19:28
Barry: does it allow to install Python into /usr/lib/whateverarch, or is it just a partial fix for something slightly unrelated to this issue?
msg132511 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2011-03-29 19:38
A proper fix is to introduce sys.libdir, which would be controllable by --libdir=${value} option of `configure`. If --libdir=${value} is not passed, then sys.libdir would default to sys.prefix + "/lib".
sysconfig, distutils etc. would have to use sys.libdir.
msg132535 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-03-29 21:39
On Mar 29, 2011, at 07:28 PM, Antoine Pitrou wrote:

>
>Antoine Pitrou <pitrou@free.fr> added the comment:
>
>Barry: does it allow to install Python into /usr/lib/whateverarch, or is it
>just a partial fix for something slightly unrelated to this issue?

Antoine, you're right that the problem and fix I'm talking about is probably
different than the original bug report.  I really should create a new issue,
since mine isn't about building multiarch for Python, but instead just being
able to build Python on a multiarch system.  My problem is thankfully much
simpler.
msg132537 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-03-29 21:49
I retract my patch for this bug because the issue described here is actually different than the one I want to fix.  See issue 11715 for the problem of building Python on multiarch Debian and Ubuntu (e.g. Ubuntu 11.04).
msg132538 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-03-29 22:05
On Mar 29, 2011, at 07:38 PM, Arfrever Frehtes Taifersar Arahesis wrote:

>A proper fix is to introduce sys.libdir, which would be controllable by
>--libdir=${value} option of `configure`. If --libdir=${value} is not passed,
>then sys.libdir would default to sys.prefix + "/lib".  sysconfig, distutils
>etc. would have to use sys.libdir.

Please note that I'm not interested (right now <wink>) in building Python
multiarch, but building Python *on* multiarch.  So I think simply adding the
right search paths to setup.py is the right way to go, and I've opened a
separate issue for it.
msg132540 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2011-03-29 22:11
heh, that's easy, just add the multiarch id to the extension name ;-)
msg132542 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2011-03-29 22:12
On 29.03.2011 21:28, Antoine Pitrou wrote:
> > Barry: does it allow to install Python into /usr/lib/whateverarch,
no, it looks for headers and libraries in more directories.  But really, this
whole testing for paths is wrong. Just use the compiler to search for headers
and libraries, no need to check these on your own.

> > or is it just a partial fix for something slightly unrelated to this issue?

IMO, unrelated to the original report.
msg132544 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-03-29 22:15
On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:

>no, it looks for headers and libraries in more directories.  But really, this
>whole testing for paths is wrong. Just use the compiler to search for headers
>and libraries, no need to check these on your own.

You're probably right about that, but reimplementing Python's build system is
out of scope for right now. ;)  For one thing, doing so wouldn't allow me to
backport to older Pythons, which I really want to do.
msg132546 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-03-29 22:15
On Mar 29, 2011, at 10:11 PM, Matthias Klose wrote:

>heh, that's easy, just add the multiarch id to the extension name ;-)

Clever! :)
msg160482 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2012-05-12 17:00
I currently think that sys.libdir should be only basename of libdir (e.g. "lib" or "lib64") to allow to easily use it with something else than sys.prefix.
msg160671 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-05-14 22:14
On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:
> no, it looks for headers and libraries in more directories.  But really, this
> whole testing for paths is wrong. Just use the compiler to search for headers
> and libraries, no need to check these on your own.

Do all compilers provide this info, including Windows ones?  If so, that would be a nice feature for distutils2.

Arfrever Frehtes Taifersar Arahesis (Arfrever) on 2012-05-12 19:00
> I currently think that sys.libdir should be only basename of libdir (e.g.
> "lib" or "lib64") to allow to easily use it with something else than sys.prefix.

With the new sysconfig module I don’t think we need to clutter sys with another attribute.
msg160691 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2012-05-15 07:42
Éric Araujo wrote:
> 
> Éric Araujo <merwok@netwok.org> added the comment:
> 
> On Mar 29, 2011, at 10:12 PM, Matthias Klose wrote:
>> no, it looks for headers and libraries in more directories.  But really, this
>> whole testing for paths is wrong. Just use the compiler to search for headers
>> and libraries, no need to check these on your own.
> 
> Do all compilers provide this info, including Windows ones?  If so, that would be a nice feature for distutils2.

This only works for a handful of system library paths, not the extra
ones that you may need to search for local installations of
libraries and which you have to inform the compiler about :-)

Many gcc installations, for example, don't include the /usr/local
or /opt/local dir trees in the search. On Windows, you have to
run the correct vc*.bat files to have the paths setup and optional
software rarely adds the correct paths to LIB and INCLUDE.

The compiler also won't help with the problem Sean originally
pointed to: building software on systems that can run both
32-bit and 64-bit and finding the right set of libs to
link at.

Another problem is finding the paths to the right version of a
library (both include files and corresponding libraries).

While it would be great to have a system tool take care of setting
things up correctly, I don't know of any such tool, so searching
paths and inspecting files using REs appears to be the only way
to build a general purpose detection scheme.

mxSetup.py (included in egenix-mx-base) uses such a scheme, distutils
has one too.
msg202341 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-11-07 15:17
Is this issue still relevant?
msg202343 - (view) Author: jan matejek (matejcik) * Date: 2013-11-07 15:40
Yes. We still have a patch for two things:
1. fix setup.py script to look for libraries in correct (lib64) prefixes, and
2. fix values returned from sysconfig, to reflect that python resides in lib64

"$prefix/lib" is hardcoded in many places. Lib64 is probably not going away anytime soon, so it would be nice if this was solved once and for all ;)
The good thing is that with sysconfig, we don't have to do much beyond teaching sysconfig about the right values.

To reiterate, our current solution is to introduce "sys.lib" (and "sys.arch", but that is never used anymore) that is either "lib" or "lib64", and use this in place of the string "lib" wherever appropriate. We find the value for sys.lib through configure magic.
msg202354 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-11-07 15:56
the patch in msg202343 is wrong, hardcoding lib64 on Debian/Ubuntu. At least the configure check should check for lib64 as a directory and not a symlink, and only then default to lib64.

two other issues with the patch:

 - I would like to see any new OS-dependent locations in the sysconfig module, not the sys module.

 - Please don't depend on uname for the autoconf check, but on the gnu host triplet.

 - Please don't add another `arch' attribute to sys. We already have enough of these in sysconfig.
msg202359 - (view) Author: jan matejek (matejcik) * Date: 2013-11-07 16:05
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dne 7.11.2013 16:56, Matthias Klose napsal(a):
> 
> Matthias Klose added the comment:
> 
> the patch in msg202343 is wrong, hardcoding lib64 on Debian/Ubuntu.

This patch is provided for reference only - it works for us at SUSE. I'll be
happy to spend some time improving it for general usage, if this has any
chance of being commited.

> At least the configure check should check for lib64 as a directory and not
> a symlink, and only then default to lib64.

Maybe this should be detected differently altogether. Perhaps by working with
LIBDIR, which is known to configure?

> 
> two other issues with the patch:
> 
> - I would like to see any new OS-dependent locations in the sysconfig
> module, not the sys module.

how would you propose to put the value into sysconfig in the first place? It
seems to rely heavily on existing attributes from sys.

> 
> - Please don't depend on uname for the autoconf check, but on the gnu host
> triplet.
> 
> - Please don't add another `arch' attribute to sys. We already have enough
> of these in sysconfig.

We don't use this one anymore, and i'm not entirely sure that we ever did. I
am happy to drop it.

> 
> ----------
> 
> _______________________________________ Python tracker
> <report@bugs.python.org> <http://bugs.python.org/issue1294959> 
> _______________________________________
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSe7pdAAoJEIskb84DCy7LvwcP/2n74K2XDsRu7K6OV9S4SzDa
v7vpDVhAgTBQlHglY+wavUQU2WLBlGyVEk2xHDV8WdI4zU7rAbn7XAW5URxznctq
t/Ptvt0IsDAqONrF8ezg8/eTUkcP3nV2Hk90RNe0gliDH6uc0wekKUZzVaTObO1L
3vM8XfEtTQstmK1VxQVpYolUPZm8n7Fe8NEPA6A8bu8CU736cg+wWdbDrr6Mjowo
OuO4b56J1P3BIQkBcOLe3mH20Bv8O03P9iNADwYHUOayvgthFWCmoDzh0Y1dQa9/
ynT+G9BuYyXOli6Yr15W0L8OFU+nwxByK81lEClz6UonCvoStnEWnXIN3JYW15Yb
rNwb5HKNmKB16yx/RuV3WCvlKbg6ziMlfWGW6qTA1g0P0ivU+sRVQXv5gI8NHcQ9
/4jmaUh7Dr1T4KHujI57Z99kLQHvSlHEM3v4aT96IZNaPghkA+e7TjhMdHmvYtQz
YoY75FIy0xVStdXzw8zbM2LVlKp8vxncrjYbuzJYgG1jiYwmF6gDoztXRVP/zwei
PypIiui4QaQc32V5dCwpQYpAvpgCVHm6sGSO0HbLWSUK71M8f1YU2BPwvglHb6jh
N9tmYfmahvQSyIFOZdb4C6HLlzEezENdYYIf7oMW07z8SOOLU+8eKP13dp6NWINP
HCSU34LLYTfwnQ+27aMk
=raBO
-----END PGP SIGNATURE-----
msg202365 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-11-07 16:51
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Nov 07, 2013, at 04:05 PM, jan matejek wrote:

>> - I would like to see any new OS-dependent locations in the sysconfig
>> module, not the sys module.
>
>how would you propose to put the value into sysconfig in the first place? It
>seems to rely heavily on existing attributes from sys.

Actually, I think it mostly parses these values from the installed Makefile.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.15 (GNU/Linux)

iQIcBAEBCAAGBQJSe8T6AAoJEBJutWOnSwa/6oAQAJM3IksGydXk9CEp7rbSrHea
DvYUccmyon1xrmu5RjWnz6ZbJdWlFwx8ouFpbzmZvfAF8E2m0HliNzW+/w28sik5
F37p5/7hScQ2x/AcmdnVrzDotMkvcvMILsCDIhSy/nPIWkI4hAuRGLdPJmgiE3HE
b3hjQDCgsVDrp8arJioepx6xWSMpE1dQ/KsL6DDuNl3rUNVR1E1jTYEqY12+fOjc
Mcgz6lVndyZreY958iQE/UR7MKoW5lcLnin44PtMYrizwWv82kgwhkqU5tubnTBD
m9tD16uCz7xvATM5XI8nmXeLcLMSMfUdaG+4ny//cIjDKYIC6XXoZvCgE7iSEws1
pEIQCmrs6mpk6d83Yz/XDXc4OqjqC+tUPY1TGNqAI/nm416uoKUuO/f1eU397EV+
2RqJqev8Ho8Sgk7skFJGwcCfTO4yDR40+0wm3u2BiM9bTcnGiJaC7z2TAp9eb4Qs
jo/cmYi3BbqPu9Xx3P4oX11NHmjTPBUcZjqsJa8w8q3lf9r5haE5EqlLaNgDnGtL
efu7OMom2yQHXdwIJ2efmefjoby812uNFSbTiMvDxZTVCCUyBczBT/Q7gu/4S9Ks
Mv3oY1bkq6qAXKKOwzKoblzHJ6VW+A3Rn15Lh6Tb2kj1pTbdS9fFJASWr6CprjWi
XbCdez4dMhd+PGwgxHs6
=/r/D
-----END PGP SIGNATURE-----
msg202366 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-11-07 16:53
On Nov 07, 2013, at 03:40 PM, jan matejek wrote:

>To reiterate, our current solution is to introduce "sys.lib" (and "sys.arch",
>but that is never used anymore) that is either "lib" or "lib64", and use this
>in place of the string "lib" wherever appropriate. We find the value for
>sys.lib through configure magic.

PEP 421 added sys.implementation, which contains provisions for
implementation-specific additions.  So a better place to put these
non-standard values is sys.implementation._lib and sys.implementation._arch,
either instead of or in addition to sysconfig variables.
msg202367 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-11-07 16:56
I disagree about sys.implementation. It's useless and wrong for cross builds.  Please use sysconfig instead. What sysconfig is maybe missing is a set of variables which you can rely on.
msg202368 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2013-11-07 16:59
configure.ac should accept an option, which allows to set any custom libdir.

Examples (architecture: libdir) in Gentoo:
x32:       libx32
mips o32:  lib
mips n32:  lib32
mips n64:  lib64
msg202378 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-11-07 17:55
On Nov 07, 2013, at 04:56 PM, Matthias Klose wrote:

>I disagree about sys.implementation. It's useless and wrong for cross builds.
>Please use sysconfig instead. What sysconfig is maybe missing is a set of
>variables which you can rely on.

Agreed that sysconfig is a better place for more general values.  My point was
that if OpenSUSE wants to carry deltas that are specific only to its platform,
then sys.implementation._<whatever> is the standard place to put it.
msg222643 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-09 21:43
Is there agreement on what needs doing here?  I'd like to see this into 3.5 before it reaches its 10th birthday :)
msg242608 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-05-05 18:57
Also see #15631
msg285126 - (view) Author: jan matejek (matejcik) * Date: 2017-01-10 15:31
at this again, when porting SUSE patches to 3.6.0 :) ( :( )

Last time there was a discussion, Barry suggested using sysconfig variables to find the proper libdir. Trouble is, to fill out the variables, sysconfig itself uses two sources:
a) compiled-in information from the binary, and
b) Makefile items
Makefile seems an obvious location for storing the libdir info, except, well, to FIND the makefile in the first place, we are using sysconfig variables.... you see where this is heading.

So, given that sysconfig is the correct Primary Source for libdir info, we have three options to get it there:
1. compile it into the binary. sys.implementation.platlibdir seems as good as any -- i'm not completely convinced that this is a property of the _implementation_, but, well.
2. modify sysconfig.py at build time
3. guess it from the environment

And options (2) and (3) seem wrong, so i'm going with (1) for now
msg285222 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2017-01-11 13:09
The downstream patch we currently use in Fedora [0].

[0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/plain/00102-lib64.patch
msg285556 - (view) Author: jan matejek (matejcik) * Date: 2017-01-16 11:20
Attached is a patch that I'd like to propose for inclusion.

It introduces a new configure option "--with-custom-platlibdir=<name>", which defaults to `basename $libdir`. This is converted to makefile variable "platlibdir", which is used in getpath.c to generate value of the lib_python variable.

sysconfig and distutils.sysconfig retrieve the variable from makefile (i learned about existence of _sysconfigdata so that is where the variable now lives) to correctly substitute in posix_prefix (and unix_prefix for distutils install) scheme.

Sysconfig is then used in pydoc and trace module, instead of locally calculating paths. Perhaps distutils.sysconfig should also use more of sysconfig instead of duplicating the functionality?

As it stands, the python stdlib is installed into /usr/$platlibdir, because I changed SCRIPTDIR to $libdir instead of $prefix/lib. Maybe this should be also separately configurable?
In any case, third-party modules get installed either into /usr/lib or /usr/$platlibdir depending on whether they are arch-independent or not. Hence the final modification in site.py that adds /usr/lib to search path on systems where $platlibdir != "lib".

This caused test_sysconfig to fail because it checks that posix_user and posix_prefix schemes generate "similar" paths. I have left the posix_user scheme alone, so all its paths are based on "lib", where the posix_prefix scheme uses $platlibdir. Maybe posix_user scheme should be modified as well?
Anyway, for now, I have accounted for the changes in the test.

What now?
(please don't make me write a PEP please don't make me write a PEP please don't make......)
msg285737 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2017-01-18 15:57
I don't think we need a PEP for this, however I would like to name the new macro libsubdir, or sublibdir, to make it clear it's just the libdir component. Or make the value absolute, like it already is for libdir ...
msg285738 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2017-01-18 16:00
On Jan 18, 2017, at 03:57 PM, Matthias Klose wrote:

>I don't think we need a PEP for this

Correct.  Generally we don't need PEPs for build system changes.
msg307231 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2017-11-29 14:47
Pinging here. Is there some way to push the issue forward?
msg308555 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2017-12-18 13:28
the patch looks ok. was it tested on a Debian or Ubuntu system to produce the same layout with this patch and without the new configure option?
msg361844 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 00:53
I closed bpo-858809 as a duplicate of this issue.
msg361847 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 01:06
I marked bpo-15631 as a duplicate of this issue.
msg361849 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 01:16
I marked bpo-34032 as a duplicate of this issue. It has an open PR written by Matěj Cepl (OpenSUSE): PR 8068. It looks similar to PR 3698.
msg361850 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 01:18
See also bpo-34058 "Default Python 3.7 install broken on openSUSE Leap 42.3: $PYTHONHOME/lib64/python3.7/lib-dynload/ not linked to $PYTHONHOME/lib/python3.7/lib-dynload/": OpenSUSE and Centos are mentioned as being impacted.
msg361851 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 01:19
See also bpo-14791: "setup.py only adds /prefix/lib, not /prefix/lib64".
msg361853 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 01:22
I marked bpo-18092 ("Python 2.7.5 installation broken on OpenSuse 12.2") as a duplicate of this issue.
msg361854 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 01:22
See also bpo-21016: "trace: $prefix and $exec_prefix improperly replaced on Fedora".
msg361882 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 12:02
New changeset 4fac7ed43ebf1771a8fe86fdfe7b9991f3be78cd by Victor Stinner in branch 'master':
bpo-21016: pydoc and trace use sysconfig (GH-18476)
https://github.com/python/cpython/commit/4fac7ed43ebf1771a8fe86fdfe7b9991f3be78cd
msg361888 - (view) Author: miss-islington (miss-islington) Date: 2020-02-12 12:32
New changeset ca133e53fafdec1aa77613fcb7558deed959383f by Miss Islington (bot) in branch '3.7':
bpo-21016: pydoc and trace use sysconfig (GH-18476)
https://github.com/python/cpython/commit/ca133e53fafdec1aa77613fcb7558deed959383f
msg361890 - (view) Author: miss-islington (miss-islington) Date: 2020-02-12 12:32
New changeset ac6f4d2db703c0ff88e496bcb7b7fe55cf2ac458 by Miss Islington (bot) in branch '3.8':
bpo-21016: pydoc and trace use sysconfig (GH-18476)
https://github.com/python/cpython/commit/ac6f4d2db703c0ff88e496bcb7b7fe55cf2ac458
msg361893 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 13:48
Summary:

* This issue is mostly about pushing a 16-years old patch from Fedora and SuSE to Python upstream. Other Linux distributions use different solutions to support multiple architectures in the same filesystem. For example, Debian and Ubuntu uses "Multiarch" which continues to use /usr/lib/ for 32-bit and 64-bit x86 architecture (x86 and x86-64).

* I pushed the non controversial part of PR 3698: fix pydoc and trace modules to use sysconfig to get the stdlib path

* CI tests pass on PR 18381.

* Matthias Klose asked to not add another sys attribute, but Jan Matějek explained that the option value is needed to import the sysconfig module. Moreover, the option value is needed in the site module and we are trying to avoid "import sysconfig" in site since it would slowdown Python startup time.

* I chose "./configure --with-python-libdir" and "sys.python_libdir" names. PR 3698 uses "./configure --with-custom-platsubdir" and sysconfig.get_config_var('platsubdir') names. Other proposed names: sys.lib, os.lib_dir. It was proposed to drop "custom" from "./configure --with-custom-platsubdir" option name.

Python use many paths. The sysconfig module is a good starting point to discover them all in the _INSTALL_SCHEMES directories. PR 18381 changes the following paths:

* posix_prefix: stdlib, platstdlib, platlib
* posix_user: stdlib, platstdlib, platlib


Examples on Fedora:

* pure Python: /usr/lib64/python3.7/os.py
* C extension: /usr/lib64/python3.7/lib-dynload/_multiprocessing.cpython-37m-x86_64-linux-gnu.so

Examples on SuSE:

* pure Python: /usr/lib64/python3.7/os.py

Ubuntu 16.04:

* pure Python: /usr/lib/python3.5/os.py
* C extension: /usr/lib/python3.5/lib-dynload/_multiprocessing.cpython-35m-x86_64-linux-gnu.so
msg361894 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 13:52
Default with prefix=/usr:

* posix_prefix:

  * stdlib: '/usr/lib/python3.9'
  * platstdlib: '/usr/lib/python3.9'
  * platlib: '/usr/lib/python3.9/site-packages'
 
* posix_user:

  * stdlib: '~/.local/lib/python3.9'
  * platstdlib: '~/.local/lib/python3.9'
  * platlib: '~/.local/lib/python3.9/site-packages'

Customized with prefix=/usr and python_libdir=lib64, PR 18381:

* posix_prefix:

  * stdlib: '/usr/lib64/python3.9'
  * platstdlib: '/usr/lib64/python3.9'
  * platlib: '/usr/lib64/python3.9/site-packages'

* posix_user:

  * stdlib: '~/.local/lib64/python3.9'
  * platstdlib: '~/.local/lib64/python3.9'
  * platlib: '~/.local/lib64/python3.9/site-packages'
msg361897 - (view) Author: Matej Cepl (mcepl) * Date: 2020-02-12 14:49
> posix_user:
> 
> * stdlib: '~/.local/lib64/python3.9'
> * platstdlib: '~/.local/lib64/python3.9'
> * platlib: '~/.local/lib64/python3.9/site-packages'

I am still not sure about the this one. I know that I have mentioned a possibility of multiplatform $HOME on NFS, but it seems too far fetched idea to break the current not-completely-standard.
msg361899 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-02-12 14:54
Also note that the extension modules have archful suffix, so they will not collide in $HOME, unless they are installed as name.so.
msg361900 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 15:12
> posix_user:
> 
> * stdlib: '~/.local/lib64/python3.9'
> * platstdlib: '~/.local/lib64/python3.9'
> * platlib: '~/.local/lib64/python3.9/site-packages'

Honestly, I have no idea how posix_user is used.

~/.local/lib64/ directory doesn't exist on my work machine, whereas I'm installed various Python modules and applications.

In practice, ~/.local/lib/ is used on Fedora, even if posix_user scheme uses ~/.local/lib64/. Examples:

* pure Python: ~/.local/lib/python3.7/site-packages/pip/__main__.py
* C extension: ~/.local/lib/python3.7/site-packages/typed_ast/_ast3.cpython-37m-x86_64-linux-gnu.so

So changing posix_user should have no impact on end users.


I also tested a legacy "python3 setup.py install" to install a C extension, it also lands into ~/.local/lib:

~/.local/lib/python3.7/site-packages/test_cext-0.0.0-py3.7-linux-x86_64.egg/test_cext.cpython-37m-x86_64-linux-gnu.so

The setup.py uses setuptools.


Miro:
> Also note that the extension modules have archful suffix, so they will not collide in $HOME, unless they are installed as name.so.

Yeah, it seems like pip install adopted Debian multiarch naming: "x86_64-linux-gnu" triplet in "_ast3.cpython-37m-x86_64-linux-gnu.so" filename.
msg361901 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-02-12 15:20
> > Also note that the extension modules have archful suffix, so they will not collide in $HOME, unless they are installed as name.so.
> 
> Yeah, it seems like pip install adopted Debian multiarch naming: "x86_64-linux-gnu" triplet in "_ast3.cpython-37m-x86_64-linux-gnu.so" filename.

For reference: issue22980
msg361902 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 15:28
SuSE currently uses this patch for lib64 in the Python 3 package:
https://build.opensuse.org/package/view_file/devel:languages:python:Factory/python3/F00102-lib64.patch?expand=1
msg361903 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 15:31
Ooops, SuSE has 2 patches.

SuSE currently uses these patches for lib64 in the Python 3 package:

(1) https://build.opensuse.org/package/view_file/devel:languages:python:Factory/python3/F00102-lib64.patch?expand=1

(2) https://build.opensuse.org/package/view_file/devel:languages:python:Factory/python3/SUSE-FEDORA-multilib.patch?expand=1
msg361907 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 17:50
lib64_tests.py: test suite hardcoded for Fedora, SuSE and PR 18381 with --with-python-libdir=lib64. I wrote it to help me to understand differences between Fedora, SuSE and PR 18381.

On Fedora, the test fails because we have an additional patch on distutils to install packages in /usr/local instead of /usr when using pip:

======================================================================
FAIL: test_distutils_install (__main__.Lib64Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "lib64_tests.py", line 81, in test_distutils_install
    self.assertEqual(cmd.install_platlib, PLATLIB)
AssertionError: '/usr/local/lib64/python3.7/site-packages' != '/usr/lib64/python3.7/site-packages'
- /usr/local/lib64/python3.7/site-packages
?     ------
+ /usr/lib64/python3.7/site-packages
msg361909 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-12 18:04
Links on Debian Multiarch:

* https://wiki.debian.org/Multiarch/HOWTO
* https://lwn.net/Articles/711199/
msg362324 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2020-02-20 15:53
Why "sys.python_libdir"?  Isn't that too public? A lot of similar path-related info is in sysconfig; why shouldn't it be there?
How does "sys.python_libdir" work on other platforms?
History
Date User Action Args
2020-02-20 15:53:06petr.viktorinsetmessages: + msg362324
2020-02-12 18:13:43pitrousetnosy: - pitrou
2020-02-12 18:04:46vstinnersetmessages: + msg361909
2020-02-12 17:50:19vstinnersetfiles: + lib64_tests.py

messages: + msg361907
2020-02-12 15:31:30vstinnersetmessages: + msg361903
2020-02-12 15:28:07vstinnersetmessages: + msg361902
2020-02-12 15:20:17hroncoksetmessages: + msg361901
2020-02-12 15:12:11vstinnersetmessages: + msg361900
2020-02-12 14:54:41hroncoksetmessages: + msg361899
2020-02-12 14:49:16mceplsetmessages: + msg361897
2020-02-12 13:52:08vstinnersetmessages: + msg361894
2020-02-12 13:48:31vstinnersetmessages: + msg361893
2020-02-12 12:32:58miss-islingtonsetmessages: + msg361890
2020-02-12 12:32:53miss-islingtonsetnosy: + miss-islington
messages: + msg361888
2020-02-12 12:02:49miss-islingtonsetpull_requests: + pull_request17856
2020-02-12 12:02:42miss-islingtonsetpull_requests: + pull_request17854
2020-02-12 12:02:42vstinnersetmessages: + msg361882
2020-02-12 09:14:37vstinnersetpull_requests: + pull_request17846
2020-02-12 08:57:22vstinnersetversions: - Python 2.7, Python 3.4, Python 3.5, Python 3.6
2020-02-12 07:57:00carlos.velascosetnosy: + carlos.velasco
2020-02-12 01:22:39vstinnersetmessages: + msg361854
2020-02-12 01:22:01vstinnersetmessages: + msg361853
2020-02-12 01:21:56vstinnerlinkissue18092 superseder
2020-02-12 01:19:16vstinnersetmessages: + msg361851
2020-02-12 01:18:30vstinnersetmessages: + msg361850
2020-02-12 01:16:08vstinnersetmessages: + msg361849
2020-02-12 01:15:18vstinnerlinkissue34032 superseder
2020-02-12 01:06:13vstinnersetmessages: + msg361847
2020-02-12 01:05:53vstinnerlinkissue15631 superseder
2020-02-12 00:53:14vstinnersetmessages: + msg361844
2020-02-12 00:53:00vstinnerlinkissue858809 superseder
2020-02-06 17:17:33vstinnersetpull_requests: + pull_request17757
2019-10-11 15:16:47mceplsetversions: + Python 3.8, Python 3.9
2019-10-11 15:16:34mceplsetnosy: + mcepl
2019-10-11 14:59:23vstinnersetnosy: + vstinner
2019-02-04 15:09:53cstrataksetpull_requests: + pull_request11696
2018-05-16 07:58:53eitan.adlersetnosy: + eitan.adler
2018-04-04 17:19:47pillarsdotnetsetnosy: + pillarsdotnet
2017-12-20 11:32:22hroncoksetnosy: + petr.viktorin, hroncok
2017-12-18 13:28:31dokosetmessages: + msg308555
2017-11-29 14:47:00cstrataksetmessages: + msg307231
2017-09-22 16:45:20matejciksetstage: patch review
pull_requests: + pull_request3682
2017-01-18 16:00:13barrysetmessages: + msg285738
2017-01-18 15:57:06dokosetmessages: + msg285737
2017-01-16 11:20:22matejciksetfiles: + python-3.6.0-multilib-new.patch

messages: + msg285556
versions: + Python 3.7
2017-01-11 13:09:06cstrataksetnosy: + cstratak
messages: + msg285222
2017-01-10 15:46:46BreamoreBoysetnosy: - BreamoreBoy
2017-01-10 15:31:20matejciksetmessages: + msg285126
versions: + Python 3.6
2015-05-05 18:57:43BreamoreBoysetmessages: + msg242608
2014-11-19 10:50:04piotr.dobrogostsetnosy: + piotr.dobrogost
2014-07-09 21:43:46BreamoreBoysetnosy: + BreamoreBoy

messages: + msg222643
versions: + Python 3.4, Python 3.5, - Python 3.2, Python 3.3
2013-11-07 17:55:15barrysetmessages: + msg202378
2013-11-07 16:59:02Arfreversetmessages: + msg202368
2013-11-07 16:56:50dokosetmessages: + msg202367
2013-11-07 16:53:50barrysetmessages: + msg202366
2013-11-07 16:51:09barrysetmessages: + msg202365
2013-11-07 16:05:54matejciksetmessages: + msg202359
2013-11-07 15:56:17dokosetmessages: + msg202354
2013-11-07 15:40:04matejciksetfiles: + Python-3.3.0b2-multilib.patch

messages: + msg202343
2013-11-07 15:17:59barrysetmessages: + msg202341
2013-08-01 20:40:31catalin.iacobsetnosy: + catalin.iacob
2012-05-15 07:42:31lemburgsetmessages: + msg160691
2012-05-14 22:14:43eric.araujosetmessages: + msg160671
versions: + Python 3.3, - Python 3.1
2012-05-12 17:00:21Arfreversetmessages: + msg160482
2011-03-29 22:15:34barrysetmessages: + msg132546
2011-03-29 22:15:07barrysetmessages: + msg132544
2011-03-29 22:12:15dokosetmessages: + msg132542
2011-03-29 22:11:04dokosetmessages: + msg132540
2011-03-29 22:05:33barrysetmessages: + msg132538
2011-03-29 21:50:14barrysetfiles: - a4dcae4cd033.diff
2011-03-29 21:49:54barrysetmessages: + msg132537
2011-03-29 21:39:30barrysetmessages: + msg132535
2011-03-29 19:38:53Arfreversetmessages: + msg132511
2011-03-29 19:28:41pitrousetnosy: + pitrou
messages: + msg132506
2011-03-29 19:19:37barrysetmessages: + msg132505
2011-03-29 19:13:41barrysetfiles: + a4dcae4cd033.diff
2011-03-29 19:13:20barrysetfiles: - a9b05b89ea39.diff
2011-03-29 19:11:50barrysetfiles: + a9b05b89ea39.diff
2011-03-29 19:11:23barrysethgrepos: + hgrepo12
messages: + msg132503
2011-03-28 21:29:10barrysetmessages: + msg132424
2011-03-17 12:53:40jceasetnosy: + jcea
2011-03-16 13:55:54andybuckleysetnosy: + andybuckley
2010-08-21 19:41:29BreamoreBoysettype: behavior
versions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6, Python 3.0
2010-08-11 05:48:58eric.araujosetnosy: + barry
2009-11-06 17:41:23lemburgsetnosy: + tarek
messages: + msg94989
2009-11-06 17:27:21lemburgsetnosy: + lemburg
messages: + msg94986
2009-11-06 16:47:16Arfreversetnosy: + Arfrever
2009-11-06 16:26:22matejciksetmessages: + msg94982
2009-11-05 19:23:07eric.araujosetnosy: + eric.araujo
messages: + msg94941
2009-09-12 12:46:31akitadasetmessages: + msg92541
2009-08-20 18:29:01matejciksetmessages: + msg91781
2009-08-20 11:01:02dokosetnosy: + doko
messages: + msg91769
2009-08-14 16:27:32matejciksetfiles: + Python-2.6.2-multilib.patch

messages: + msg91559
2009-03-30 04:23:41ajaksu2linkissue1536339 superseder
2009-03-30 03:23:22ajaksu2linkissue1553166 superseder
2009-03-03 16:17:33matejciksetnosy: + matejcik
2009-02-28 17:32:01akitadasetmessages: + msg82929
2009-02-03 21:23:29akitadasetnosy: + akitada
messages: + msg81099
2008-03-27 14:31:02belopolskysetmessages: + msg64593
2008-03-27 14:11:17belopolskysetnosy: + belopolsky
messages: + msg64592
2008-02-03 07:18:05ivazquezsetnosy: + ivazquez
2008-01-05 19:33:54christian.heimessetkeywords: + patch
nosy: + christian.heimes
messages: + msg59310
components: + Build, - None
versions: + Python 2.6, Python 3.0
2005-09-19 03:05:28jafocreate