classification
Title: distutils.sysconfig.get_python_lib gives surprising result when used with a Python build
Type: behavior Stage: patch review
Components: Distutils Versions: Python 3.7, Python 3.6, Python 3.4, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: Arfrever, alexis, carljm, dstufft, eric.araujo, kesara, rbcollins, tarek, vinay.sajip, ysj.ray
Priority: normal Keywords: easy, patch

Created on 2009-05-22 12:39 by vinay.sajip, last changed 2016-09-20 02:59 by kesara.

Files
File name Uploaded Description Edit
distutils_sysconfig.diff vinay.sajip, 2010-09-13 06:06 Patch for distutils/sysconfig.py
sysconfig.diff vinay.sajip, 2010-09-13 06:07 Patch for sysconfig.py
libdir.py vinay.sajip, 2011-03-28 11:02 Simple script which prints include/lib/makefile settings
6087_distutils_sysconfig.diff kesara, 2016-09-13 04:11
6087_sysconfig.diff kesara, 2016-09-13 04:11
fix-6087.diff vinay.sajip, 2016-09-18 13:31 review
Messages (25)
msg88194 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2009-05-22 12:39
When used with a build from source (i.e. non-installed) of Python,
get_python_lib gives the result (on Linux)

[prefix]/lib/python[version]

Shouldn't this be [prefix]/Lib?

Note that get_python_inc() returns "Include". That's because it makes
use of the "python_build" flag which is set for builds from source.
msg102703 - (view) Author: ysj.ray (ysj.ray) Date: 2010-04-09 08:59
Yes, I agree with you, this may be a bug? 

Seemingly, the sysconfig.get_python_lib() should act just like the get_python_inc() function, return "[prefix]/lib/python[version]" in case of "python_build" flag is False, and "[prefix]/Lib" in case of "python_build" flag is True.
msg115843 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-09-08 02:42
What ./configure line did you use?
msg115879 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2010-09-08 15:01
Just plain

./configure

as far as I remember.
msg116253 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-09-12 23:36
Can you propose a patch?  Also, can you say if the bug exists with the new sysconfig.get_path in 2.7 and 3.2?  (I can’t test right now, I ./configure with a --prefix, which logically gets used in the paths).
msg116278 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2010-09-13 06:06
Attached is a patch for distutils/sysconfig.py.
msg116279 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2010-09-13 06:07
Attached is a patch for sysconfig.py.
msg116280 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2010-09-13 06:13
N.B. without the sysconfig.py patch, sysconfig.get_path() fails, but differently.

With the patch, sysconfig.get_path('stdlib') and sysconfig.get_path('include') give the expected values, but _CONFIG_VARS['srcdir'] still seems wrong:

vinay@eta-jaunty:/tmp$ python3.2
Python 3.2a2+ (py3k:84759, Sep 13 2010, 05:14:55) 
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig._CONFIG_VARS['srcdir']
'/tmp'
>>> 

Someone please take a look!
msg131570 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-03-20 23:34
The patches look good, thank you.  Do you have a command, script or test that would print the values, to make sure they’re right?
msg132124 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-03-25 17:48
get_makefile_filename also fails when run from a checkout.
msg132392 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-03-28 11:02
Attached is a simple script to show include and lib paths.

The makefile location seems OK on my system.
msg132648 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-03-31 11:02
Actually, the makefile location *is* wrong - ignore my last comment contradicting that.
msg132691 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-03-31 17:58
I’m not sure yet whether this is a bug or a new feature, but when python runs from an uninstalled build, I would like paths to refer to the source directory, not the default configure prefix.
msg138608 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-18 21:45
See also #12141.
msg146949 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-03 17:21
> I’m not sure yet whether this is a bug or a new feature, but when python runs from an
> uninstalled build, I would like paths to refer to the source directory, not the default
> configure prefix.

I need this fixed for packaging and it may affect venv too, so I will consider it a bug, fix it and wait for the python-dev response :)
msg247120 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-07-22 14:41
Moving this back to patch review since I'm 90% sure that the patch won't apply anymore (if I had a little more time I'd pull it down and double check) - but I've hit this myself and would totally commit it if updated.
msg276156 - (view) Author: Kesara Rathnayake (kesara) Date: 2016-09-13 04:10
This issue still exists in Python 3.7.
msg276157 - (view) Author: Kesara Rathnayake (kesara) Date: 2016-09-13 04:11
Updated patch based on vinay.sajip's patch.
msg276158 - (view) Author: Kesara Rathnayake (kesara) Date: 2016-09-13 04:11
Updated patch based on vinay.sajip's patch.
msg276165 - (view) Author: Kesara Rathnayake (kesara) Date: 2016-09-13 04:35
These patches break both test_distutils & test_sysconfig tests.
msg276168 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2016-09-13 04:45
Ok, so we need to figure out whether the tests are wrong, or the 'fix' is wrong.
msg276169 - (view) Author: Kesara Rathnayake (kesara) Date: 2016-09-13 04:47
Test output:

[cpython.25567]$ ./python.exe -m test test_distutils test_sysconfig -j3
Run tests in parallel using 3 child processes
0:00:01 [1/2/1] test_sysconfig failed
test test_sysconfig failed -- Traceback (most recent call last):
  File "/Users/kesara/pycon/cpython.25567/Lib/test/test_sysconfig.py", line 283, in test_user_similar
    self.assertEqual(user_path, global_path.replace(base, user, 1))
AssertionError: '/Users/kesara/.local/lib/python3.6' != '/Users/kesara/pycon/cpython.25567/Lib'
- /Users/kesara/.local/lib/python3.6
+ /Users/kesara/pycon/cpython.25567/Lib
0:00:05 [2/2/2] test_distutils failed
test test_distutils failed -- Traceback (most recent call last):
  File "/Users/kesara/pycon/cpython.25567/Lib/distutils/tests/test_sysconfig.py", line 40, in test_get_python_lib
    sysconfig.get_python_lib(prefix=TESTFN))
AssertionError: '/Users/kesara/pycon/cpython.25567/Lib' == '/Users/kesara/pycon/cpython.25567/Lib'

2 tests failed:
    test_distutils test_sysconfig

Total duration: 6 sec
Tests result: FAILURE
msg276284 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2016-09-13 15:04
> Ok, so we need to figure out whether the tests are wrong, or the 'fix' is wrong.

The tests were never updated, so most likely they will need changing. I was waiting for feedback about my point about _CONFIG_VARS['srcdir'], but I never got any at the time ... will look at updating the patch with updated tests.
msg276890 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2016-09-18 13:31
Updated patch to cover tests, and changed the implementation slightly. Tests pass on Linux and Windows (I'm not able to build on OS X). The patch is against the default branch but the diff should apply equally well to 3.6.
msg276998 - (view) Author: Kesara Rathnayake (kesara) Date: 2016-09-20 02:59
fix-6087.diff passes on OSX (Python 3.6)

$ ./python.exe -m test test_distutils test_sysconfig -j3
Run tests in parallel using 3 child processes
0:00:01 [1/2] test_sysconfig passed
0:00:13 [2/2] test_distutils passed
All 2 tests OK.

Total duration: 14 sec
Tests result: SUCCESS
History
Date User Action Args
2016-09-20 02:59:41kesarasetmessages: + msg276998
2016-09-18 13:31:31vinay.sajipsetfiles: + fix-6087.diff

messages: + msg276890
2016-09-13 15:04:53vinay.sajipsetmessages: + msg276284
2016-09-13 04:47:33kesarasetmessages: + msg276169
2016-09-13 04:45:50rbcollinssetmessages: + msg276168
2016-09-13 04:35:18kesarasetmessages: + msg276165
2016-09-13 04:11:31kesarasetfiles: + 6087_sysconfig.diff

messages: + msg276158
2016-09-13 04:11:14kesarasetfiles: + 6087_distutils_sysconfig.diff

messages: + msg276157
2016-09-13 04:10:40kesarasetnosy: + kesara

messages: + msg276156
versions: + Python 3.7
2015-07-22 14:41:46rbcollinssetversions: + Python 3.6
nosy: + rbcollins

messages: + msg247120

stage: commit review -> patch review
2014-07-05 22:04:58BreamoreBoysetnosy: + dstufft

components: - Distutils2
versions: + Python 3.4, Python 3.5, - 3rd party, Python 2.7, Python 3.2, Python 3.3
2011-11-03 17:21:51eric.araujosetmessages: + msg146949
versions: - Python 3.1
2011-06-18 21:45:25eric.araujosetmessages: + msg138608
2011-04-01 18:24:51carljmsetnosy: + carljm
2011-03-31 17:58:53eric.araujosetmessages: + msg132691
2011-03-31 11:02:24vinay.sajipsetmessages: + msg132648
2011-03-28 11:02:30vinay.sajipsetfiles: + libdir.py

messages: + msg132392
2011-03-25 17:48:37eric.araujosetmessages: + msg132124
2011-03-20 23:34:35eric.araujosetkeywords: + patch, easy, - needs review, buildbot
nosy: + alexis

stage: commit review
2011-03-20 23:34:00eric.araujosetnosy: - alexis
versions: + Python 3.3
messages: + msg131570

assignee: tarek -> eric.araujo
keywords: + needs review, buildbot, - patch, easy
2011-02-13 14:49:29alexissetnosy: + alexis
2010-09-29 23:43:53eric.araujosetversions: + 3rd party
2010-09-13 06:13:47vinay.sajipsetmessages: + msg116280
2010-09-13 06:07:31vinay.sajipsetfiles: + sysconfig.diff

messages: + msg116279
2010-09-13 06:06:28vinay.sajipsetfiles: + distutils_sysconfig.diff
keywords: + patch
messages: + msg116278
2010-09-12 23:36:16eric.araujosetnosy: vinay.sajip, tarek, eric.araujo, Arfrever, ysj.ray
messages: + msg116253
components: + Distutils2
versions: + Python 3.1, - Python 2.6
2010-09-08 18:04:04Arfreversetnosy: + Arfrever
2010-09-08 15:01:59vinay.sajipsetmessages: + msg115879
2010-09-08 02:42:18eric.araujosetmessages: + msg115843
2010-04-09 09:00:28ysj.raysetversions: + Python 2.6, Python 3.2
2010-04-09 09:00:06ysj.raysetversions: + Python 2.7, - Python 3.2
2010-04-09 08:59:13ysj.raysetnosy: + ysj.ray

messages: + msg102703
versions: + Python 3.2, - Python 2.6, Python 2.7
2010-04-09 00:24:23eric.araujosetnosy: + eric.araujo
2009-05-22 12:39:09vinay.sajipcreate