classification
Title: 3.2's sysconfig doesn't work with virtualenv
Type: behavior Stage:
Components: Distutils, macOS Versions: Python 3.2
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: tarek Nosy List: barry, eric.araujo, georg.brandl, ned.deily, palm.kevin, pitrou, srid, tarek
Priority: release blocker Keywords:

Created on 2010-12-20 22:06 by srid, last changed 2011-02-25 18:36 by palm.kevin. This issue is now closed.

Messages (14)
msg124405 - (view) Author: Sridhar Ratnakumar (srid) Date: 2010-12-20 22:06
From http://code.google.com/p/virtualenv5/issues/detail?id=6 - it seems that the `sysconfig` module is looking for Makefile in wrong directory, while ideally it must be looking into the base Python install.

>> import sysconfig; sysconfig.get_paths('purelib')
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 332, in _init_posix
    _parse_makefile(makefile, vars)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 220, in _parse_makefile
    with open(filename, errors="surrogateescape") as f:
IOError: [Errno 2] No such file or directory: '/tmp/e/lib/python3.2/config-3.2m/Makefile'
msg124412 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-12-20 23:57
That does seem to be a regression since distutils.sysconfig works correctly in a virtualenv:

$ python3.2 -c 'import distutils.sysconfig;print(distutils.sysconfig.get_makefile_filename())'
/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/config-3.2m/Makefile
$ python3.2 -c 'import sysconfig;print(sysconfig.get_makefile_filename())'
/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/config-3.2m/Makefile

but in a virtualenv:

$ source t/bin/activate
$ python3.2 -c 'import distutils.sysconfig;print(distutils.sysconfig.get_makefile_filename())'
/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/config-3.2m/Makefile
$ python3.2 -c 'import sysconfig;print(sysconfig.get_makefile_filename())'
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 332, in _init_posix
    _parse_makefile(makefile, vars)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 220, in _parse_makefile
    with open(filename, errors="surrogateescape") as f:
IOError: [Errno 2] No such file or directory: '/Users/nad/t/lib/python3.2/config-3.2m/Makefile'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 322, in get_makefile_filename
    return os.path.join(get_path('stdlib'),
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 451, in get_path
    return get_paths(scheme, vars, expand)[name]
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 442, in get_paths
    return _expand_vars(scheme, vars)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 168, in _expand_vars
    _extend_dict(vars, get_config_vars())
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 487, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 337, in _init_posix
    raise IOError(msg)


BTW, your example will fail in any case since get_paths takes a scheme name; "purelib" is a path name.
msg124413 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-12-20 23:59
Setting to deferred blocker for 3.2 release manager evaluation.
msg124439 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-12-21 17:50
Agreed it should probably be a release blocker. I guess it has to do with (not) following symlinks, right?
msg124441 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-12-21 17:59
OK, let's make this one.  Tarek to the rescue!
msg124442 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2010-12-21 18:06
Will do tonight
msg124546 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2010-12-23 11:58
I have investigated the problem and it turns out virtualenv patches distutils.sysconfig behavior by adding to the sys module a "real_prefix" attribute that points to the global Python install and is used instead of sys.prefix that points to the virtualenv local install, when some distutils.sysconfig APIs are used.

The fix is to change virtualenv so it patches sysconfig the same way it does for distutils.sysconfig.

To simulate this patch, just change how _EXEC_PREFIX and _PREFIX are set in sysconfig, by setting them to sys.real_prefix instead of sys.prefix/sys.exec_prefix when using virtualenv. 

I suggest to Georg that this bug gets resolved to "invalid" and that we move it to the virtualenv bitbucket tracker.
msg124562 - (view) Author: Sridhar Ratnakumar (srid) Date: 2010-12-23 18:27
Sounds good, but this doesn't belong to the virtualenv bug tracker (virtualenv does even support Python 3). Instead, it belongs to the virtualenv5 tracker: http://code.google.com/p/virtualenv5/issues/detail?id=6
msg129361 - (view) Author: Palm Kevin (palm.kevin) Date: 2011-02-25 13:16
I think that this issue needs to be reopened... since it never has been resolved...
I just downloaded the new version of Python 3.2 and tried to compile, install and use it on Redhat Linux.

Installation went fine (configure with --prefix=/usr/mysoft/python32)...

But when using Python I get the following error message:
Traceback (most recent call last):
  File "/usr/mysoft/python32/lib/python3.2/sysconfig.py", line 332, in _init_posix
    _parse_makefile(makefile, vars)
  File "/usr/mysoft/python32/lib/python3.2/sysconfig.py", line 220, in _parse_makefile
    with open(filename, errors="surrogateescape") as f:
IOError: [Errno 2] No such file or directory: 'lib/python3.2/config-3.2m/Makefile'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/mysoft/python32/lib/python3.2/site.py", line 529, in <module>
    main()
  File "/usr/mysoft/python32/lib/python3.2/site.py", line 517, in main
    known_paths = addusersitepackages(known_paths)
  File "/usr/mysoft/python32/lib/python3.2/site.py", line 263, in addusersitepackages
    user_site = getusersitepackages()
  File "/usr/mysoft/python32/lib/python3.2/site.py", line 238, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/usr/mysoft/python32/lib/python3.2/site.py", line 228, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/usr/mysoft/python32/lib/python3.2/sysconfig.py", line 590, in get_config_var
    return get_config_vars().get(name)
  File "/usr/mysoft/python32/lib/python3.2/sysconfig.py", line 487, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/usr/mysoft/python32/lib/python3.2/sysconfig.py", line 337, in _init_posix
    raise IOError(msg)
IOError: invalid Python installation: unable to open lib/python3.2/config-3.2m/Makefile (No such file or directory)

... I never heard about virtualenv before and I don't know what is the impact of that library, but I guess a just released version of Python should work out of the box?!? (Final 3.2 release)
(I was using Python 3.1.3 on the same machine without any hitch...)
msg129367 - (view) Author: Palm Kevin (palm.kevin) Date: 2011-02-25 13:58
Please find here a small C app embedding python that shows  how to reproduce the problem (It turned out that the problem is caused by the new method "Py_SetPath"): 

#include "Python.h"

main(int argc, char **argv)
{
  Py_SetPath(Py_GetPath());
  printf("Init\n");
  Py_Initialize();
  printf("-- END\n");
}


--> Python crashes on Py_Initialize... only if the Py_SetPath is executed...
msg129370 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-02-25 14:18
Le vendredi 25 février 2011 à 13:58 +0000, Palm Kevin a écrit :
> Palm Kevin <kevin.palm@labsolution.lu> added the comment:
> 
> Please find here a small C app embedding python that shows  how to
> reproduce the problem (It turned out that the problem is caused by the
> new method "Py_SetPath"): 

Can you open a separate issue?
Also, please include your error message.
msg129373 - (view) Author: Palm Kevin (palm.kevin) Date: 2011-02-25 14:32
Done: #11320
msg129388 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2011-02-25 16:06
On Feb 25, 2011, at 01:16 PM, Palm Kevin wrote:

>I think that this issue needs to be reopened... since it never has been
>resolved...  I just downloaded the new version of Python 3.2 and tried to
>compile, install and use it on Redhat Linux.
>
>Installation went fine (configure with --prefix=/usr/mysoft/python32)...

On Ubuntu 10.10, I built Python 3.2 from the svn head with
--prefix=/tmp/prefix, did a 'make install' and then ran
/tmp/prefix/bin/python.  It worked fine and I did not get the exception.
msg129399 - (view) Author: Palm Kevin (palm.kevin) Date: 2011-02-25 18:36
@Barry: Yes python runs correctly.
The problem arises when you embed python in a C application when calling the Py_SetPath-method (see the follow up issue #11320)
History
Date User Action Args
2011-02-25 18:36:42palm.kevinsetnosy: barry, georg.brandl, pitrou, tarek, ned.deily, eric.araujo, srid, palm.kevin
messages: + msg129399
2011-02-25 16:06:31barrysetnosy: barry, georg.brandl, pitrou, tarek, ned.deily, eric.araujo, srid, palm.kevin
messages: + msg129388
2011-02-25 14:32:29palm.kevinsetnosy: barry, georg.brandl, pitrou, tarek, ned.deily, eric.araujo, srid, palm.kevin
messages: + msg129373
2011-02-25 14:18:36pitrousetnosy: barry, georg.brandl, pitrou, tarek, ned.deily, eric.araujo, srid, palm.kevin
messages: + msg129370
2011-02-25 13:58:45palm.kevinsetnosy: barry, georg.brandl, pitrou, tarek, ned.deily, eric.araujo, srid, palm.kevin
messages: + msg129367
2011-02-25 13:16:29palm.kevinsetnosy: + palm.kevin
messages: + msg129361
2010-12-23 18:27:07sridsetstatus: open -> closed

messages: + msg124562
resolution: not a bug
nosy: barry, georg.brandl, pitrou, tarek, ned.deily, eric.araujo, srid
2010-12-23 11:58:21tareksetnosy: barry, georg.brandl, pitrou, tarek, ned.deily, eric.araujo, srid
messages: + msg124546
2010-12-21 18:06:42tareksetnosy: barry, georg.brandl, pitrou, tarek, ned.deily, eric.araujo, srid
messages: + msg124442
2010-12-21 17:59:24georg.brandlsetpriority: deferred blocker -> release blocker
nosy: barry, georg.brandl, pitrou, tarek, ned.deily, eric.araujo, srid
messages: + msg124441
2010-12-21 17:50:44pitrousetnosy: + pitrou
messages: + msg124439
2010-12-20 23:59:53ned.deilysetpriority: normal -> deferred blocker
nosy: + georg.brandl
messages: + msg124413

2010-12-20 23:57:56ned.deilysetnosy: + barry, ned.deily
messages: + msg124412
2010-12-20 22:06:41sridcreate