The new API method Py_SetPath seems bugged. When executing the following code, then the application crashes on Py_Initialize()-call:
#include "Python.h"

main(int argc, char **argv)
{
  Py_SetPath(Py_GetPath());
  printf("Init\n");
  Py_Initialize();
  printf("-- END\n");
}

The raised exception is the following:

Traceback (most recent call last):
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 332, in _init_posix
    _parse_makefile(makefile, vars)
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 220, in _parse_makefile
    with open(filename, errors="surrogateescape") as f:
IOError: [Errno 2] No such file or directory: 'lib/python3.2/config-3.2m/Makefile'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 529, in <module>
    main()
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 517, in main
    known_paths = addusersitepackages(known_paths)
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 263, in addusersitepackages
    user_site = getusersitepackages()
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 238, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 228, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 590, in get_config_var
    return get_config_vars().get(name)
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 487, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 337, in _init_posix
    raise IOError(msg)
IOError: invalid Python installation: unable to open lib/python3.2/config-3.2m/Makefile (No such file or directory)

(perhaps linked to issue #10743 ?!)

This issue is potentially breaking virtualenv5,
http://code.google.com/p/virtualenv5/issues/detail?id=6

Can you explain why this is a problem in Python?
Can't lib/python3.2/config-3.2m/Makefile simply be provided by virtualenv (by copying it, I guess)?

@Sridhar: Could you please provide input for the question asked by Antoine?
I'd love to have this issue fixed for next Python release.

Note that this will be fixed for 3.3, as siteconfig will be updated to use static data (generated during the build process) rather than relying on build artifacts being available at runtime.

However, virtualenv (et al) will still need to provide the relevant static config file in the isolated environments.

[pitrou] 
> Can you explain why this is a problem in Python?
> Can't lib/python3.2/config-3.2m/Makefile simply be 
> provided by virtualenv (by copying it, I guess)?

Yes, I believe virtualenv already does that (or symlinks to it). Python 3.2 changed the path to config and include directories for some reason, viz.

$ ls -d /opt/ActivePython-3.*/lib/python3.?/*config*/
/opt/ActivePython-3.1/lib/python3.1/config/
/opt/ActivePython-3.2/lib/python3.2/config-3.2m/
$ 

and:

$ ls -d /opt/ActivePython-3.*/include/python3.?*
/opt/ActivePython-3.1/include/python3.1  
/opt/ActivePython-3.2/include/python3.2m
$

It is possible that virtualenv is hardcoding the relative path to 'config' (and 'include') directories and thus failing to find the new 'config-3.2m' dir. If that is the case, this is not a problem with Python.

Although msg129372 does point to a Python bug, it may or may not be related to the virtualenv issue noted earlier. Given that virtualenv doesn't officially Python 3 and virtualenv5 is more of a hack, I haven't investigated into this much.

Does that answer your question?

Sridhar: I'm sorry, but at this point you should investigate a bit more into the actual causes of the problem. I'm not going to dig in virtualenv myself. Also, since there's already #10743 for the virtualenv issue, perhaps further virtualenv-related comments should be posted there.

(Palm, I take it your issue doesn't have anything to do with virtualenv?)

Ok, Palm's example even segfaults here:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff72452bc in free () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff72452bc in free () from /lib64/libc.so.6
#1  0x000000000042a591 in Py_SetPath (
    path=0x88c330 L"/home/antoine/cpython/default/usr/lib/python33.zip:/home/antoine/cpython/default/usr/lib/python3.3/:/home/antoine/cpython/default/usr/lib/python3.3/plat-linux2:/home/antoine/cpython/default/usr/lib/py"...)
    at ./Modules/getpath.c:729
#2  0x0000000000416600 in main ()


The problem is calling free() on a pointer to statically allocated memory.

Crash fixed in a791dd7d51f3 (3.2) and b4104ffd5127 (3.3), but the original problem remains, or a variant of it. I now get:

Init
Fatal Python error: Py_Initialize: Unable to get the locale encoding
LookupError: no codec search functions registered: can't find encoding
Abandon

Ok, the issue here is that you can't call Py_SetPath() on the point returned by Py_GetPath(), since the memory refered to that pointer is free()d at the beginning of Py_SetPath(). The following works, though:

main(int argc, char **argv)
{
  wchar_t *path, *newpath;
  path = Py_GetPath();
  newpath = malloc((wcslen(path) + 1) * sizeof(wchar_t));
  wcscpy(newpath, path);
  Py_SetPath(newpath);
  free(newpath);
  printf("Init\n");
  Py_Initialize();
  printf("-- END\n");
}


Perhaps we could modify Py_SetPath() so that it copies the new path first before deallocating the old one, but I'm not sure I see the point of calling Py_SetPath() with the pointer returned by Py_GetPath().

Antoine,

Your guess that my issue initially wasn't related to virtualenv is correct (I've never heard about that project before posting this issue...)

As for passing the output of Py_GetPath directly to Py_SetPath: You are right, there is no point in doing this...
Now,  I remember that the initial problem I had was the one you reported:
    Fatal Python error: Py_Initialize: Unable to get the locale encoding
    LookupError: no codec search functions registered: can't find encoding

Shall I create a separate issue to report this problem?

Furthermore I would propose to rename this issue: The problem is not that Py_SetPath cannot be called on pointer returned by Py_GetPath. I think that the problem is more general: Calling Py_SetPath NEVER works.

--> I get the same exception as documented in my initial post when calling Py_SetPyth like this:
   Py_SetPath(L"/usr/labsolution/python32/lib/python3.2/");
or like this:
  wchar_t * path;
  path = malloc(128 * sizeof(wchar_t));
  wcscpy(path,L"/usr/labsolution/python32/lib/python3.2/");
  Py_SetPath(path);

As for this error:
  Fatal Python error: Py_Initialize: Unable to get the locale encoding
  LookupError: no codec search functions registered: can't find encoding

It seems to me that this error appears if the path passed to Py_SetPath does not point to a valid python lib folder.

> As for this error:
>   Fatal Python error: Py_Initialize: Unable to get the locale encoding
>   LookupError: no codec search functions registered: can't find encoding
> 
> It seems to me that this error appears if the path passed to
> Py_SetPath does not point to a valid python lib folder.

Indeed, it does. The error message could perhaps get improved.
What Python does is try to find the current locale's encoding in the
"encodings" package and load it. If it can't find it, it assumes that's
because the encoding is unknown, not because the path is wrong.

Hello everyone,

I have found the reason for the problem.

From the manual http://docs.python.org/py3k/c-api/init.html#Py_SetPath , 
we can see that:

After we call Py_SetPath,both sys.prefix and sys.exec_prefix will be empty.

However, sys.prefix will be used in the sysconfig.py to generate the makefile' s name, and the empty sys.prefix will cause the wrong path: lib/python3.2/config-3.2m/Makefile

sysconfig.py imported by site.py, and site.py used in the Py_InitializeEx.

So ...