The new API method Py_SetPath seems bugged. When executing the following code, then the application crashes on Py_Initialize()-call:
#include "Python.h"

main(int argc, char **argv)
{
  Py_SetPath(Py_GetPath());
  printf("Init\n");
  Py_Initialize();
  printf("-- END\n");
}

The raised exception is the following:

Traceback (most recent call last):
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 332, in _init_posix
    _parse_makefile(makefile, vars)
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 220, in _parse_makefile
    with open(filename, errors="surrogateescape") as f:
IOError: [Errno 2] No such file or directory: 'lib/python3.2/config-3.2m/Makefile'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 529, in <module>
    main()
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 517, in main
    known_paths = addusersitepackages(known_paths)
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 263, in addusersitepackages
    user_site = getusersitepackages()
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 238, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 228, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 590, in get_config_var
    return get_config_vars().get(name)
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 487, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 337, in _init_posix
    raise IOError(msg)
IOError: invalid Python installation: unable to open lib/python3.2/config-3.2m/Makefile (No such file or directory)

(perhaps linked to issue bpo-10743 ?!)

This issue is potentially breaking virtualenv5,
http://code.google.com/p/virtualenv5/issues/detail?id=6

Can you explain why this is a problem in Python?
Can't lib/python3.2/config-3.2m/Makefile simply be provided by virtualenv (by copying it, I guess)?

@sridhar: Could you please provide input for the question asked by Antoine?
I'd love to have this issue fixed for next Python release.

Note that this will be fixed for 3.3, as siteconfig will be updated to use static data (generated during the build process) rather than relying on build artifacts being available at runtime.

However, virtualenv (et al) will still need to provide the relevant static config file in the isolated environments.

[pitrou]

Can you explain why this is a problem in Python?
Can't lib/python3.2/config-3.2m/Makefile simply be
provided by virtualenv (by copying it, I guess)?

Yes, I believe virtualenv already does that (or symlinks to it). Python 3.2 changed the path to config and include directories for some reason, viz.

$ ls -d /opt/ActivePython-3.*/lib/python3.?/*config*/
/opt/ActivePython-3.1/lib/python3.1/config/
/opt/ActivePython-3.2/lib/python3.2/config-3.2m/
$ 

and:

$ ls -d /opt/ActivePython-3.*/include/python3.?*
/opt/ActivePython-3.1/include/python3.1  
/opt/ActivePython-3.2/include/python3.2m
$

It is possible that virtualenv is hardcoding the relative path to 'config' (and 'include') directories and thus failing to find the new 'config-3.2m' dir. If that is the case, this is not a problem with Python.

Although msg129372 does point to a Python bug, it may or may not be related to the virtualenv issue noted earlier. Given that virtualenv doesn't officially Python 3 and virtualenv5 is more of a hack, I haven't investigated into this much.

Does that answer your question?

Sridhar: I'm sorry, but at this point you should investigate a bit more into the actual causes of the problem. I'm not going to dig in virtualenv myself. Also, since there's already bpo-10743 for the virtualenv issue, perhaps further virtualenv-related comments should be posted there.

(Palm, I take it your issue doesn't have anything to do with virtualenv?)

Ok, Palm's example even segfaults here:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff72452bc in free () from /lib64/libc.so.6
(gdb) bt
#0 0x00007ffff72452bc in free () from /lib64/libc.so.6
#1 0x000000000042a591 in Py_SetPath (
path=0x88c330 L"/home/antoine/cpython/default/usr/lib/python33.zip:/home/antoine/cpython/default/usr/lib/python3.3/:/home/antoine/cpython/default/usr/lib/python3.3/plat-linux2:/home/antoine/cpython/default/usr/lib/py"...)
at ./Modules/getpath.c:729
#2 0x0000000000416600 in main ()

The problem is calling free() on a pointer to statically allocated memory.

Crash fixed in a791dd7d51f3 (3.2) and b4104ffd5127 (3.3), but the original problem remains, or a variant of it. I now get:

Init
Fatal Python error: Py_Initialize: Unable to get the locale encoding
LookupError: no codec search functions registered: can't find encoding
Abandon

Ok, the issue here is that you can't call Py_SetPath() on the point returned by Py_GetPath(), since the memory refered to that pointer is free()d at the beginning of Py_SetPath(). The following works, though:

main(int argc, char **argv)
{
  wchar_t *path, *newpath;
  path = Py_GetPath();
  newpath = malloc((wcslen(path) + 1) * sizeof(wchar_t));
  wcscpy(newpath, path);
  Py_SetPath(newpath);
  free(newpath);
  printf("Init\n");
  Py_Initialize();
  printf("-- END\n");
}

Perhaps we could modify Py_SetPath() so that it copies the new path first before deallocating the old one, but I'm not sure I see the point of calling Py_SetPath() with the pointer returned by Py_GetPath().

Antoine,

Your guess that my issue initially wasn't related to virtualenv is correct (I've never heard about that project before posting this issue...)

As for passing the output of Py_GetPath directly to Py_SetPath: You are right, there is no point in doing this...
Now, I remember that the initial problem I had was the one you reported:
Fatal Python error: Py_Initialize: Unable to get the locale encoding
LookupError: no codec search functions registered: can't find encoding

Shall I create a separate issue to report this problem?

Furthermore I would propose to rename this issue: The problem is not that Py_SetPath cannot be called on pointer returned by Py_GetPath. I think that the problem is more general: Calling Py_SetPath NEVER works.

--> I get the same exception as documented in my initial post when calling Py_SetPyth like this:
   Py_SetPath(L"/usr/labsolution/python32/lib/python3.2/");
or like this:
  wchar_t * path;
  path = malloc(128 * sizeof(wchar_t));
  wcscpy(path,L"/usr/labsolution/python32/lib/python3.2/");
  Py_SetPath(path);

As for this error:
Fatal Python error: Py_Initialize: Unable to get the locale encoding
LookupError: no codec search functions registered: can't find encoding

It seems to me that this error appears if the path passed to Py_SetPath does not point to a valid python lib folder.

As for this error:
Fatal Python error: Py_Initialize: Unable to get the locale encoding
LookupError: no codec search functions registered: can't find encoding

It seems to me that this error appears if the path passed to
Py_SetPath does not point to a valid python lib folder.

Indeed, it does. The error message could perhaps get improved.
What Python does is try to find the current locale's encoding in the
"encodings" package and load it. If it can't find it, it assumes that's
because the encoding is unknown, not because the path is wrong.

Hello everyone,

I have found the reason for the problem.

From the manual http://docs.python.org/py3k/c-api/init.html#Py_SetPath ,
we can see that:

After we call Py_SetPath,both sys.prefix and sys.exec_prefix will be empty.

However, sys.prefix will be used in the sysconfig.py to generate the makefile' s name, and the empty sys.prefix will cause the wrong path: lib/python3.2/config-3.2m/Makefile

sysconfig.py imported by site.py, and site.py used in the Py_InitializeEx.

So ...

The problem seems still not resolved in Python 3.2.6 :-(
The reason has been found by Debao (msg144244). Isn't there anybody motivated to fix this bug?

Unfortunately, the answer to the question "Isn't there anybody motivated to fix this bug?" is "No, not really". As far as I am aware, all of the currently active core developers are primarily interested in the use of the default runtime interpreter as is, rather than embedding it in larger applications (which is the main case where PySys_SetPath is needed).

I'm *personally* interested in that area (hence my intermittent updates to PEP-432), but it's purely on my own time rather than being particularly work related.

That said, Steve Dower did do some significant work to provide an embedding friendly variant of the Windows builds for 3.5, so I've added him to the nosy list here in case he might be able to take a look.

Seems like a fairly obvious bug. From https://docs.python.org/3/c-api/init.html#c.Py_SetPath

This also causes ... sys.prefix and sys.exec_prefix to be empty.
It is up to the caller to modify these if required after calling
Py_Initialize().

Apparently you can't set sys.[exec_]prefix before calling Py_Initialize, and you can't call Py_Initialize without setting sys.[exec_]prefix (much like bpo-20891, where you can't initialize threads without holding the GIL, and you can't acquire the GIL without having initialized threads.)

I don't know how to go about resolving this though (my changes were limited to getpathp.c - and I really need to go add the same changes to the non-Windows getpath.c too...). The best way seems to be forcing Nick to finish PEP-432, but unfortunately I have no leverage over him :)

I think the current situation is mainly an artifact of our relative dearth
of regression testing for embedded configurations, and the fact that we
have very few core developers working for companies embedding CPython in
larger applications.

I do care about that space (hence PEP-432), but operating system
integration, software distribution tools, and improved modularisation take
precedence on work time, and the science & education sectors on my personal
time.

I did recently file bpo-24932 to propose investigating and adopting a C
level unit testing framework for the embedding tests, as one of the things
my preliminary work on PEP-432 highlighted is how limited our current
direct testing capabilities for the embedding API are - at this point,
we're mostly limited to testing it the way CPython uses it, which makes it
unfortunately fragile for embedders (as issues like this one show).

@ncoghlan: Not PySys_SetPath, but Py_SetPath is causing the problem (you mentionned PySys_SetPath in your message msg249311)

I discovered PySys_SetPath only last Friday.
In fact, in my case, the usage of PySys_SetPath (after Py_Initialize) instead of Py_SetPath (before Py_Initialize) is sufficient to respond to my needs!

The cause for this error is any cause for not being able to load a module. This includes version conflicts, so when trying to load a virtualenv with python 3.3 interpreter by a uWSGI embedded python3.4 interpreter the action fails.

This is - bluntly overstated - a design flaw or overly confident claim with respect to virtualenv support in uWSGI. This configuration can never work and becomes a configuration error.

It really deserves a less generic error message, indicating the version conflict and possibly the path to the encoding module it tried to load. This helps in debugging the cause and possibly abandon the idea before wading through the list of reports generated by this error message (only one of which I found to hint at the version conflict and most don't have a clear solution or different root cause).

I understand that this bug has been fixed by the implementation of the PEP-587 in bpo-36763: Py_SethPath() now uses dynamically allocated memory and can be called multiple times. Moreover, the PEP-587 now provides a better API for the Path Configuration. I close the issue.

Reopen/comment the issue if I misunderstood it :-)