Here is a prototype test for embedding and sub-interpreters.

+1 for this kind of tests. 
But I would not have their source in the "official" Modules directory. I expect that there will be several tests of this kind, each one with different modules to import, functions to run, global settings to change.
IMO the C code should be somewhere in the test directory; later we could use templates to generate source code for similar tests.

Is it possible to use distutils to compile this test, and remove it from the Makefile?

There's already precedent for test modules in Modules/ what with xxmodule.c and _testcapimodule.c.

> Is it possible to use distutils to compile this test, and remove it
> from the Makefile?

I'm not aware that distutils is able to compile applications rather than
extension modules, but that would certainly be better than a rule in the
Makefile, yes.

In distutils.command.config, config().try_run(body) is probably what we need. Now, I don't know how to use it...

I like the idea of adding these tests as well.

Probably worth bringing up on python-dev - folks like Tarek should definitely be able to help with alternative ways of building a test application for this.

There is also always the possibility of writing it in Python and using ctypes to get at the necessary C API calls to create subinterpreters.

> There is also always the possibility of writing it in Python and using 
> ctypes to get at the necessary C API calls to create subinterpreters.

Well, I don't think calling Py_Initialize, Py_Finalize and Py_NewInterpreter from pure Python code is a very good idea.

Tarek, can you help? We'd like to use distutils to compile an executable embedding the interpreter.

New changeset a791dd7d51f3 by Antoine Pitrou in branch '3.2':
Issue #10914: fix bogus memory management in Modules/getpath.c, leading to a possible crash when calling Py_SetPath()
http://hg.python.org/cpython/rev/a791dd7d51f3

Ouch, sorry, I got my commit message wrong.

distutils sure knows how to build .o or .so files, but I don’t know about standalone executables (because I don’t know how the .o end up making an executable).  If you want to try to do it, I would advise you not to use the config command but rather a compiler object directly.  distutils.ccompiler.new_compiler() should give you an instance of a subclass of distutils.ccompiler.CCompiler suitable for your system, and then you can use help or the source to find what methods to call.

> distutils sure knows how to build .o or .so files, but I don’t know
> about standalone executables (because I don’t know how the .o end up
> making an executable).  If you want to try to do it, I would advise
> you not to use the config command but rather a compiler object
> directly.  distutils.ccompiler.new_compiler() should give you an
> instance of a subclass of distutils.ccompiler.CCompiler suitable for
> your system, and then you can use help or the source to find what
> methods to call.

Well, config._link() seems to do what is needed here. If you think it is
bad to rely on it, perhaps we should "inline" its code somehow in the
test module.

> Well, config._link() seems to do what is needed here.
My point is that it’s easier to write a few lines of code directly using a compiler object (copying and simplifying code from try_run or _link) than go through the distutils command machinery.  Both are doable, but I think the former is easier.

Ok, this is what I came up with to build an exe using distutils. At this point, the complication is downright silly and it doesn't even work under Windows, so we may prefer to go down the Makefile route instead.

Note that I think it would be a perfectly reasonable feature request to better isolate and improve the compiler in packaging/distutils2.  (This does not mean this issue should be delayed.)

After wrestling with startup issues on the OS X buildbots, here is a new patch, tested on several different UNIX platforms.

Unless someone objects, I would like to commit that latest patch, since at least it enables the start of a regression suite for Python embedding.
(it also allowed me to notice how crufty and incredibly obscure the getpath.c mechanisms are)

Sounds good. I still have that embedded pickle module issue to deal with, and this should let me actually add a test for it along with the fix.

New changeset d195ff5c44f4 by Antoine Pitrou in branch '3.2':
Issue #10914: Add a minimal embedding test to test_capi.
http://hg.python.org/cpython/rev/d195ff5c44f4

New changeset 77cf9e4b144b by Antoine Pitrou in branch '3.2':
Issue #10914: add NEWS item.
http://hg.python.org/cpython/rev/77cf9e4b144b

New changeset ef1ad39e3c40 by Antoine Pitrou in branch 'default':
Issue #10914: Add a minimal embedding test to test_capi.
http://hg.python.org/cpython/rev/ef1ad39e3c40

Should be fixed.

There's a failure on certain locales that Victor, I believe, is currently investigating.

Modules/_testembed fails with ISO-8859-15 locale encoding because of a bootstrap issue. The problem is that the filesystem encoding codec is implemented in Python: Python requires the codec to loads modules, but it has to load a module to load the codec. I already solved this issue in Python using C functions instead of the Python codec until Python is able to load modules (able to load the Python codec).

The problem is the test on the "python initialization": PyUnicode_EncodeFSDefault / PyUnicode_DecodeFSDefaultAndSize check the global Py_FileSystemDefaultEncoding variable (use C functions if it is NULL).

We should use C functions if Py_FileSystemDefaultEncoding is NULL *or* if interp->codecs_initialized is zero.

I think that Python used interp->codecs_initialized flag. I removed the test on interp->codecs_initialized because I thought that it was useless.

> I think that Python used interp->codecs_initialized flag

Yes in PyUnicode_AsEncodedString(), in Python 3.0 and 3.1. Python 3.2 has also the test, but PyUnicode_AsEncodedString() is no more used to encode filenames. I removed the test in Python 3.3.

Extract of Python 3.1:

        /* During bootstrap, we may need to find the encodings
           package, to load the file system encoding, and require the
           file system encoding in order to load the encodings
           package.

           Break out of this dependency by assuming that the path to
           the encodings module is ASCII-only.  XXX could try wcstombs
           instead, if the file system encoding is the locale's
           encoding. */
        else if (Py_FileSystemDefaultEncoding &&
                 strcmp(encoding, Py_FileSystemDefaultEncoding) == 0 &&
                 !PyThreadState_GET()->interp->codecs_initialized)
            return PyUnicode_AsASCIIString(unicode);

I implemented the "XXX could try wcstombs" part, but in new functions:  PyUnicode_EncodeFSDefault and PyUnicode_DecodeFSDefaultAndSize.

Hmmm, I wander if that is related to the workaround I have added in mod_wsgi recently of:

    /*
     * Force loading of codecs into interpreter. This has to be
     * done as not otherwise done in sub interpreters and if not
     * done, code running in sub interpreters can fail on some
     * platforms if a unicode string is added in sys.path and an
     * import then done.
     */

    item = PyCodec_Encoder("ascii");
    Py_XDECREF(item);

This fixes problem some have seen where one gets:

  LookupError: no codec search functions registered: can't find encoding

I haven't been able to reproduce the problem myself so no bug report ever lodged.

Have been told it affects some other embedded systems as well which use sub interpreters but other systems don't employ the workaround that I am now using.

> The problem is the test on the "python initialization":
> PyUnicode_EncodeFSDefault / PyUnicode_DecodeFSDefaultAndSize
> check the global Py_FileSystemDefaultEncoding variable (use
> C functions if it is NULL).

The problem is a little bit more complex. interp->codecs_initialized is set to 1 by _PyCodecRegistry_Init(), but the filesystem codec is not loaded yet. We need another variable: interp->fscodec_initialized.

fscodec_subinterpreter.patch fixes this issue.

Note: the patch replaces also PyErr_Print() by PyErr_PrintEx(0) to avoid a crash on error, if PyErr_PrintEx(0) is called before the sys module is initialized. For example, if PyModule_GetDict(_PyImport_FindBuiltin("builtins")) is NULL. Anyway, it's useless to set sys attributes on error, because sys is destroyed just after printing the error.

New changeset c8338cfa3578 by Victor Stinner in branch 'default':
Issue #10914: Py_NewInterpreter() uses PyErr_PrintEx(0)
http://hg.python.org/cpython/rev/c8338cfa3578

New changeset d3af2a2b621b by Victor Stinner in branch 'default':
Issue #10914: Initialize correctly the filesystem codec when creating a new
http://hg.python.org/cpython/rev/d3af2a2b621b

test_capi pass on x86 debian parallel 3.x: I close this issue again :-)

Victor, the fix needs to go into 3.2 as well.

New changeset 5a983773c09a by Victor Stinner in branch '3.2':
Issue #10914: Py_NewInterpreter() uses PyErr_PrintEx(0)
http://hg.python.org/cpython/rev/5a983773c09a

New changeset 2caf82aee7a4 by Victor Stinner in branch '3.2':
Issue #10914: Initialize correctly the filesystem codec when creating a new
http://hg.python.org/cpython/rev/2caf82aee7a4

> Victor, the fix needs to go into 3.2 as well.

Oh, I thought that Modules/_testembed.c was only in 3.3. Anyway, it is a real bug in 3.2, and it is now fixed (I backported the 2 fixes in 3.2).

BTW, about this comment in the code:
# XXX only tested under Unix checkouts
It is possible to install the _testembed.c file alongside xxmodule.c and compile it at test time.  To test under Windows, I’ve re-read your first patch using distutils and it’s not that long or hard to read after all.

> It is possible to install the _testembed.c file alongside xxmodule.c
> and compile it at test time.

We probably could, using the approach from the distutils patch. But it feels really quirky to me. Granted, the Makefile is quirky as well.

> To test under Windows, I’ve re-read your first patch using distutils
> and it’s not that long or hard to read after all.

Except that it didn't work under Windows...

> Except that it didn't work under Windows...
Ah.  If it was an extension module, I’d have a clue (the debug thing), but I know nothing about embedding.  An alternative would be to install the file with the msi system and use a project file to have it compiled (sounds like much hassle).

> Ah.  If it was an extension module, I’d have a clue (the debug thing),
> but I know nothing about embedding.  An alternative would be to
> install the file with the msi system and use a project file to have it
> compiled (sounds like much hassle).

Agreed, but it's not something that I know how to do :)
(and yet, of course, it would be good to test that embedding works under
Windows too, as well as under Unix installs)