classification
Title: Python sub-interpreter test
Type: Stage: resolved
Components: Interpreter Core, Tests Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: haypo Nosy List: amaury.forgeotdarc, benjamin.peterson, christian.heimes, eric.araujo, eric.snow, grahamd, haypo, loewis, ncoghlan, pitrou, python-dev, tarek
Priority: normal Keywords: patch

Created on 2011-01-15 13:38 by pitrou, last changed 2012-11-28 06:45 by eric.snow. This issue is now closed.

Files
File name Uploaded Description Edit
embedtest.patch pitrou, 2011-01-15 13:38
embedtest-distutils.patch pitrou, 2011-03-18 19:48 review
embedtest2.patch pitrou, 2011-04-24 17:08 review
fscodec_subinterpreter.patch haypo, 2011-04-25 23:27 review
Messages (33)
msg126328 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-15 13:38
Here is a prototype test for embedding and sub-interpreters.
msg126380 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-01-16 22:16
+1 for this kind of tests. 
But I would not have their source in the "official" Modules directory. I expect that there will be several tests of this kind, each one with different modules to import, functions to run, global settings to change.
IMO the C code should be somewhere in the test directory; later we could use templates to generate source code for similar tests.

Is it possible to use distutils to compile this test, and remove it from the Makefile?
msg126382 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2011-01-16 22:27
There's already precedent for test modules in Modules/ what with xxmodule.c and _testcapimodule.c.
msg126384 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-16 22:32
> Is it possible to use distutils to compile this test, and remove it
> from the Makefile?

I'm not aware that distutils is able to compile applications rather than
extension modules, but that would certainly be better than a rule in the
Makefile, yes.
msg126385 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-01-16 22:46
In distutils.command.config, config().try_run(body) is probably what we need. Now, I don't know how to use it...
msg126401 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-01-17 15:02
I like the idea of adding these tests as well.

Probably worth bringing up on python-dev - folks like Tarek should definitely be able to help with alternative ways of building a test application for this.

There is also always the possibility of writing it in Python and using ctypes to get at the necessary C API calls to create subinterpreters.
msg126403 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-17 15:14
> There is also always the possibility of writing it in Python and using 
> ctypes to get at the necessary C API calls to create subinterpreters.

Well, I don't think calling Py_Initialize, Py_Finalize and Py_NewInterpreter from pure Python code is a very good idea.

Tarek, can you help? We'd like to use distutils to compile an executable embedding the interpreter.
msg131295 - (view) Author: Roundup Robot (python-dev) Date: 2011-03-17 22:37
New changeset a791dd7d51f3 by Antoine Pitrou in branch '3.2':
Issue #10914: fix bogus memory management in Modules/getpath.c, leading to a possible crash when calling Py_SetPath()
http://hg.python.org/cpython/rev/a791dd7d51f3
msg131296 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-03-17 22:40
Ouch, sorry, I got my commit message wrong.
msg131302 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-03-17 23:36
distutils sure knows how to build .o or .so files, but I don’t know about standalone executables (because I don’t know how the .o end up making an executable).  If you want to try to do it, I would advise you not to use the config command but rather a compiler object directly.  distutils.ccompiler.new_compiler() should give you an instance of a subclass of distutils.ccompiler.CCompiler suitable for your system, and then you can use help or the source to find what methods to call.
msg131303 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-03-17 23:53
> distutils sure knows how to build .o or .so files, but I don’t know
> about standalone executables (because I don’t know how the .o end up
> making an executable).  If you want to try to do it, I would advise
> you not to use the config command but rather a compiler object
> directly.  distutils.ccompiler.new_compiler() should give you an
> instance of a subclass of distutils.ccompiler.CCompiler suitable for
> your system, and then you can use help or the source to find what
> methods to call.

Well, config._link() seems to do what is needed here. If you think it is
bad to rely on it, perhaps we should "inline" its code somehow in the
test module.
msg131321 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-03-18 08:48
> Well, config._link() seems to do what is needed here.
My point is that it’s easier to write a few lines of code directly using a compiler object (copying and simplifying code from try_run or _link) than go through the distutils command machinery.  Both are doable, but I think the former is easier.
msg131360 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-03-18 19:48
Ok, this is what I came up with to build an exe using distutils. At this point, the complication is downright silly and it doesn't even work under Windows, so we may prefer to go down the Makefile route instead.
msg132106 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-03-25 16:02
Note that I think it would be a perfectly reasonable feature request to better isolate and improve the compiler in packaging/distutils2.  (This does not mean this issue should be delayed.)
msg134334 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-04-24 17:08
After wrestling with startup issues on the OS X buildbots, here is a new patch, tested on several different UNIX platforms.
msg134335 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-04-24 17:13
Unless someone objects, I would like to commit that latest patch, since at least it enables the start of a regression suite for Python embedding.
(it also allowed me to notice how crufty and incredibly obscure the getpath.c mechanisms are)
msg134336 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-04-24 17:30
Sounds good. I still have that embedded pickle module issue to deal with, and this should let me actually add a test for it along with the fix.
msg134404 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-25 19:23
New changeset d195ff5c44f4 by Antoine Pitrou in branch '3.2':
Issue #10914: Add a minimal embedding test to test_capi.
http://hg.python.org/cpython/rev/d195ff5c44f4

New changeset 77cf9e4b144b by Antoine Pitrou in branch '3.2':
Issue #10914: add NEWS item.
http://hg.python.org/cpython/rev/77cf9e4b144b

New changeset ef1ad39e3c40 by Antoine Pitrou in branch 'default':
Issue #10914: Add a minimal embedding test to test_capi.
http://hg.python.org/cpython/rev/ef1ad39e3c40
msg134405 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-04-25 19:25
Should be fixed.
msg134413 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-04-25 21:06
There's a failure on certain locales that Victor, I believe, is currently investigating.
msg134417 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-04-25 21:28
Modules/_testembed fails with ISO-8859-15 locale encoding because of a bootstrap issue. The problem is that the filesystem encoding codec is implemented in Python: Python requires the codec to loads modules, but it has to load a module to load the codec. I already solved this issue in Python using C functions instead of the Python codec until Python is able to load modules (able to load the Python codec).

The problem is the test on the "python initialization": PyUnicode_EncodeFSDefault / PyUnicode_DecodeFSDefaultAndSize check the global Py_FileSystemDefaultEncoding variable (use C functions if it is NULL).

We should use C functions if Py_FileSystemDefaultEncoding is NULL *or* if interp->codecs_initialized is zero.

I think that Python used interp->codecs_initialized flag. I removed the test on interp->codecs_initialized because I thought that it was useless.
msg134421 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-04-25 21:51
> I think that Python used interp->codecs_initialized flag

Yes in PyUnicode_AsEncodedString(), in Python 3.0 and 3.1. Python 3.2 has also the test, but PyUnicode_AsEncodedString() is no more used to encode filenames. I removed the test in Python 3.3.

Extract of Python 3.1:

        /* During bootstrap, we may need to find the encodings
           package, to load the file system encoding, and require the
           file system encoding in order to load the encodings
           package.

           Break out of this dependency by assuming that the path to
           the encodings module is ASCII-only.  XXX could try wcstombs
           instead, if the file system encoding is the locale's
           encoding. */
        else if (Py_FileSystemDefaultEncoding &&
                 strcmp(encoding, Py_FileSystemDefaultEncoding) == 0 &&
                 !PyThreadState_GET()->interp->codecs_initialized)
            return PyUnicode_AsASCIIString(unicode);

I implemented the "XXX could try wcstombs" part, but in new functions:  PyUnicode_EncodeFSDefault and PyUnicode_DecodeFSDefaultAndSize.
msg134423 - (view) Author: Graham Dumpleton (grahamd) Date: 2011-04-25 22:01
Hmmm, I wander if that is related to the workaround I have added in mod_wsgi recently of:

    /*
     * Force loading of codecs into interpreter. This has to be
     * done as not otherwise done in sub interpreters and if not
     * done, code running in sub interpreters can fail on some
     * platforms if a unicode string is added in sys.path and an
     * import then done.
     */

    item = PyCodec_Encoder("ascii");
    Py_XDECREF(item);

This fixes problem some have seen where one gets:

  LookupError: no codec search functions registered: can't find encoding

I haven't been able to reproduce the problem myself so no bug report ever lodged.

Have been told it affects some other embedded systems as well which use sub interpreters but other systems don't employ the workaround that I am now using.
msg134425 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-04-25 23:27
> The problem is the test on the "python initialization":
> PyUnicode_EncodeFSDefault / PyUnicode_DecodeFSDefaultAndSize
> check the global Py_FileSystemDefaultEncoding variable (use
> C functions if it is NULL).

The problem is a little bit more complex. interp->codecs_initialized is set to 1 by _PyCodecRegistry_Init(), but the filesystem codec is not loaded yet. We need another variable: interp->fscodec_initialized.

fscodec_subinterpreter.patch fixes this issue.

Note: the patch replaces also PyErr_Print() by PyErr_PrintEx(0) to avoid a crash on error, if PyErr_PrintEx(0) is called before the sys module is initialized. For example, if PyModule_GetDict(_PyImport_FindBuiltin("builtins")) is NULL. Anyway, it's useless to set sys attributes on error, because sys is destroyed just after printing the error.
msg134510 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-26 22:24
New changeset c8338cfa3578 by Victor Stinner in branch 'default':
Issue #10914: Py_NewInterpreter() uses PyErr_PrintEx(0)
http://hg.python.org/cpython/rev/c8338cfa3578

New changeset d3af2a2b621b by Victor Stinner in branch 'default':
Issue #10914: Initialize correctly the filesystem codec when creating a new
http://hg.python.org/cpython/rev/d3af2a2b621b
msg134515 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-04-26 22:39
test_capi pass on x86 debian parallel 3.x: I close this issue again :-)
msg134606 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-04-27 17:48
Victor, the fix needs to go into 3.2 as well.
msg134855 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-30 12:17
New changeset 5a983773c09a by Victor Stinner in branch '3.2':
Issue #10914: Py_NewInterpreter() uses PyErr_PrintEx(0)
http://hg.python.org/cpython/rev/5a983773c09a

New changeset 2caf82aee7a4 by Victor Stinner in branch '3.2':
Issue #10914: Initialize correctly the filesystem codec when creating a new
http://hg.python.org/cpython/rev/2caf82aee7a4
msg134857 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-04-30 12:21
> Victor, the fix needs to go into 3.2 as well.

Oh, I thought that Modules/_testembed.c was only in 3.3. Anyway, it is a real bug in 3.2, and it is now fixed (I backported the 2 fixes in 3.2).
msg143464 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-09-03 16:06
BTW, about this comment in the code:
# XXX only tested under Unix checkouts
It is possible to install the _testembed.c file alongside xxmodule.c and compile it at test time.  To test under Windows, I’ve re-read your first patch using distutils and it’s not that long or hard to read after all.
msg143498 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-09-04 16:48
> It is possible to install the _testembed.c file alongside xxmodule.c
> and compile it at test time.

We probably could, using the approach from the distutils patch. But it feels really quirky to me. Granted, the Makefile is quirky as well.

> To test under Windows, I’ve re-read your first patch using distutils
> and it’s not that long or hard to read after all.

Except that it didn't work under Windows...
msg143518 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-09-05 14:40
> Except that it didn't work under Windows...
Ah.  If it was an extension module, I’d have a clue (the debug thing), but I know nothing about embedding.  An alternative would be to install the file with the msi system and use a project file to have it compiled (sounds like much hassle).
msg143519 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-09-05 14:48
> Ah.  If it was an extension module, I’d have a clue (the debug thing),
> but I know nothing about embedding.  An alternative would be to
> install the file with the msi system and use a project file to have it
> compiled (sounds like much hassle).

Agreed, but it's not something that I know how to do :)
(and yet, of course, it would be good to test that embedding works under
Windows too, as well as under Unix installs)
History
Date User Action Args
2012-11-28 06:45:09eric.snowsetnosy: + eric.snow
2012-11-26 20:00:28pitroulinkissue1977 superseder
2011-09-05 14:48:08pitrousetmessages: + msg143519
2011-09-05 14:40:42eric.araujosetmessages: + msg143518
2011-09-04 16:48:14pitrousetmessages: + msg143498
2011-09-03 16:06:59eric.araujosetmessages: + msg143464
2011-04-30 12:21:19hayposetstatus: open -> closed

messages: + msg134857
2011-04-30 12:17:48python-devsetmessages: + msg134855
2011-04-27 17:48:11pitrousetstatus: closed -> open
assignee: haypo
messages: + msg134606
2011-04-26 22:39:21hayposetstatus: open -> closed

messages: + msg134515
2011-04-26 22:24:25python-devsetmessages: + msg134510
2011-04-25 23:27:34hayposetfiles: + fscodec_subinterpreter.patch

messages: + msg134425
2011-04-25 22:01:13grahamdsetmessages: + msg134423
2011-04-25 21:51:08hayposetmessages: + msg134421
2011-04-25 21:28:56hayposetmessages: + msg134417
2011-04-25 21:06:28pitrousetstatus: pending -> open

messages: + msg134413
2011-04-25 19:25:04pitrousetstatus: open -> pending
resolution: fixed
messages: + msg134405

stage: resolved
2011-04-25 19:23:41python-devsetmessages: + msg134404
2011-04-24 17:30:25ncoghlansetmessages: + msg134336
2011-04-24 17:13:26pitrousetmessages: + msg134335
2011-04-24 17:08:15pitrousetfiles: + embedtest2.patch

messages: + msg134334
2011-03-25 16:02:32eric.araujosetmessages: + msg132106
2011-03-18 19:48:14pitrousetfiles: + embedtest-distutils.patch
nosy: loewis, amaury.forgeotdarc, ncoghlan, pitrou, haypo, christian.heimes, benjamin.peterson, tarek, eric.araujo, grahamd, python-dev
messages: + msg131360
2011-03-18 08:48:22eric.araujosetnosy: loewis, amaury.forgeotdarc, ncoghlan, pitrou, haypo, christian.heimes, benjamin.peterson, tarek, eric.araujo, grahamd, python-dev
messages: + msg131321
2011-03-17 23:53:46pitrousetnosy: loewis, amaury.forgeotdarc, ncoghlan, pitrou, haypo, christian.heimes, benjamin.peterson, tarek, eric.araujo, grahamd, python-dev
messages: + msg131303
2011-03-17 23:36:40eric.araujosetnosy: + eric.araujo
messages: + msg131302
2011-03-17 22:40:18pitrousetnosy: loewis, amaury.forgeotdarc, ncoghlan, pitrou, haypo, christian.heimes, benjamin.peterson, tarek, grahamd, python-dev
messages: + msg131296
2011-03-17 22:37:22python-devsetnosy: + python-dev
messages: + msg131295
2011-01-17 15:14:05pitrousetnosy: + tarek
messages: + msg126403
2011-01-17 15:02:08ncoghlansetnosy: loewis, amaury.forgeotdarc, ncoghlan, pitrou, haypo, christian.heimes, benjamin.peterson, grahamd
messages: + msg126401
2011-01-17 14:57:03ncoghlanlinkissue10915 dependencies
2011-01-16 22:46:49amaury.forgeotdarcsetnosy: loewis, amaury.forgeotdarc, ncoghlan, pitrou, haypo, christian.heimes, benjamin.peterson, grahamd
messages: + msg126385
2011-01-16 22:32:56pitrousetnosy: loewis, amaury.forgeotdarc, ncoghlan, pitrou, haypo, christian.heimes, benjamin.peterson, grahamd
messages: + msg126384
2011-01-16 22:27:18benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg126382
2011-01-16 22:16:40amaury.forgeotdarcsetnosy: loewis, amaury.forgeotdarc, ncoghlan, pitrou, haypo, christian.heimes, grahamd
messages: + msg126380
2011-01-15 14:53:35pitrousetnosy: + amaury.forgeotdarc, ncoghlan
2011-01-15 13:38:37pitroucreate