classification
Title: Py_NewInterpreter does not work
Type: behavior Stage:
Components: Extension Modules Versions: Python 3.0
process
Status: closed Resolution: fixed
Dependencies: 4213 Superseder:
Assigned To: barry Nosy List: amaury.forgeotdarc, barry, benjamin.peterson, christian.heimes, djc, grahamd, loewis
Priority: release blocker Keywords: needs review, patch

Created on 2008-08-29 12:11 by amaury.forgeotdarc, last changed 2008-10-30 21:48 by christian.heimes. This issue is now closed.

Files
File name Uploaded Description Edit
add_init_funcs.patch benjamin.peterson, 2008-09-18 22:15
pythonrun.c.diff grahamd, 2008-09-30 12:31
importexc.diff loewis, 2008-10-17 15:47
subinterpreter.patch christian.heimes, 2008-10-26 22:58
Messages (17)
msg72127 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-08-29 12:11
The example Demo/embed/importexc.c crashes, because Py_NewInterpreter
cannot reimport builtins and sys modules. This problem seems important
for embedding applications like mod_python, for example.

(the "import exceptions" statement does not work with python 3.0, but
replacing with e.g. "import types" does not change anything: the
interpreter is not correctly renewed)

I tried to put "-1" in the m_size structure of these modules, and they
seem to import correctly. However, "builtins" comes with its original
dictionary - without the standard exceptions.

I think that these modules should be made re-importable, with specific
functions.

Maybe two related problems:
- once you've done
    del sys.modules['sys']
it's not possible to get it back, "import sys" raises an error...

- the usual trick to call sys.setdefaultencoding will not work, since it
needs to "imp.reload(sys)"
msg73403 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-09-18 22:15
Maybe, I'm not seeing the whole problem, but can't we just add
_PySys_Init and _PyBuiltin_Init to config.c like in the attached patch?
Obviously, we will eventually want to make a separate state to store
module globals in, but I think this will work for 3.0 final.
msg73530 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-09-21 21:02
I applied the patch to PC/config.c, but this did not change anything.
msg73531 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-09-21 21:15
Interesting, here it lets import.c's init_builtin reinitalize modules...
msg74069 - (view) Author: Graham Dumpleton (grahamd) Date: 2008-09-30 12:15
Adding the functions as initfunc in module init table is of no use as 
they aren't invoked when creating a sub interpreter.

One thing that does appear to work, although no idea of whether it is 
correct way to solve problem, is to duplicate the builtin/sys 
initialisation that occurs in Py_InitializeEx() function.

Attached diff shows nature of changes. Diff is bit messy as have left 
existing code in there but #ifdef'd out.

Maybe this will give someone who knows how overall interpreter 
initialisation is supposed to work a head start on coming up with proper 
fix. But then it could be totally wrong as well.

At least with change as is, mod_wsgi works for sub interpreters now. 
I'll do more work later on whether it is correct way to solve it.
msg74070 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-09-30 12:22
Your patch may go in the right direction, but please provide only
context diff or unified diff files. 
Use "diff -du", or "svn diff" to generate the file.
msg74071 - (view) Author: Graham Dumpleton (grahamd) Date: 2008-09-30 12:26
Argh. Personally I like to provide context diff's but more often than not 
get abused for providing them over a unified diff. Was in a hurry this 
time as had only a couple of minutes of battery life left on the laptop, 
so quickly did it without thinking and then ran off to find a power point. 
:-)
msg74072 - (view) Author: Graham Dumpleton (grahamd) Date: 2008-09-30 12:31
Unified diff now attached.
msg74919 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-10-17 15:47
Here is a patch that fixed importexc.c. It consists of the following parts:
- set m_size of the builtins module and the sys module to -1, indicating
that these modules don't support repeated initialization. This should be
reviewed; perhaps it's better (and necessary) to record the init
function not only for dynamically-loaded modules in m_init, but also for
statically linked ones, so that the reinit code can call them again
(whether or not it is safe to call them again for sys and builtins
should then be studied).
- add a field to the interpreter state indicating that the codecs are
not ready. when trying to use the file system encoding in this state,
use ASCII instead.
- Fix importexc to use the types module.
msg74938 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-10-17 21:24
I think the patch goes in the right direction.

But in addition, Py_NewInterpreter() has to call initstdio() between initmain() and initsite() (the same sequence as in Py_InitializeEx)
Found by using the following command string in importexc.c:
  "import types; print(types.XXX)"
msg74939 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-10-17 21:27
Wouldn't it make more sense to move interpreter initialization things to
Py_NewInterpreter and call it from Py_InitializeEx?
msg74940 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-10-17 21:29
Sounds like a good plan, Benjamin
msg74943 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-10-17 22:34
> Wouldn't it make more sense to move interpreter initialization things to
> Py_NewInterpreter and call it from Py_InitializeEx?

Can you propose a specific patch? I'm worried that doing so blindly
introduces other bugs.
msg75248 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-10-26 22:58
The patch "subinterpreter.patch" is based on Martin's patch
"importexc.diff". The patch contains additional code to setup a
preliminary stderr object and a call to initstdio(). Amaury is right. I
had to add initstdio() to initialize the standard streams. But I can't
get it to work.

$ rm -f Demo/embed/importexc.o; cd Demo/embed; make; ./importexc; cd ../..
gcc -g -I../../Include -I../..  -c -o importexc.o importexc.c
gcc -Xlinker -export-dynamic importexc.o ../../libpython3.0.a  -lnsl
-ldl -lreadline -ltermcap -lieee -lpthread -lutil -lm -o importexc
Initialize interpreter
<module 'types' from '/usr/local/lib/python3.0/types.py'>

Initialize subinterpreter
Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
  File "/usr/local/lib/python3.0/encodings/__init__.py", line 32, in
<module>
ValueError: Cannot encode path item
Aborted
msg75277 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-10-28 11:20
In combination with the patch in #4213, "subinterpreter.patch" fixes the
problem.

I'm assigning the bug to Barry for his final decision.
msg75338 - (view) Author: Graham Dumpleton (grahamd) Date: 2008-10-30 00:10
In conjunction with #4213, the attached subinterpreter.patch appears to 
fix issue for mod_wsgi.
msg75385 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-10-30 21:48
Applied in r67057
History
Date User Action Args
2008-10-30 21:48:53christian.heimessetstatus: open -> closed
resolution: fixed
messages: + msg75385
2008-10-30 00:10:06grahamdsetmessages: + msg75338
2008-10-28 11:20:44christian.heimessetassignee: barry
type: behavior
messages: + msg75277
nosy: + barry
2008-10-27 13:29:28christian.heimessetdependencies: + Lower case file system encoding
2008-10-26 22:58:41christian.heimessetfiles: + subinterpreter.patch
messages: + msg75248
2008-10-17 22:34:12loewissetmessages: + msg74943
2008-10-17 21:29:10christian.heimessetnosy: + christian.heimes
messages: + msg74940
2008-10-17 21:27:42benjamin.petersonsetmessages: + msg74939
2008-10-17 21:24:35amaury.forgeotdarcsetmessages: + msg74938
2008-10-17 15:47:28loewissetfiles: + importexc.diff
messages: + msg74919
2008-10-02 12:56:08barrysetpriority: deferred blocker -> release blocker
2008-09-30 12:31:04grahamdsetfiles: + pythonrun.c.diff
messages: + msg74072
2008-09-30 12:30:23grahamdsetfiles: - pythonrun.c.diff
2008-09-30 12:27:00grahamdsetmessages: + msg74071
2008-09-30 12:22:01amaury.forgeotdarcsetmessages: + msg74070
2008-09-30 12:15:30grahamdsetfiles: + pythonrun.c.diff
nosy: + grahamd
messages: + msg74069
2008-09-26 22:21:04barrysetpriority: release blocker -> deferred blocker
2008-09-25 12:28:43djcsetnosy: + djc
2008-09-21 21:15:31benjamin.petersonsetmessages: + msg73531
2008-09-21 21:02:27amaury.forgeotdarcsetmessages: + msg73530
2008-09-21 20:06:30benjamin.petersonsetkeywords: + needs review
2008-09-21 19:29:24amaury.forgeotdarclinkissue3919 superseder
2008-09-18 22:15:12benjamin.petersonsetfiles: + add_init_funcs.patch
nosy: + benjamin.peterson
messages: + msg73403
keywords: + patch
2008-09-18 05:41:24barrysetpriority: deferred blocker -> release blocker
2008-09-04 01:12:11benjamin.petersonsetpriority: release blocker -> deferred blocker
2008-08-29 12:11:09amaury.forgeotdarccreate