classification
Title: second python execution fails when embedding
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.4, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, amaury.forgeotdarc, isoschiz, ncoghlan, ned.deily, pconnell, pitrou, python-dev, terry.reedy, theDarkBrainer
Priority: normal Keywords:

Created on 2013-03-13 14:09 by theDarkBrainer, last changed 2013-05-05 06:22 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
Python33Test.zip theDarkBrainer, 2013-03-13 14:18
Messages (10)
msg184081 - (view) Author: Vlad (theDarkBrainer) Date: 2013-03-13 14:09
This issue is for Python3.3 and doesn't exist in Python3.2

Detailed description with source code can be found here:
http://stackoverflow.com/questions/15387035/second-python-execution-fails
msg184082 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-03-13 14:13
Please add the detailed description of the problem and any test files to the issue here.  Information stored off-site is not searchable within the issue tracker and may not be permanently available.
msg184083 - (view) Author: Vlad (theDarkBrainer) Date: 2013-03-13 14:18
I'm trying to embed the python 3.3 engine for an app that need to run custom scripts in python. Since the scripts might be completely different, and sometimes user provided, I am trying to make each execution isolated and there is not need to preserve any data between execution of the different scripts.

So, my solution is to wrap each execution between 'Py_Initialize' and 'Py_Finalize'. It looks something like that:

  void ExecuteScript(const char* script)
  {
    Py_Initialize();
  
    PyRun_SimpleString( script );
  
    Py_Finalize();
  }

However, this fails for a particular python script the second time a script is executed with:

  done!
  Traceback (most recent call last):
    File "<string>", line 8, in <module>
    File "\Python33Test\Output\Debug\Python33\Lib\copy.py", line 89, in copy
      rv = reductor(2)
  TypeError: attribute of type 'NoneType' is not callable


The python script looks like this:

  class Data:
      value1 = 'hello'
      value2 = 0
  
  import copy
  
  d = Data()
  dd = copy.copy( d )
  print ( 'done!' )

As you can see, the first time around the script was executed the 'done!' was printed out. But the second time it rises an exception inside the copy function.

It looks like the python engine was left in some weird state after the first initialize-finalize. Note, this is python 3.3.

Also, it is very interesting to note that Python 2.7 and Python 3.2 did not have this problem.

I guess there might be other examples that could reveal better what's going, but i haven't had the time to find yet.

I also put a copy of the project containing flag to switch between Python 3 and Python 2.7 (the file is 31 MB): https://docs.google.com/file/d/0B86-G0mwwxZvbWRldTd5b2NNMWM/edit?usp=sharing

Thanks, Vlad
msg184085 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-03-13 15:15
Reproduced on Linux.
The reason is in Objects/typeobject.c: import_copyreg() has a static cache of the copyreg module.
When the interpreter stops, the module is filled with None... but gets reused in the next instance.

Resetting this "mod_copyreg" variable to NULL fixes the issue.
pickle is also broken for the same reason.
msg184261 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-03-15 21:25
Can you tell if the relevant 3.2 to 3.3 change is in Py_Initialize, Py_Finalize(), or typeobject.c?
msg184299 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-03-16 08:17
In 3.2, typeobject.c did not cache the copyreg module in import_copyreg(); PyImport_Import was always called.
msg188393 - (view) Author: Roundup Robot (python-dev) Date: 2013-05-04 18:46
New changeset 8c1385205a35 by Antoine Pitrou in branch '3.3':
Issue #17408: Avoid using an obsolete instance of the copyreg module when the interpreter is shutdown and then started again.
http://hg.python.org/cpython/rev/8c1385205a35

New changeset 0b34fd75b937 by Antoine Pitrou in branch 'default':
Issue #17408: Avoid using an obsolete instance of the copyreg module when the interpreter is shutdown and then started again.
http://hg.python.org/cpython/rev/0b34fd75b937
msg188395 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-05-04 18:47
Thank you for reporting! This should be fixed now.
msg188427 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2013-05-05 04:57
Commit 8c1385205a35 causes segmentation fault (in non-debug build) or abort (in debug build) during interpreter shutdown after copying of e.g. bytes or object().

$ python -c 'import copy; copy.copy(b""); print("text")'
text
Segmentation fault
$ python -c 'import copy; copy.copy(object()); print("text")'
text
Segmentation fault
msg188428 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-05-05 06:22
Should be fixed in 7de9852cdc0e, sorry.
History
Date User Action Args
2013-05-05 06:22:49pitrousetstatus: open -> closed
resolution: fixed
messages: + msg188428

stage: resolved
2013-05-05 04:57:25Arfreversetstatus: closed -> open

nosy: + Arfrever
messages: + msg188427

resolution: fixed -> (no value)
stage: resolved -> (no value)
2013-05-04 18:47:28pitrousetstatus: open -> closed

components: + Interpreter Core, - None
versions: + Python 3.4
nosy: + pitrou

messages: + msg188395
resolution: fixed
stage: needs patch -> resolved
2013-05-04 18:46:34python-devsetnosy: + python-dev
messages: + msg188393
2013-04-20 10:55:57isoschizsetnosy: + pconnell, isoschiz
2013-03-19 07:37:16ezio.melottisetstage: needs patch
2013-03-16 08:17:57amaury.forgeotdarcsetmessages: + msg184299
2013-03-15 21:25:55terry.reedysetnosy: + terry.reedy, ncoghlan
messages: + msg184261
2013-03-13 15:15:18amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg184085
2013-03-13 14:18:37theDarkBrainersetfiles: + Python33Test.zip

messages: + msg184083
2013-03-13 14:13:49ned.deilysetnosy: + ned.deily
messages: + msg184082
2013-03-13 14:09:54theDarkBrainercreate