classification
Title: second python execution fails when embedding
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.5, Python 3.4, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, Denny Weinberg, amaury.forgeotdarc, berker.peksag, isoschiz, ncoghlan, ned.deily, palm.kevin, pconnell, pitrou, python-dev, r.david.murray, theDarkBrainer
Priority: normal Keywords:

Created on 2013-03-13 14:09 by theDarkBrainer, last changed 2016-06-27 13:50 by Denny Weinberg. This issue is now closed.

Files
File name Uploaded Description Edit
Python33Test.zip theDarkBrainer, 2013-03-13 14:18
Messages (15)
msg184081 - (view) Author: Vlad (theDarkBrainer) Date: 2013-03-13 14:09
This issue is for Python3.3 and doesn't exist in Python3.2

Detailed description with source code can be found here:
http://stackoverflow.com/questions/15387035/second-python-execution-fails
msg184082 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-03-13 14:13
Please add the detailed description of the problem and any test files to the issue here.  Information stored off-site is not searchable within the issue tracker and may not be permanently available.
msg184083 - (view) Author: Vlad (theDarkBrainer) Date: 2013-03-13 14:18
I'm trying to embed the python 3.3 engine for an app that need to run custom scripts in python. Since the scripts might be completely different, and sometimes user provided, I am trying to make each execution isolated and there is not need to preserve any data between execution of the different scripts.

So, my solution is to wrap each execution between 'Py_Initialize' and 'Py_Finalize'. It looks something like that:

  void ExecuteScript(const char* script)
  {
    Py_Initialize();
  
    PyRun_SimpleString( script );
  
    Py_Finalize();
  }

However, this fails for a particular python script the second time a script is executed with:

  done!
  Traceback (most recent call last):
    File "<string>", line 8, in <module>
    File "\Python33Test\Output\Debug\Python33\Lib\copy.py", line 89, in copy
      rv = reductor(2)
  TypeError: attribute of type 'NoneType' is not callable


The python script looks like this:

  class Data:
      value1 = 'hello'
      value2 = 0
  
  import copy
  
  d = Data()
  dd = copy.copy( d )
  print ( 'done!' )

As you can see, the first time around the script was executed the 'done!' was printed out. But the second time it rises an exception inside the copy function.

It looks like the python engine was left in some weird state after the first initialize-finalize. Note, this is python 3.3.

Also, it is very interesting to note that Python 2.7 and Python 3.2 did not have this problem.

I guess there might be other examples that could reveal better what's going, but i haven't had the time to find yet.

I also put a copy of the project containing flag to switch between Python 3 and Python 2.7 (the file is 31 MB): https://docs.google.com/file/d/0B86-G0mwwxZvbWRldTd5b2NNMWM/edit?usp=sharing

Thanks, Vlad
msg184085 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-03-13 15:15
Reproduced on Linux.
The reason is in Objects/typeobject.c: import_copyreg() has a static cache of the copyreg module.
When the interpreter stops, the module is filled with None... but gets reused in the next instance.

Resetting this "mod_copyreg" variable to NULL fixes the issue.
pickle is also broken for the same reason.
msg184261 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-03-15 21:25
Can you tell if the relevant 3.2 to 3.3 change is in Py_Initialize, Py_Finalize(), or typeobject.c?
msg184299 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-03-16 08:17
In 3.2, typeobject.c did not cache the copyreg module in import_copyreg(); PyImport_Import was always called.
msg188393 - (view) Author: Roundup Robot (python-dev) Date: 2013-05-04 18:46
New changeset 8c1385205a35 by Antoine Pitrou in branch '3.3':
Issue #17408: Avoid using an obsolete instance of the copyreg module when the interpreter is shutdown and then started again.
http://hg.python.org/cpython/rev/8c1385205a35

New changeset 0b34fd75b937 by Antoine Pitrou in branch 'default':
Issue #17408: Avoid using an obsolete instance of the copyreg module when the interpreter is shutdown and then started again.
http://hg.python.org/cpython/rev/0b34fd75b937
msg188395 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-05-04 18:47
Thank you for reporting! This should be fixed now.
msg188427 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2013-05-05 04:57
Commit 8c1385205a35 causes segmentation fault (in non-debug build) or abort (in debug build) during interpreter shutdown after copying of e.g. bytes or object().

$ python -c 'import copy; copy.copy(b""); print("text")'
text
Segmentation fault
$ python -c 'import copy; copy.copy(object()); print("text")'
text
Segmentation fault
msg188428 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-05-05 06:22
Should be fixed in 7de9852cdc0e, sorry.
msg265851 - (view) Author: Denny Weinberg (Denny Weinberg) Date: 2016-05-19 08:32
Hi,

I think that the problem exists also in python 3.5.1

After calling Py_Finalize and Py_Initialize I get the message "attribute of type 'NoneType' is not callable" on the datetime.strptime method.

Example:
from datetime import datetime
s = '20160505 160000'
refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')

The first call works find but it crashes after the re initialization.

Workaround:
from datetime import datetime
s = '20160505 160000'
try:
    refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')
except TypeError:
    import time
    refdatim = datetime.fromtimestamp(time.mktime(time.strptime(s, '%Y%m%d %H%M%S')))

Can anyone confirm this bug? Can you tell me if this will be fixed for python 3.5/3.6/...?

All the other modules are working find after the re initialization but datetime.strptime.
msg269374 - (view) Author: Denny Weinberg (Denny Weinberg) Date: 2016-06-27 13:02
Can we please reopen this issue?
msg269377 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-06-27 13:37
We don't re-open old issues (this was closed more than 3 years ago). Please open a new issue and provide a minimal reproducer. Thanks!
msg269378 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-06-27 13:41
The interpreter is not crashing in your case, so this is a mostly-unrelated problem.  (The part that is related is that it is triggered by module finalization.)  As Berker said, please open a new issue, but be warned that it may get closed as a being addressed by the issue for rewriting the python startup sequence (pep 432 I think, which I think Nick has resurrected), since this is a systemic problem.  On the other hand there might be a way to fix it in the datetime module if someone decides it is worth doing.
msg269380 - (view) Author: Denny Weinberg (Denny Weinberg) Date: 2016-06-27 13:50
Ok,

thank you very much for your comments.

See Issue27400
History
Date User Action Args
2016-06-27 13:50:43Denny Weinbergsetmessages: + msg269380
2016-06-27 13:41:20r.david.murraysetnosy: + r.david.murray
messages: + msg269378
2016-06-27 13:37:22berker.peksagsetnosy: + berker.peksag
messages: + msg269377
2016-06-27 13:02:00Denny Weinbergsetmessages: + msg269374
2016-05-19 16:24:16terry.reedysetnosy: - terry.reedy
2016-05-19 08:32:43Denny Weinbergsetnosy: + Denny Weinberg

messages: + msg265851
versions: + Python 3.5
2016-05-19 08:14:10palm.kevinsetnosy: + palm.kevin
2013-05-05 06:22:49pitrousetstatus: open -> closed
resolution: fixed
messages: + msg188428

stage: resolved
2013-05-05 04:57:25Arfreversetstatus: closed -> open

nosy: + Arfrever
messages: + msg188427

resolution: fixed -> (no value)
stage: resolved -> (no value)
2013-05-04 18:47:28pitrousetstatus: open -> closed

components: + Interpreter Core, - None
versions: + Python 3.4
nosy: + pitrou

messages: + msg188395
resolution: fixed
stage: needs patch -> resolved
2013-05-04 18:46:34python-devsetnosy: + python-dev
messages: + msg188393
2013-04-20 10:55:57isoschizsetnosy: + pconnell, isoschiz
2013-03-19 07:37:16ezio.melottisetstage: needs patch
2013-03-16 08:17:57amaury.forgeotdarcsetmessages: + msg184299
2013-03-15 21:25:55terry.reedysetnosy: + terry.reedy, ncoghlan
messages: + msg184261
2013-03-13 15:15:18amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg184085
2013-03-13 14:18:37theDarkBrainersetfiles: + Python33Test.zip

messages: + msg184083
2013-03-13 14:13:49ned.deilysetnosy: + ned.deily
messages: + msg184082
2013-03-13 14:09:54theDarkBrainercreate