classification
Title: TypeError with pickle in embedded python3.3 when starting multiple Interpreters.
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 3.4, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: alexandre.vassalotti Nosy List: Larry.Pete, alexandre.vassalotti, amaury.forgeotdarc, pitrou, python-dev
Priority: normal Keywords:

Created on 2013-09-25 14:11 by Larry.Pete, last changed 2013-11-30 10:35 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
fail.c Larry.Pete, 2013-09-25 14:11 Simplified example of the same failure
Messages (5)
msg198386 - (view) Author: Larry Pete (Larry.Pete) Date: 2013-09-25 14:11
Hexchat (fork of XChat IRC Client) switched with version 2.9.6 to Python 3.3 for their Python plugins.
Hexchat loads plugins in a similar fashion like the attached C file (not entirely sure I used the C-API right though). For every plugin a new interpreter is started and Py_EndInterpreter is used to unload the plugin.
When using pickle in such a python plugin, it raises the Exception (in the attached example):

Traceback (most recent call last):
  File "<string>", line 7, in <module>
TypeError: attribute of type 'NoneType' is not callable

when trying to pickle an object (using pickle.dump or pickle.dumps).
The Exception happens on the line where pickle.dumps is called, though pickle.dumps is not None.

It happens:
- with python3.3.2 (I also tested it with python3.4.0a2 since I happened to have it installed, same issue). No issue with python2.7. I did not test it with 3.x versions prior to 3.3.2
- when trying to pickle a user defined class. Python objects like dictionaries and lists work fine.
- only with the second (and any additional) interpreter.
- when destroying the interpreter and starting a new one, in that order. When two interpreters are started and execute the code before any of them is destroyed, it works fine.

The full output of the attached file when executed is btw:

First output:
Pickle dumps:
b'\x80\x03c__main__\nSomeClass\nq\x00)\x81q\x01}q\x02X\x08\x00\x00\x00some_varq\x03K{sb.'

Second output:
Pickle dumps:
Traceback (most recent call last):
  File "<string>", line 7, in <module>
TypeError: attribute of type 'NoneType' is not callable
msg198417 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-09-25 20:53
The NoneType that fails is in typeobject.c:
   _PyObject_CallMethodId(copyreg, &PyId__slotnames, "O", cls);

The error here is that copyreg comes from a cached reference to the module, stored in a static variable (cached_copyreg_module).  When the interpreter shuts down the first time, all the module members are set to None... but the reference is kept forever.

This variable should be reset, and reloaded (with a fresh module from the new interpreter) the next time it's used.
I added a call to _PyType_Fini() in Py_EndInterpreter, this fixes the given example.

Two sets of questions though:

- There are many of these _Fini functions, called in Py_Finalize. Which ones should we call in Py_EndInterpreter? and why? Maybe this PyType_Fini() is not a _Fini, but should be renamed PyType_EndInterpreter?

- one copyreg for multiple interpreters... this looks wrong: each interpreter has its own list of modules, but copyreg.__globals__ belongs to only one...
  A good solution would be to cache the copyreg module in the PyInterpreterState (it already has codec_search_cache). Or use PyImport_GetModuleDict()
msg198420 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-09-25 21:00
> one copyreg for multiple interpreters... this looks wrong: each interpreter has its own list of modules, but copyreg.__globals__ belongs to only one...
> A good solution would be to cache the copyreg module in the PyInterpreterState (it already has codec_search_cache).

Yes, this is a reasonable solution.
msg204788 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-30 09:06
New changeset 96d1207d33d0 by Alexandre Vassalotti in branch '3.3':
Issue #19088: Fix incorrect caching of the copyreg module.
http://hg.python.org/cpython/rev/96d1207d33d0

New changeset 1ceb6f84b617 by Alexandre Vassalotti in branch 'default':
Issue #19088: Merge with 3.3.
http://hg.python.org/cpython/rev/1ceb6f84b617
msg204796 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-11-30 10:35
Thanks for the fix. Perhaps you could have added some tests for this?
History
Date User Action Args
2013-11-30 10:35:25pitrousetmessages: + msg204796
2013-11-30 09:11:05alexandre.vassalottisetstatus: open -> closed
assignee: alexandre.vassalotti
resolution: fixed
stage: needs patch -> resolved
2013-11-30 09:06:15python-devsetnosy: + python-dev
messages: + msg204788
2013-09-25 21:00:44pitrousetstage: needs patch
messages: + msg198420
versions: + Python 3.4
2013-09-25 20:53:57amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg198417
2013-09-25 14:11:14Larry.Petecreate