classification
Title: Crash when importing an extension after Py_Initialize, Py_Finalize and Py_Initialize
Type: crash Stage:
Components: macOS Versions: Python 3.1
process
Status: closed Resolution: duplicate
Dependencies: Superseder: 9th import of module _pickle always crashes
View: 11321
Assigned To: belopolsky Nosy List: amaury.forgeotdarc, belopolsky, loewis, mpalluat, ncoghlan, ronaldoussoren
Priority: normal Keywords:

Created on 2010-07-08 08:50 by mpalluat, last changed 2011-02-28 01:15 by ncoghlan. This issue is now closed.

Files
File name Uploaded Description Edit
OSDP2.zip mpalluat, 2010-07-08 08:53 Mac project that highlights the crash
simpletest.zip amaury.forgeotdarc, 2011-01-31 18:55
Messages (12)
msg109521 - (view) Author: Palluat de Besset (mpalluat) Date: 2010-07-08 08:53
there is a crash log inside the archive
msg109524 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-07-08 09:18
The title is somewhat misleading, this is a C program that uses dlopen to load the python framework. 

I will look into this, but it is just as likely that code that loads or uses the python framework has a problem.
msg109525 - (view) Author: Palluat de Besset (mpalluat) Date: 2010-07-08 09:23
Hi Ronald,

Thank you for looking into it, and sorry for the misleading title.

You will find a crash log and some instructions on how to reproduce the problem inside the archive.

Thanks,

Marc

On 8 Jul 2010, at 10:18, Ronald Oussoren wrote:

Ronald Oussoren <ronaldoussoren@mac.com<mailto:ronaldoussoren@mac.com>> added the comment:

The title is somewhat misleading, this is a C program that uses dlopen to load the python framework.

I will look into this, but it is just as likely that code that loads or uses the python framework has a problem.

----------

_______________________________________
Python tracker <report@bugs.python.org<mailto:report@bugs.python.org>>
<http://bugs.python.org/issue9197>
_______________________________________

Marc Palluat de Besset
Mac Software Engineer

email : marc.palluatdebesset@sophos.com<mailto:marc.palluatdebesset@sophos.com>
tel : +44 1235 544056
loc : E2 floor 2

SOPHOS - simply secure

________________________________
Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 348 3873 20.
msg109537 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-07-08 11:41
I reproduce the problem on Linux (./configure --enable-shared), after I modified the source code a bit to directly use Python.h and to link with libpython3.2.so (no call to dlopen). In gdb the stack trace has exactly the same symbols as the attached crash log, so the issue is not specific to Mac frameworks.

The crash is inside _PyImport_LoadDynamicModule("_pickle"), when _PyImport_FixupExtension() calls Py_DECREF(def->m_base.m_copy), this certainly frees objects allocated in a previous incarnation of the interpreter, and segfaults in type_dealloc(): _PyObject_GC_UNTRACK(type).

In import.c::_PyImport_FixupExtension(), the Py_DECREF is preceded by a comment: """ /* Somebody already imported the module, 
   likely under a different name.
   XXX this should really not happen. */
Py_DECREF(def->m_base.m_copy);
"""
I removed this statement, and the program now runs correctly. I don't know about memory leaks though...
msg109539 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-07-08 13:22
Removing the call to Py_DECREF is fishy. 

The cleanup of the interpreter state in Py_Finalize doesn't clean up all state when m_copy refers to a version from a previous instance of the interpreter.

Maybe the tp_dealloc of a module object clean up m_copy? I don't know enough of this code and its intended behavior to know if that would be a valid change.
msg109540 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-07-08 13:29
We definitely need unit tests about embedded python interpreter, I think there are none.
msg109974 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-07-11 09:32
Unassigning, this is not mac-specific and I don't plan to work on this in the near future.
msg111452 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-07-24 10:52
(changing subject to better tell what this issue is about)
msg127634 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-31 18:10
Amaury,

Can you post your Linux code reproducing the issue?  I would like to take a look, but don't want to dig through an XCode project.
msg127642 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-01-31 18:55
Here are the files I used to reproduce the crash.
See "build.sh" for the build command, there may be better ways...
msg129669 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-02-27 23:34
Does the patch for issue11321 fix this issue?
msg129676 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-02-28 01:15
Merging this into issue11321 since that relates to the same underlying problem and includes a simpler test case.
History
Date User Action Args
2011-02-28 01:15:21ncoghlansetstatus: open -> closed

nosy: + ncoghlan
messages: + msg129676

superseder: 9th import of module _pickle always crashes
resolution: duplicate
2011-02-27 23:34:16amaury.forgeotdarcsetnosy: loewis, ronaldoussoren, amaury.forgeotdarc, belopolsky, mpalluat
messages: + msg129669
2011-01-31 18:55:13amaury.forgeotdarcsetfiles: + simpletest.zip
nosy: loewis, ronaldoussoren, amaury.forgeotdarc, belopolsky, mpalluat
messages: + msg127642
2011-01-31 18:10:43belopolskysettitle: Crash when importer an extension after Py_Initialize, Py_Finalize and Py_Initialize -> Crash when importing an extension after Py_Initialize, Py_Finalize and Py_Initialize
nosy: + belopolsky

messages: + msg127634

assignee: belopolsky
2011-01-31 18:04:23belopolskysetfiles: - unnamed
nosy: loewis, ronaldoussoren, amaury.forgeotdarc, mpalluat
2010-07-24 10:52:30ronaldoussorensetmessages: + msg111452
title: subprocess module causing crash -> Crash when importer an extension after Py_Initialize, Py_Finalize and Py_Initialize
2010-07-11 09:32:38ronaldoussorensetassignee: ronaldoussoren -> (no value)
messages: + msg109974
2010-07-08 13:29:11amaury.forgeotdarcsetmessages: + msg109540
2010-07-08 13:22:51ronaldoussorensetmessages: + msg109539
2010-07-08 11:41:21amaury.forgeotdarcsetnosy: + loewis, amaury.forgeotdarc
messages: + msg109537
2010-07-08 09:23:01mpalluatsetfiles: + unnamed

messages: + msg109525
2010-07-08 09:18:19ronaldoussorensetmessages: + msg109524
2010-07-08 08:53:56mpalluatsetfiles: + OSDP2.zip

messages: + msg109521
2010-07-08 08:50:50mpalluatcreate