classification
Title: time.strptime not thread safe
Type: crash Stage: needs patch
Components: Library (Lib) Versions: Python 3.2, Python 3.1, Python 2.7, Python 2.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: William.McBrine, amaury.forgeotdarc, belopolsky, brett.cannon, ccorbacho, ced, cptnwillard, davidfraser, eric.smith, flox, kzsolt, pitrou, r.david.murray, ruseel
Priority: normal Keywords:

Created on 2010-02-22 10:46 by cptnwillard, last changed 2013-03-13 03:53 by William.McBrine.

Files
File name Uploaded Description Edit
issue7980-test.patch ruseel, 2010-10-05 12:16 unittest for py3k branch
Messages (19)
msg99718 - (view) Author: Willard (cptnwillard) Date: 2010-02-22 10:46
The following script raises several "_strptime_time" AttributeErrors (on OS X 10.4 at least).

If time.strptime is used before starting the threads, then no exception is raised (the issue may thus come from strptime.py not being imported in a thread safe manner).


import time
import thread

def f():
    for m in xrange(1, 13):
        for d in xrange(1,29):
            time.strptime("2010%02d%02d"%(m,d),"%Y%m%d")

for _ in xrange(10):
    thread.start_new_thread(f, ())
time.sleep(3)


> Traceback (most recent call last):
>   File "[...]/test.py", line 75, in f
>     time.strptime("2010%02d%02d"%(m,d),"%Y%m%d")
> AttributeError: _strptime_time
msg99724 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2010-02-22 12:05
With 10.6's stock python, I've had this test either work or crash Python.

On trunk, I get it to either work or give the same error as the original report.

Unfortunately I've been unable to get it to crash again in a debugger so I can get a stack trace. But I'm still trying.
msg99725 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-02-22 12:19
Which version is "10.6's stock python"? Note that r59678 fixed a "problem with dead locks when mixin threads and imports".
msg99726 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2010-02-22 12:24
Oops, sorry for not specifying that. It's:
Python 2.6.4 (r264:75706, Jan 27 2010, 12:09:19) 
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
msg99749 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-02-22 15:34
Lib/_strptime.py itself should be thread-safe. I am guessing that it has something to do with the way the C code in time.c imports _strptime.py. A possible solution if it is the C code's import stuff is to create a time.py that imports a _time.c, but that's a total guess as to whether that will solve anything.
msg99754 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-02-22 15:49
It would be easier to move the PyImport_ImportModuleNoBlock("_strptime") currently in strptime() into the timemodule.c init function.
msg99768 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-02-22 16:09
FYI there's been a proposal to create a time.py module anyway in order to add some pure python functions not worth writing in c.
msg99803 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-02-22 18:29
For reference, it's issue 7989.
msg99805 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2010-02-22 18:31
I just tried it again under gdb on MacOS 10.6, the supplied python 2.6.4 and got this backtrace:

Program received signal SIGTRAP, Trace/breakpoint trap.
[Switching to process 61133]
0x00007fff87a4b790 in __CFInitialize ()
(gdb) bt
#0  0x00007fff87a4b790 in __CFInitialize ()
#1  0x00007fff5fc0d5ce in __dyld__ZN16ImageLoaderMachO11doImageInitERKN11ImageLoader11LinkContextE ()
#2  0x00007fff5fc0d607 in __dyld__ZN16ImageLoaderMachO16doInitializationERKN11ImageLoader11LinkContextE ()
#3  0x00007fff5fc0bcec in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#4  0x00007fff5fc0bc9d in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#5  0x00007fff5fc0bc9d in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#6  0x00007fff5fc0bda6 in __dyld__ZN11ImageLoader15runInitializersERKNS_11LinkContextE ()
#7  0x00007fff5fc08fbb in __dyld_dlopen ()
#8  0x00007fff8276fe00 in dlopen ()
#9  0x00000001000e876f in _PyImport_GetDynLoadFunc ()
#10 0x00000001000d2ce4 in _PyImport_LoadDynamicModule ()
#11 0x00000001000d0fbf in import_submodule ()
#12 0x00000001000d14da in load_next ()
#13 0x00000001000d17ec in PyImport_ImportModuleLevel ()
#14 0x00000001000aeeb3 in builtin___import__ ()
#15 0x000000010000c092 in PyObject_Call ()
#16 0x00000001000b00e7 in PyEval_CallObjectWithKeywords ()
#17 0x00000001000b43ae in PyEval_EvalFrameEx ()
#18 0x00000001000b8fd0 in PyEval_EvalCodeEx ()
#19 0x00000001000b90b6 in PyEval_EvalCode ()
#20 0x00000001000cdeb0 in PyImport_ExecCodeModuleEx ()
#21 0x00000001000cf112 in load_source_module ()
#22 0x00000001000d0fbf in import_submodule ()
#23 0x00000001000d14da in load_next ()
#24 0x00000001000d17ec in PyImport_ImportModuleLevel ()
#25 0x00000001000aeeb3 in builtin___import__ ()
#26 0x000000010000c092 in PyObject_Call ()
#27 0x00000001000b00e7 in PyEval_CallObjectWithKeywords ()
#28 0x00000001000b43ae in PyEval_EvalFrameEx ()
#29 0x00000001000b8fd0 in PyEval_EvalCodeEx ()
#30 0x00000001000b90b6 in PyEval_EvalCode ()
#31 0x00000001000cdeb0 in PyImport_ExecCodeModuleEx ()
#32 0x00000001000cf112 in load_source_module ()
#33 0x00000001000d0fbf in import_submodule ()
#34 0x00000001000d14da in load_next ()
#35 0x00000001000d17ec in PyImport_ImportModuleLevel ()
#36 0x00000001000aeeb3 in builtin___import__ ()
#37 0x000000010000c092 in PyObject_Call ()
#38 0x000000010000fbe5 in PyObject_CallFunction ()
#39 0x00000001000d1f9b in PyImport_Import ()
#40 0x00000001000d250f in PyImport_ImportModuleNoBlock ()
#41 0x0000000100515733 in time_strptime ()
#42 0x00000001000b8278 in PyEval_EvalFrameEx ()
#43 0x00000001000b8fd0 in PyEval_EvalCodeEx ()
#44 0x000000010003b355 in function_call ()
#45 0x000000010000c092 in PyObject_Call ()
#46 0x00000001000b00e7 in PyEval_CallObjectWithKeywords ()
#47 0x00000001000ef7ae in t_bootstrap ()
#48 0x00007fff827a2f8e in _pthread_start ()
#49 0x00007fff827a2e41 in thread_start ()
(gdb)
msg99807 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-02-22 18:36
Attribute error confirmed on linux on trunk and py3k.
msg118005 - (view) Author: MunSic JEONG (ruseel) Date: 2010-10-05 12:16
Attribute error confirmed on OSX, and py3k.

http://codereview.appspot.com/2371041
msg118021 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-10-05 17:10
MunSic,

It looks like issue7980.patch is just a unit test.  Do you have a patch to fix the issue?
msg118022 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-10-05 17:16
RDM> FYI there's been a proposal to create a time.py module anyway
RDM> in order to add some pure python functions not worth writing in c.
RDM> For reference, it's issue 7989.

Issue #7989 was retargeted to deal with datetime module.  The time.py issue is #9528.
msg118338 - (view) Author: MunSic JEONG (ruseel) Date: 2010-10-10 16:42
> Do you have a patch to fix the issue?
no, I don't. I just wanted to move stage from "unittest needed" to "needs patch" :)

After reading issue8098, now I realized that this is broad issue as belopolsky said in msg110095. 


For new reader of this issue after me, I would like to share my understanding.

Patch like below can solve this ticket. (changing "ImportModuleNoBlock" call to "ImportNoBlock")

 static PyObject *
 time_strptime(PyObject *self, PyObject *args)
 {
-    PyObject *strptime_module = PyImport_ImportModuleNoBlock("_strptime");
+    PyObject *strptime_module = PyImport_ImportModule("_strptime");


But "the function PyImport_ImportModuleNoBlock() was introduced and used in modules such as the time module to supposedly avoid deadlocks when using threads." 

So we could not apply that patch.
msg118530 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-10-13 15:26
It seems to me that this is related to if not a duplicate of issue 8098.
msg127872 - (view) Author: Carlos Corbacho (ccorbacho) Date: 2011-02-04 08:08
As per my comments on Issue11108 - I suspect that PyImport_ImportModuleNoBlock is a bit of a red herring here - in Python 2.5 and earlier versions (well before PyImport_ImportModuleNoBlock was added), we have occasionally seen 'AttributeError: strptime' from threaded code calling time.strptime(), so whatever this bug is, I don't believe it's caused by the call now being non-blocking from 2.6 onwards.
msg147110 - (view) Author: Zsolt Kendi (kzsolt) Date: 2011-11-05 20:59
I recognize the same problem problem at linux with 4 core 64 processor. The python version is 2.6.5 . Looks like occured randomly but maybe this is depend from thread sequence.
msg160270 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-09 11:12
For the record, issue9260 proposes to fix the issues with PyImport_ImportModuleNoBlock.
msg184067 - (view) Author: William McBrine (William.McBrine) Date: 2013-03-13 03:53
I'm still seeing this, in the 2.7.2 that comes with OS X 10.8.2.
History
Date User Action Args
2013-03-13 03:53:02William.McBrinesetnosy: + William.McBrine
messages: + msg184067
2012-05-09 11:12:32pitrousetnosy: + pitrou
messages: + msg160270
2012-04-19 09:07:31cedsetnosy: + ced
2012-01-19 07:23:09floxsetnosy: + flox
2011-11-05 20:59:29kzsoltsetnosy: + kzsolt
messages: + msg147110
2011-02-04 08:08:12ccorbachosetnosy: + ccorbacho
messages: + msg127872
2011-02-04 00:27:14amaury.forgeotdarclinkissue11108 superseder
2010-10-13 15:26:35belopolskysetkeywords: - patch

messages: + msg118530
2010-10-10 16:42:42ruseelsetmessages: + msg118338
2010-10-05 17:16:06belopolskysetmessages: + msg118022
2010-10-05 17:10:28belopolskysetmessages: + msg118021
2010-10-05 15:09:42pitrousetnosy: + belopolsky

stage: test needed -> needs patch
2010-10-05 12:16:20ruseelsetfiles: + issue7980-test.patch
keywords: + patch
messages: + msg118005
2010-09-28 12:33:20ruseelsetnosy: + ruseel
2010-07-21 17:55:35davidfrasersetnosy: + davidfraser
2010-02-22 18:37:13r.david.murraysetversions: + Python 3.1, Python 2.7, Python 3.2
2010-02-22 18:36:46r.david.murraysetpriority: normal

messages: + msg99807
stage: test needed
2010-02-22 18:31:36eric.smithsetmessages: + msg99805
2010-02-22 18:29:29r.david.murraysetmessages: + msg99803
2010-02-22 16:09:45r.david.murraysetnosy: + r.david.murray
messages: + msg99768
2010-02-22 15:49:13amaury.forgeotdarcsetmessages: + msg99754
2010-02-22 15:34:21brett.cannonsetnosy: + brett.cannon
messages: + msg99749
2010-02-22 12:24:21eric.smithsetmessages: + msg99726
2010-02-22 12:19:02amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg99725
2010-02-22 12:05:22eric.smithsetnosy: + eric.smith
messages: + msg99724
2010-02-22 10:46:36cptnwillardcreate