classification
Title: time.strptime not thread safe
Type: crash Stage: needs patch
Components: Library (Lib) Versions: Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Romuald, William.McBrine, amaury.forgeotdarc, belopolsky, brett.cannon, ccorbacho, ced, cptnwillard, davidfraser, dpalms2011, epu, eric.smith, flox, kzsolt, mark.dickinson, matrixise, pitrou, r.david.murray, ruseel
Priority: normal Keywords: patch

Created on 2010-02-22 10:46 by cptnwillard, last changed 2014-04-10 19:34 by pitrou.

Files
File name Uploaded Description Edit
issue7980-test.patch ruseel, 2010-10-05 12:16 unittest for py3k branch
issue7980-fix.patch Romuald, 2014-03-18 20:26 Fix attempt for 2.7 branch review
Messages (24)
msg99718 - (view) Author: Willard (cptnwillard) Date: 2010-02-22 10:46
The following script raises several "_strptime_time" AttributeErrors (on OS X 10.4 at least).

If time.strptime is used before starting the threads, then no exception is raised (the issue may thus come from strptime.py not being imported in a thread safe manner).


import time
import thread

def f():
    for m in xrange(1, 13):
        for d in xrange(1,29):
            time.strptime("2010%02d%02d"%(m,d),"%Y%m%d")

for _ in xrange(10):
    thread.start_new_thread(f, ())
time.sleep(3)


> Traceback (most recent call last):
>   File "[...]/test.py", line 75, in f
>     time.strptime("2010%02d%02d"%(m,d),"%Y%m%d")
> AttributeError: _strptime_time
msg99724 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2010-02-22 12:05
With 10.6's stock python, I've had this test either work or crash Python.

On trunk, I get it to either work or give the same error as the original report.

Unfortunately I've been unable to get it to crash again in a debugger so I can get a stack trace. But I'm still trying.
msg99725 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-02-22 12:19
Which version is "10.6's stock python"? Note that r59678 fixed a "problem with dead locks when mixin threads and imports".
msg99726 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2010-02-22 12:24
Oops, sorry for not specifying that. It's:
Python 2.6.4 (r264:75706, Jan 27 2010, 12:09:19) 
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
msg99749 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-02-22 15:34
Lib/_strptime.py itself should be thread-safe. I am guessing that it has something to do with the way the C code in time.c imports _strptime.py. A possible solution if it is the C code's import stuff is to create a time.py that imports a _time.c, but that's a total guess as to whether that will solve anything.
msg99754 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-02-22 15:49
It would be easier to move the PyImport_ImportModuleNoBlock("_strptime") currently in strptime() into the timemodule.c init function.
msg99768 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-02-22 16:09
FYI there's been a proposal to create a time.py module anyway in order to add some pure python functions not worth writing in c.
msg99803 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-02-22 18:29
For reference, it's issue 7989.
msg99805 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2010-02-22 18:31
I just tried it again under gdb on MacOS 10.6, the supplied python 2.6.4 and got this backtrace:

Program received signal SIGTRAP, Trace/breakpoint trap.
[Switching to process 61133]
0x00007fff87a4b790 in __CFInitialize ()
(gdb) bt
#0  0x00007fff87a4b790 in __CFInitialize ()
#1  0x00007fff5fc0d5ce in __dyld__ZN16ImageLoaderMachO11doImageInitERKN11ImageLoader11LinkContextE ()
#2  0x00007fff5fc0d607 in __dyld__ZN16ImageLoaderMachO16doInitializationERKN11ImageLoader11LinkContextE ()
#3  0x00007fff5fc0bcec in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#4  0x00007fff5fc0bc9d in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#5  0x00007fff5fc0bc9d in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#6  0x00007fff5fc0bda6 in __dyld__ZN11ImageLoader15runInitializersERKNS_11LinkContextE ()
#7  0x00007fff5fc08fbb in __dyld_dlopen ()
#8  0x00007fff8276fe00 in dlopen ()
#9  0x00000001000e876f in _PyImport_GetDynLoadFunc ()
#10 0x00000001000d2ce4 in _PyImport_LoadDynamicModule ()
#11 0x00000001000d0fbf in import_submodule ()
#12 0x00000001000d14da in load_next ()
#13 0x00000001000d17ec in PyImport_ImportModuleLevel ()
#14 0x00000001000aeeb3 in builtin___import__ ()
#15 0x000000010000c092 in PyObject_Call ()
#16 0x00000001000b00e7 in PyEval_CallObjectWithKeywords ()
#17 0x00000001000b43ae in PyEval_EvalFrameEx ()
#18 0x00000001000b8fd0 in PyEval_EvalCodeEx ()
#19 0x00000001000b90b6 in PyEval_EvalCode ()
#20 0x00000001000cdeb0 in PyImport_ExecCodeModuleEx ()
#21 0x00000001000cf112 in load_source_module ()
#22 0x00000001000d0fbf in import_submodule ()
#23 0x00000001000d14da in load_next ()
#24 0x00000001000d17ec in PyImport_ImportModuleLevel ()
#25 0x00000001000aeeb3 in builtin___import__ ()
#26 0x000000010000c092 in PyObject_Call ()
#27 0x00000001000b00e7 in PyEval_CallObjectWithKeywords ()
#28 0x00000001000b43ae in PyEval_EvalFrameEx ()
#29 0x00000001000b8fd0 in PyEval_EvalCodeEx ()
#30 0x00000001000b90b6 in PyEval_EvalCode ()
#31 0x00000001000cdeb0 in PyImport_ExecCodeModuleEx ()
#32 0x00000001000cf112 in load_source_module ()
#33 0x00000001000d0fbf in import_submodule ()
#34 0x00000001000d14da in load_next ()
#35 0x00000001000d17ec in PyImport_ImportModuleLevel ()
#36 0x00000001000aeeb3 in builtin___import__ ()
#37 0x000000010000c092 in PyObject_Call ()
#38 0x000000010000fbe5 in PyObject_CallFunction ()
#39 0x00000001000d1f9b in PyImport_Import ()
#40 0x00000001000d250f in PyImport_ImportModuleNoBlock ()
#41 0x0000000100515733 in time_strptime ()
#42 0x00000001000b8278 in PyEval_EvalFrameEx ()
#43 0x00000001000b8fd0 in PyEval_EvalCodeEx ()
#44 0x000000010003b355 in function_call ()
#45 0x000000010000c092 in PyObject_Call ()
#46 0x00000001000b00e7 in PyEval_CallObjectWithKeywords ()
#47 0x00000001000ef7ae in t_bootstrap ()
#48 0x00007fff827a2f8e in _pthread_start ()
#49 0x00007fff827a2e41 in thread_start ()
(gdb)
msg99807 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-02-22 18:36
Attribute error confirmed on linux on trunk and py3k.
msg118005 - (view) Author: MunSic JEONG (ruseel) Date: 2010-10-05 12:16
Attribute error confirmed on OSX, and py3k.

http://codereview.appspot.com/2371041
msg118021 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-10-05 17:10
MunSic,

It looks like issue7980.patch is just a unit test.  Do you have a patch to fix the issue?
msg118022 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-10-05 17:16
RDM> FYI there's been a proposal to create a time.py module anyway
RDM> in order to add some pure python functions not worth writing in c.
RDM> For reference, it's issue 7989.

Issue #7989 was retargeted to deal with datetime module.  The time.py issue is #9528.
msg118338 - (view) Author: MunSic JEONG (ruseel) Date: 2010-10-10 16:42
> Do you have a patch to fix the issue?
no, I don't. I just wanted to move stage from "unittest needed" to "needs patch" :)

After reading issue8098, now I realized that this is broad issue as belopolsky said in msg110095. 


For new reader of this issue after me, I would like to share my understanding.

Patch like below can solve this ticket. (changing "ImportModuleNoBlock" call to "ImportNoBlock")

 static PyObject *
 time_strptime(PyObject *self, PyObject *args)
 {
-    PyObject *strptime_module = PyImport_ImportModuleNoBlock("_strptime");
+    PyObject *strptime_module = PyImport_ImportModule("_strptime");


But "the function PyImport_ImportModuleNoBlock() was introduced and used in modules such as the time module to supposedly avoid deadlocks when using threads." 

So we could not apply that patch.
msg118530 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-10-13 15:26
It seems to me that this is related to if not a duplicate of issue 8098.
msg127872 - (view) Author: Carlos Corbacho (ccorbacho) Date: 2011-02-04 08:08
As per my comments on Issue11108 - I suspect that PyImport_ImportModuleNoBlock is a bit of a red herring here - in Python 2.5 and earlier versions (well before PyImport_ImportModuleNoBlock was added), we have occasionally seen 'AttributeError: strptime' from threaded code calling time.strptime(), so whatever this bug is, I don't believe it's caused by the call now being non-blocking from 2.6 onwards.
msg147110 - (view) Author: Zsolt Kendi (kzsolt) Date: 2011-11-05 20:59
I recognize the same problem problem at linux with 4 core 64 processor. The python version is 2.6.5 . Looks like occured randomly but maybe this is depend from thread sequence.
msg160270 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-09 11:12
For the record, issue9260 proposes to fix the issues with PyImport_ImportModuleNoBlock.
msg184067 - (view) Author: William McBrine (William.McBrine) Date: 2013-03-13 03:53
I'm still seeing this, in the 2.7.2 that comes with OS X 10.8.2.
msg206340 - (view) Author: David Palms (dpalms2011) Date: 2013-12-16 18:25
I am still seeing this in 2.7.5, has a patch been created yet?
msg206393 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-12-17 02:11
There's no patch posted here, and the issue is still open, so no.

On the other hand, the way imports are handled in 3.3 and later is a bit different, and if I run the unit test (which does still fail for me on 2.7 tip) on 3.3 and 3.4 tip it passes.

Given that, I'm guessing it is likely that none of the current Python committers are going to have time to look in to this, so unless someone else comes up with a patch, it isn't likely to get fixed in 2.7.
msg214013 - (view) Author: St├ęphane Wirtel (matrixise) * Date: 2014-03-18 20:16
This bug has not been fixed in 2.7.6 on OSX 10.9.2.
msg214014 - (view) Author: Romuald Brunet (Romuald) Date: 2014-03-18 20:26
This is a patch attempt to fix issue: import the _strptime module at time / datetime module load instead of method call.

Please note that the test is not fully reliable: it may pass without the patch / on current version, since threaded execution isn't controlled
msg215900 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-10 19:34
There are probably reasons why the import is currently done lazily, so I would advise against changing this in 2.7, unless the reasons are investigated and clearly understood.
History
Date User Action Args
2014-04-10 19:34:14pitrousetmessages: + msg215900
2014-04-10 17:35:51mark.dickinsonsetnosy: + mark.dickinson
2014-04-10 16:35:52epusetnosy: + epu
2014-03-18 20:26:27Romualdsetfiles: + issue7980-fix.patch
keywords: + patch
messages: + msg214014
2014-03-18 20:16:13matrixisesetnosy: + matrixise
messages: + msg214013
2014-03-18 14:00:32Romualdsetnosy: + Romuald
2013-12-17 02:11:42r.david.murraysetmessages: + msg206393
versions: - Python 2.6, Python 3.1, Python 3.2
2013-12-16 18:25:18dpalms2011setnosy: + dpalms2011
messages: + msg206340
2013-03-13 03:53:02William.McBrinesetnosy: + William.McBrine
messages: + msg184067
2012-05-09 11:12:32pitrousetnosy: + pitrou
messages: + msg160270
2012-04-19 09:07:31cedsetnosy: + ced
2012-01-19 07:23:09floxsetnosy: + flox
2011-11-05 20:59:29kzsoltsetnosy: + kzsolt
messages: + msg147110
2011-02-04 08:08:12ccorbachosetnosy: + ccorbacho
messages: + msg127872
2011-02-04 00:27:14amaury.forgeotdarclinkissue11108 superseder
2010-10-13 15:26:35belopolskysetkeywords: - patch

messages: + msg118530
2010-10-10 16:42:42ruseelsetmessages: + msg118338
2010-10-05 17:16:06belopolskysetmessages: + msg118022
2010-10-05 17:10:28belopolskysetmessages: + msg118021
2010-10-05 15:09:42pitrousetnosy: + belopolsky

stage: test needed -> needs patch
2010-10-05 12:16:20ruseelsetfiles: + issue7980-test.patch
keywords: + patch
messages: + msg118005
2010-09-28 12:33:20ruseelsetnosy: + ruseel
2010-07-21 17:55:35davidfrasersetnosy: + davidfraser
2010-02-22 18:37:13r.david.murraysetversions: + Python 3.1, Python 2.7, Python 3.2
2010-02-22 18:36:46r.david.murraysetpriority: normal

messages: + msg99807
stage: test needed
2010-02-22 18:31:36eric.smithsetmessages: + msg99805
2010-02-22 18:29:29r.david.murraysetmessages: + msg99803
2010-02-22 16:09:45r.david.murraysetnosy: + r.david.murray
messages: + msg99768
2010-02-22 15:49:13amaury.forgeotdarcsetmessages: + msg99754
2010-02-22 15:34:21brett.cannonsetnosy: + brett.cannon
messages: + msg99749
2010-02-22 12:24:21eric.smithsetmessages: + msg99726
2010-02-22 12:19:02amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg99725
2010-02-22 12:05:22eric.smithsetnosy: + eric.smith
messages: + msg99724
2010-02-22 10:46:36cptnwillardcreate