classification
Title: Deadlock because of the import lock when loading the utf8 codec
Type: behavior Stage:
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, brett.cannon, henrietta, pitrou, vstinner
Priority: normal Keywords:

Created on 2011-01-17 02:17 by henrietta, last changed 2011-01-20 13:34 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
start.py henrietta, 2011-01-17 21:37 Script that reproduces the bug
Messages (12)
msg126390 - (view) Author: Piotr Maślanka (henrietta) Date: 2011-01-17 02:17
Python 2.7.1(x86 MSI), binary downloaded from python.org, hangs quite reliably. Code:

with open(threadspecific, 'ab') as x:
 txt = unicode(str_or_unicode_parameter).encode('utf8')
 x.write(txt+'\r\n')

However, it doesn't hang if I insert a print statement between with and txt, with anything. Previous testing determined that it hangs on encode().

Aforementioned code is executed in a threading environment, and it hangs in thread that is spawned by master thread. Interpreter is left with an open file.

Same behaviour is repeatable on Python 2.5.1.
msg126392 - (view) Author: Piotr Maślanka (henrietta) Date: 2011-01-17 03:07
I runned it over again with code:

print 'Acquiring lock'
self.loglock.acquire()
print 'Attempting to convert'
if type(text) == unicode: text = text.encode('utf8', errors='strict')
print 'Opening '+threadspecific
with open(threadspecific, 'ab') as x: x.write(text+'\r\n')
print 'Closing '+threadspecific
self.loglock.release()
print 'Releasing lock'

It behaves erratically, sometimes working and yielding:
Acquiring lock
Acquiring lock
Attempting to convert
Opening threadspecific_master
Closing threadspecific_master
Releasing lock
Attempting to convert
Opening threadspecific_slave
Closing threadspecific_slave
Releasing lock

And sometimes hanging with:

Acquiring lock
Attempting to convert
Acquiring lock

Looks like a particularly nasty race condition. It gives off no exceptions. Platform is Windows 7 x64, running with admin privileges.
msg126407 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-17 15:50
Please attach a simple script reproducing the perceived problem.
msg126438 - (view) Author: Piotr Maślanka (henrietta) Date: 2011-01-17 21:37
I ripped some stuff from the project I'm working on, undependencing it on any my libraries.

Can someone with similar conf(Win7 x86-64, Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on win32) confirm this?

Directory configuration when I run that script was that it was in a child directory and was invoked by a script with "import child_directory".

Before you say 'can't reproduce', run it at least 15 times, please.
msg126546 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-01-19 17:48
> "import child_directory"
Then it's certainly an effect of the "import lock":
http://docs.python.org/library/threading.html#importing-in-threaded-code

In your case, the first call to encode('utf8') indirectly imports utf8.py, while the import lock is held by 'import child_directory'.
Then the self.loglock is the second resource that closes the deadlock loop.

A workaround is to add some call to encode('utf8') before the thread is created.
msg126585 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-01-20 03:13
Dummy question: can't we raise a RuntimeError on a deadlock?
msg126599 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-01-20 12:13
> can't we raise a RuntimeError on a deadlock?

Deadlock detection is difficult, and probably impossible if the involved locks don't use the same underlying mechanism. (A lock can be a pthread object, a file opened with os.O_EXCL, and even a loop that tests some atomic variable)
msg126601 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-01-20 12:35
> > can't we raise a RuntimeError on a deadlock?

(I mean: deadlock on the import lock)

> Deadlock detection is difficult, and probably impossible if the
> involved locks don't use the same underlying mechanism

If it is impossible to detect deadlocks, can't we raise an exception if two threads try to import a module at the same time? (change completly how the import "lock" is handled)

Antoine changed recently the io module to raise a RuntimeError on reentrant calls in the io module (io.Buffered*.*()): #10478.
msg126603 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-20 13:07
> If it is impossible to detect deadlocks, can't we raise an exception
> if two threads try to import a module at the same time? (change
> completly how the import "lock" is handled)
> 
> Antoine changed recently the io module to raise a RuntimeError on
> reentrant calls in the io module (io.Buffered*.*()): #10478.

Reentrant calls and concurrent calls are not the same. If the import
lock is changed to raise an exception, heaps of multi-threaded software
will be broken.

What we could do is set a timeout on the import lock, but then we need
to choose a rather large one (e.g. 5 minutes), and who will wait 5
minutes before killing the process?
msg126609 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-01-20 13:30
> If the import lock is changed to raise an exception, 
> heaps of multi-threaded software will be broken.

You are right. It has done so for 12 years already (10011), so it's a bit late to do anything about it. And backward compatibility is very important, so I close this issue.
msg126610 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-01-20 13:31
> for 12 years already (10011)

Oops, it's r10011 (to get a nice URL on the commit).
msg126611 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-01-20 13:34
See also issue #9260 for a possible improvment.
History
Date User Action Args
2011-01-20 13:34:33vstinnersetnosy: brett.cannon, amaury.forgeotdarc, pitrou, vstinner, henrietta
messages: + msg126611
2011-01-20 13:31:51vstinnersetnosy: brett.cannon, amaury.forgeotdarc, pitrou, vstinner, henrietta
messages: + msg126610
2011-01-20 13:30:03vstinnersetstatus: open -> closed
nosy: brett.cannon, amaury.forgeotdarc, pitrou, vstinner, henrietta
messages: + msg126609
2011-01-20 13:07:40pitrousetnosy: brett.cannon, amaury.forgeotdarc, pitrou, vstinner, henrietta
messages: + msg126603
2011-01-20 12:35:04vstinnersetnosy: brett.cannon, amaury.forgeotdarc, pitrou, vstinner, henrietta
messages: + msg126601
2011-01-20 12:13:37amaury.forgeotdarcsetnosy: brett.cannon, amaury.forgeotdarc, pitrou, vstinner, henrietta
messages: + msg126599
2011-01-20 03:13:12vstinnersetnosy: brett.cannon, amaury.forgeotdarc, pitrou, vstinner, henrietta
messages: + msg126585
2011-01-20 03:07:28vstinnersetnosy: brett.cannon, amaury.forgeotdarc, pitrou, vstinner, henrietta
title: Python 2.7 hangs on Unicode+threading -> Deadlock because of the import lock when loading the utf8 codec
2011-01-19 23:29:06brett.cannonsetnosy: + brett.cannon
2011-01-19 17:48:42amaury.forgeotdarcsetresolution: works for me

messages: + msg126546
nosy: + amaury.forgeotdarc
2011-01-18 21:30:32vstinnersetnosy: + vstinner
2011-01-17 21:37:53henriettasetfiles: + start.py
nosy: pitrou, henrietta
messages: + msg126438
2011-01-17 15:50:42pitrousetnosy: + pitrou
messages: + msg126407
components: + Library (Lib), - None
2011-01-17 14:21:44brian.curtinsettype: crash -> behavior
2011-01-17 03:07:43henriettasettype: crash
messages: + msg126392
2011-01-17 02:17:28henriettacreate