classification
Title: queue.Queue() is not reentrant, so signals and GC can cause deadlocks
Type: crash Stage:
Components: Library (Lib) Versions: Python 3.3, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: JohanAR, itamarst, rhettinger, sbt, tim.peters
Priority: normal Keywords:

Created on 2012-06-01 08:43 by JohanAR, last changed 2014-02-24 14:55 by rhettinger.

Files
File name Uploaded Description Edit
queue_deadlock.py JohanAR, 2012-06-01 09:39 signal hander reading/writing a queue
queue_sendint.c JohanAR, 2012-06-01 09:40 send 2 signals to specified process in rapid succession
Messages (5)
msg162062 - (view) Author: Johan Aires Rastén (JohanAR) Date: 2012-06-01 08:43
Python 2.7.3 (default, Apr 20 2012, 22:39:59) 
[GCC 4.6.3] on linux2

If a signal handler calls Queue.PriorityQueue.put and a second signal is received while one is already being processed, neither of the calls to put will terminate.

Highly dependent on timing so it might be difficult to reproduce.
msg162065 - (view) Author: Johan Aires Rastén (JohanAR) Date: 2012-06-01 09:44
Start queue_deadlock.py in one terminal and note the PID it prints.

Compile queue_sendint.c in another terminal and execute it with previous PID as only argument.

If the bug is triggered, nothing will print in the python terminal window when you press Enter. To terminate the application you can press Ctrl-\
msg162069 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-06-01 12:02
I don't think there is anything special about PriorityQueue.

There is a similar concerning the use of the Python implementation of RLock in signal handlers -- see http://bugs.python.org/issue13697.

Maybe the signal handler should temporarily mask or ignore SIGINT while it runs.
msg162076 - (view) Author: Johan Aires Rastén (JohanAR) Date: 2012-06-01 14:43
I did read some more on the subject and it seems like using locks with interrupts is in general a very difficult problem and not limited to Python.

Don't know if this is considered common knowledge among programmers or if it would be useful with at least a warning on the signal manual page.
msg211436 - (view) Author: Itamar Turner-Trauring (itamarst) Date: 2014-02-17 18:50
This is not specifically a signal issue; it can happen with garbage collection as well if you have a Queue.put that runs in __del__ or a weakref callback function.

This can happen in real code. In my case, a thread that reads log messages from a Queue and writes them to disk. The thread putting log messages into the Queue can deadlock if GC happens to cause a log message to be written right after Queue.put() acquired the lock.
(see https://github.com/itamarst/crochet/issues/25).
History
Date User Action Args
2014-02-24 14:55:10rhettingersetnosy: + tim.peters, rhettinger
2014-02-17 18:50:43itamarstsetversions: + Python 3.3
nosy: + itamarst
title: Queue.PriorityQueue() is not interrupt safe -> queue.Queue() is not reentrant, so signals and GC can cause deadlocks
messages: + msg211436

2012-06-01 14:43:01JohanARsetmessages: + msg162076
2012-06-01 12:02:09sbtsetnosy: + sbt
messages: + msg162069
2012-06-01 09:44:19JohanARsetmessages: + msg162065
2012-06-01 09:40:02JohanARsetfiles: + queue_sendint.c
2012-06-01 09:39:02JohanARsetfiles: + queue_deadlock.py
2012-06-01 08:43:03JohanARcreate