classification
Title: simple bsddb interface potential for deadlock with threads
Type: behavior Stage:
Components: Extension Modules Versions: Python 2.6
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, ajaksu2, gregory.p.smith, jcea
Priority: low Keywords:

Created on 2003-11-02 09:30 by gregory.p.smith, last changed 2010-08-18 23:03 by BreamoreBoy. This issue is now closed.

Messages (6)
msg18861 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2003-11-02 09:30
From Lib/bsddb/__init__.py

        # FIXME-20031101-greg: I believe there is still the potential
        # for deadlocks in a multithreaded environment if someone
        # attempts to use the any of the cursor interfaces in one
        # thread while doing a put or delete in another thread.  The
        # reason is that _checkCursor and _closeCursors are not atomic
        # operations.  Doing our own locking around self.dbc,
        # self.saved_dbc_key and self._iter_cursors could prevent this.
        # TODO: A test case demonstrating the problem needs to be written.
msg18862 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2003-11-02 09:47
Logged In: YES 
user_id=413

While I'm at it, I think there might be a memory leak in my
__iter__ and iteritems() implementation when the resulting
generator objects are destroyed without completing their
iteration (as will happen in UserDict.DictMixIn.popitem
among other things).

They add their DBCursor to _DBWithCursor._iter_cursors but
only ever delete it from that hash before a return rather
than a yield.  The solution to this should be simple: have
_closeCursors() empty the _iter_cursors hash after calling
close() on all of the cursors.  __iter__ and iteritems()
already ignore a KeyError when trying to remove their cursor
from the map when returning.
msg18863 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2003-11-03 03:29
Logged In: YES 
user_id=413

A fix for my first comment about the memory leak has been committed that uses weak references and callbacks so that DBCursor objects are always destroyed and their references removed from _DBWithCursor's pool when the iterator that created them is destroyed.

The potential threading deadlock issues still stand.  Lockng seems like it could have a serious performance impact by surrounding way too many database operations (all cursor operations, iterator use and __delitem__, __setitem__ and close) with locks.

what should be done:

_DBWithCursor __delitem__, __setitem__ and close methods need to prevent any cursors from being created until they have completed.  But for performance reasons multiple database read operations should be allowed at the same time (get, cursor methods or iterators).
msg66980 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2008-05-17 07:01
This bug was created to track my FIXME in the code.  Nobody else has
commented on it.  I doubt its an issue worth fixing.  It is a valid
FIXME but I'm marking it as low priority and leaving it unassigned just
to track the issue incase someone actually does want to improve the
legacy/simple bsddb interface.
msg82022 - (view) Author: Daniel Diniz (ajaksu2) (Python triager) Date: 2009-02-14 12:09
Jesus, is this one still relevant or can it be closed?
msg114293 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-08-18 23:03
Closed as bsddb has been deprecated in 2.x and removed from py3k.
History
Date User Action Args
2010-08-18 23:03:04BreamoreBoysetstatus: open -> closed

nosy: + BreamoreBoy
messages: + msg114293

resolution: out of date
2009-02-14 12:09:36ajaksu2settype: behavior
messages: + msg82022
nosy: + ajaksu2
versions: + Python 2.6, - Python 2.5
2008-05-17 07:01:09gregory.p.smithsetpriority: normal -> low
assignee: gregory.p.smith ->
messages: + msg66980
2008-03-10 18:28:40jceasetnosy: + jcea
2003-11-02 09:30:13gregory.p.smithcreate