Title: mailbox (2.5b1): locking doesn't work (esp. on FreeBSD)
Type: behavior Stage: test needed
Components: Documentation, Library (Lib) Versions: Python 3.1, Python 3.2, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: baikie, nnorwitz
Priority: normal Keywords:

Created on 2006-06-25 15:38 by baikie, last changed 2014-12-31 16:21 by akuchling.

File name Uploaded Description Edit
mailbox-lock-via-methods.diff baikie, 2006-11-12 23:50 Use methods for internal locking operations
mailbox-doc-locking.diff baikie, 2006-11-12 23:52 Remove mention of flock() from docs
mailbox-lock-via-methods-52779.diff baikie, 2006-11-18 18:20 Methods patch against rev. 52779 review
Messages (10)
msg28902 - (view) Author: David Watson (baikie) Date: 2006-06-25 15:38
fcntl/flock() locking of mailboxes is actually disabled
due to a typo:

@@ -15,7 +15,7 @@
 import rfc822
 import StringIO
-    import fnctl
+    import fcntl
 except ImportError:
     fcntl = None

However, once enabled, it doesn't work on FreeBSD - the
lock() method always raises ExternalClashError (tested
on 5.3).  This is because flock(), lockf() and fcntl
locking share the same underlying mechanism on this
system (and probably others), and so the second lock
can never be acquired.
msg28903 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2006-06-25 19:52
Logged In: YES 

Andrew, are you helping maintain this module?  David is
correct that fcntl is mispelled in current svn.  Not sure
what to do about the locking issue.

David, would it be possible for you to work on a patch to
correct this problem?  I'm not sure if any developer has
access to a FreeBSD box similar to yours.  (There are a
couple that use FreeBSD though.)  It would be great to come
up with tests for this if the current ones don't stress this
msg28904 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2006-06-26 12:56
Logged In: YES 

Interesting.  The man page for flock() on my Linux system
says of flock() in a certain version:  "This  yields  true 
BSD  semantics: there is no interaction
between the types of lock placed by flock(2) and fcntl(2),
and flock(2) does not detect deadlock."  Apparently this is
out of date, and placing two locks is harmful.

I think it's best to just use one set of locking primitives,
and that one should be lockf(); the flock() calls should be
msg28905 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2006-06-26 13:27
Logged In: YES 

Typo fixed in rev. 47099; flock() calls removed in rev.
47100; a test for lock conflicts added in rev. 47101. 
David, please try out the current SVN head and let me know
if matters have improved.  I'll wait to see the buildbot
reports before closing this bug.

(Surprisingly, we don't seem to have a FreeBSD buildbot,
though we do have an OpenBSD one that didn't seem to have a
problem with the original version of the mailbox code.)

msg28906 - (view) Author: David Watson (baikie) Date: 2006-06-26 22:56
Logged In: YES 

Yeah, it works fine now (locks successfully when the mailbox
isn't locked by another process, and fails when it is
locked), although I was reminded that the Python install
process didn't work properly (I've submitted another bug

However, I should point out that flock() locking is required
when working with qmail on (for example) Linux - flock() is
the only locking mechanism it uses, and as noted below,
flock() on Linux is independent from fcntl/lockf().
msg28907 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2006-06-26 23:24
Logged In: YES 

Hm, this looks messy; I doubt the module can know if you want lockf() or 
fcntl() locking.  Maybe Mailbox.lock() should grow an optional parameter that 
lets you specify the locking mechanism to use -- lockf, fcntl, or both.  (But 
defaulting to what?)

On the other hand, maybe it's too late to go making API changes for beta2 
(but I suspect it would be OK in this case).

msg28908 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2006-06-27 03:21
Logged In: YES 

Is the optional parameter necessary if you use inheritance
(assuming there's a class/object in there)?  Maybe it would
be better to have a subclass?  Would that be better for us
to provide or just to add something to the docs that a user
could make a subclass to work around locking issues?

I haven't looked at the issues, but if we really need an API
change, I'm probably ok with it since it would seem to be
quite small.  If an API change is really required (doc isn't
sufficient), I'd like it to go in ASAP.

We could doc for 2.5 and if there's a problem fix for 2.6.
msg28909 - (view) Author: David Watson (baikie) Date: 2006-11-12 23:52
Logged In: YES 

I'd forgotten about this for a while, but I see the docs
didn't actually get changed for 2.5.  I was going to submit
a patch for them today and suggest using subclasses as
described below, but then I realized that subclassing
wouldn't work, since the code makes use of the module
functions _lock_file() and _unlock_file() outside of the
lock()/unlock() methods.

A possible solution is attached; the Mailbox class is given
methods _acquire_lock() and _release_lock() wrapping
_lock_file() and _unlock_file(), and these are used instead
(thus, they can be overridden as methods, but if someone had
replaced _lock_file() in the module namespace instead, then
their code would still work).

Also attached is a minimal doc patch to describe the new
(shorter) list of locking methods.  I've called the
system-call locking "fcntl()" since that's the usual term
when dealing with mailboxes, and the name fcntl.lockf() is
misleading (it calls fcntl() directly, not lockf(), and the
Linux man pages say that "In general, the relation between
lockf() and fcntl() is unspecified.").
msg114673 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-08-22 11:02
Is this still an issue on later versions of Python and/or FreeBSD?
msg115013 - (view) Author: David Watson (baikie) Date: 2010-08-26 18:03
> Is this still an issue on later versions of Python and/or FreeBSD?

Yes, there is still an issue.  There is no longer a deadlock on
FreeBSD because the module been changed to use only lockf() and
dot-locking (on all platforms), but the issue is now about how
users can enable other locking mechanisms that they need, such as
flock(), without causing a deadlock on platforms where they refer
to the same lock as lockf().

They can't just override the classes' .lock() and .unlock()
methods, because some parts of the code perform locking
operations directly without calling those methods.
Date User Action Args
2014-12-31 16:21:09akuchlingsetnosy: - akuchling
2014-02-03 19:40:16BreamoreBoysetnosy: - BreamoreBoy
2010-11-12 20:59:41akuchlingsetassignee: akuchling ->
2010-08-26 18:03:39baikiesetmessages: + msg115013
2010-08-22 11:02:31BreamoreBoysetnosy: + BreamoreBoy

messages: + msg114673
versions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6
2009-03-30 06:33:15ajaksu2setstage: test needed
type: behavior
components: + Documentation
versions: + Python 2.6, - Python 2.5
2006-06-25 15:38:46baikiecreate