classification
Title: maildir.items() broken
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, flox, georg.brandl, marco.ghidinelli, petri.lehtinen, python-dev, r.david.murray
Priority: release blocker Keywords: buildbot

Created on 2011-10-24 09:14 by marco.ghidinelli, last changed 2011-11-08 11:21 by petri.lehtinen. This issue is now closed.

Files
File name Uploaded Description Edit
issue13254.patch petri.lehtinen, 2011-11-01 20:04 review
issue13254_v3.diff flox, 2011-11-01 21:44 review
Messages (16)
msg146283 - (view) Author: marco ghidinelli (marco.ghidinelli) Date: 2011-10-24 09:14
since python 2.7.2 
maildir.items() doesn't return anything:

>>> from mailbox import Maildir
>>> x = Maildir('test')
>>> x.items()
[]

untils something was written on the maildir.

>>> x.add('bubu')
'1319447613.M259111P6097Q1.deeppurple'
>>> x.items()
[('1319447013.M913187P6096Q1.deeppurple', <rfc822.Message instance at 0x7f25f5b612d8>), ('1319447613.M259111P6097Q1.deeppurple', <rfc822.Message instance at 0x7f25f5b61680>)]
msg146286 - (view) Author: marco ghidinelli (marco.ghidinelli) Date: 2011-10-24 09:48
i verified now, and the 2.7.1 version is ok, the bug is present since the 2.7.2
msg146287 - (view) Author: marco ghidinelli (marco.ghidinelli) Date: 2011-10-24 10:19
3.1 is ok
3.2.2 is affected
msg146301 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-10-24 13:16
I'm assigning this to myself so I don't lose it, but if someone wants to work on a patch please do.  It'll be a bit before I can look at it.
msg146302 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-10-24 13:18
Since this is a regression I'm upping the priority further to release blocker, so that we don't put out another release with the regression still in it.
msg146801 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-11-01 19:29
More specifically, this happens if the Maildir instance is created two seconds before items() is called:

>>> import time
>>> from mailbox import Maildir
>>> x = Maildir('test')  # has messages
>>> time.sleep(2.5)
>>> x.items()
[]

This happens because __init__() doesn't populate _toc, and mtimes haven't changed either, so _refresh() doesn't populate _toc either.
msg146803 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-11-01 20:04
Attached a patch. It makes _refresh() re-read the _toc uncoditionally when called the first time.
msg146809 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2011-11-01 21:33
The patch by Petri looks good.
I've uploaded a variant which set the initial mtimes to epoch instead of creating a _toc_read attribute. The tests are not changed.
msg146810 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2011-11-01 21:36
Re-uploaded, because we can set initialize _last_read to 0 instead of time.time(). It is more consistent.
msg146811 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2011-11-01 21:44
Finally, the patch can be only 5 lines...
msg146825 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-11-02 06:09
The last patch looks good to me and is a clever approach indeed :)
msg147073 - (view) Author: Roundup Robot (python-dev) Date: 2011-11-05 07:53
New changeset 5f27a9f67a34 by Petri Lehtinen in branch '3.2':
Fix Maildir initialization so that maildir contents are read correctly.
http://hg.python.org/cpython/rev/5f27a9f67a34

New changeset 91a0f6879173 by Petri Lehtinen in branch 'default':
Fix Maildir initialization so that maildir contents are read correctly.
http://hg.python.org/cpython/rev/91a0f6879173

New changeset 0b754ee12dbd by Petri Lehtinen in branch '2.7':
Fix Maildir initialization so that maildir contents  are read correctly.
http://hg.python.org/cpython/rev/0b754ee12dbd
msg147074 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-11-05 07:55
In the absence of any complaints, I went on and committed the fix. Thanks for reporting the issue!
msg147257 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2011-11-07 21:24
test fails on x86 Windows7 2.7 buildbot
(ok on 3.2 and 3.3)


test_unix_mbox (test.test_mailbox.MaildirTestCase) ... D:\cygwin\home\db3l\buildarea\2.7.bolen-windows7\build\lib\sqlite3\test\dbapi.py:649: DeprecationWarning: buffer() not supported in 3.x
  b = sqlite.Binary(chr(0) + "'")
D:\cygwin\home\db3l\buildarea\2.7.bolen-windows7\build\lib\sqlite3\test\types.py:70: DeprecationWarning: buffer() not supported in 3.x
  val = buffer("Guglhupf")
D:\cygwin\home\db3l\buildarea\2.7.bolen-windows7\build\lib\sqlite3\test\types.py:234: DeprecationWarning: buffer() not supported in 3.x
  val = buffer("Guglhupf")
D:\cygwin\home\db3l\buildarea\2.7.bolen-windows7\build\lib\sqlite3\test\types.py:350: DeprecationWarning: buffer() not supported in 3.x
  result = self.con.execute('select ? as "x [bin]"', (buffer(zlib.compress(testdata)),)).fetchone()[0]
D:\cygwin\home\db3l\buildarea\2.7.bolen-windows7\build\lib\sqlite3\test\userfunctions.py:39: DeprecationWarning: buffer() not supported in 3.x
  return buffer("blob")
D:\cygwin\home\db3l\buildarea\2.7.bolen-windows7\build\lib\sqlite3\test\userfunctions.py:200: DeprecationWarning: buffer() not supported in 3.x
  self.assertEqual(val, buffer("blob"))
D:\cygwin\home\db3l\buildarea\2.7.bolen-windows7\build\lib\sqlite3\test\userfunctions.py:237: DeprecationWarning: buffer() not supported in 3.x
  cur.execute("select isblob(?)", (buffer("blob"),))
D:\cygwin\home\db3l\buildarea\2.7.bolen-windows7\build\lib\sqlite3\test\userfunctions.py:255: DeprecationWarning: buffer() not supported in 3.x
  ("foo", 5, 3.14, None, buffer("blob"),))
D:\cygwin\home\db3l\buildarea\2.7.bolen-windows7\build\lib\sqlite3\test\userfunctions.py:347: DeprecationWarning: buffer() not supported in 3.x
  cur.execute("select checkType('blob', ?)", (buffer("blob"),))
test test_mailbox failed -- Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\2.7.bolen-windows7\build\lib\test\test_mailbox.py", line 803, in test_reread
    self.assertFalse(refreshed())
AssertionError: True is not false

ok

======================================================================
FAIL: test_reread (test.test_mailbox.TestMaildir)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\2.7.bolen-windows7\build\lib\test\test_mailbox.py", line 803, in test_reread
    self.assertFalse(refreshed())
AssertionError: True is not false

----------------------------------------------------------------------
Ran 280 tests in 5.250s
msg147274 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2011-11-08 07:17
This is a transient failure, other builds are successful.

Maybe the line "time.sleep(2.01 + self._box._skewfactor)" could be changed to "time.sleep(2.5 + self._box._skewfactor)" in test_mailbox.TestMaildir.test_reread.

Closing because it's not related to this issue.
msg147281 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-11-08 11:21
Even better would be to set skewfactor to something negative in this
test, too. I was told that long sleeps in tests are frowned upon.
History
Date User Action Args
2011-11-08 11:21:20petri.lehtinensetmessages: + msg147281
2011-11-08 07:17:13floxsetstatus: open -> closed

messages: + msg147274
2011-11-07 21:24:31floxsetstatus: closed -> open
keywords: + buildbot, - patch
messages: + msg147257
2011-11-05 07:55:13petri.lehtinensetassignee: r.david.murray ->
messages: + msg147074
2011-11-05 07:53:45python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg147073

resolution: fixed
stage: patch review -> resolved
2011-11-04 08:49:40floxsetkeywords: - needs review
2011-11-04 08:49:13floxsetfiles: - issue13254_v2.diff
2011-11-02 06:09:09petri.lehtinensetmessages: + msg146825
2011-11-01 21:44:21floxsetfiles: + issue13254_v3.diff

messages: + msg146811
2011-11-01 21:37:07floxsetfiles: - issue13254_v2.diff
2011-11-01 21:36:30floxsetfiles: + issue13254_v2.diff

messages: + msg146810
2011-11-01 21:33:15floxsetfiles: + issue13254_v2.diff
nosy: + flox
messages: + msg146809

2011-11-01 20:04:04petri.lehtinensetkeywords: + patch, needs review
files: + issue13254.patch
messages: + msg146803

stage: needs patch -> patch review
2011-11-01 19:29:19petri.lehtinensetmessages: + msg146801
2011-10-24 13:18:35r.david.murraysetpriority: high -> release blocker
nosy: + georg.brandl, benjamin.peterson
messages: + msg146302

2011-10-24 13:16:23r.david.murraysetpriority: normal -> high
assignee: r.david.murray
messages: + msg146301

stage: needs patch
2011-10-24 10:19:15marco.ghidinellisetmessages: + msg146287
2011-10-24 09:57:53petri.lehtinensetnosy: + petri.lehtinen
2011-10-24 09:48:23marco.ghidinellisetmessages: + msg146286
2011-10-24 09:38:20pitrousetnosy: + r.david.murray

versions: + Python 3.3
2011-10-24 09:14:58marco.ghidinellicreate