classification
Title: list(obj) can swallow KeyboardInterrupt
Type: Stage:
Components: Interpreter Core Versions: Python 3.0, Python 3.1, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: amaury.forgeotdarc, benjamin.peterson, georg.brandl, kmtracey, ncoghlan, rhettinger, stevea_zope
Priority: high Keywords: patch

Created on 2005-07-21 22:22 by stevea_zope, last changed 2009-02-02 21:54 by rhettinger. This issue is now closed.

Files
File name Uploaded Description Edit
lenhint.diff rhettinger, 2009-01-15 07:45 Patch and tests for Py2.7
Messages (15)
msg25840 - (view) Author: Steve Alexander (stevea_zope) Date: 2005-07-21 22:22
The example below shows that list(f) swallows the
KeyboardInterrupt.

It swallows any other exception too, such as
MemoryError or application-specific ConflictErrors.

I think the "get the length of the object" optimisation
should catch only AttributeError and TypeError.

>>> class F(object):
...     def __iter__(self):
...         yield 23
...     def __len__(self):
...         print "len called.  raising Keyboard
Interrupt."
...         raise KeyboardInterrupt
...
>>> f = F()
>>> list(f)
len called.  raising Keyboard Interrupt.
[23]
msg25841 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2005-08-21 11:10
Logged In: YES 
user_id=80475

Okay, fixed in several places.  This little fragment
occurred in a number of places including code listextend(),
map(), zip(), filter(),  and PySequence_Tuple().  

msg25842 - (view) Author: Steve Alexander (stevea_zope) Date: 2005-09-16 09:02
Logged In: YES 
user_id=492001

Interestingly, I just saw some code that was causing a
problem in the SchoolTool project.  It had a __len__ method
that used list(self), causing great slowness without this
fix, and a RuntimeError "maximum recursion depth exceeded"
with this fix.
msg69965 - (view) Author: Karen Tracey (kmtracey) Date: 2008-07-18 18:35
This behavior has reappeared in the 2.6 beta releases:

Python 2.6b2 (r26b2:65082, Jul 18 2008, 13:36:54) 
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class F(object):
...     def __iter__(self):
...         yield 23
...     def __len__(self):
...         print 'len called, raising KeyboardInterrupt'
...         raise KeyboardInterrupt
... 
>>> f = F()
>>> list(f)
len called, raising KeyboardInterrupt
[23]

Should a new bug be opened for this?  (I can't find one but I'm not too
experienced searchign Python's bug tracker.)
msg69973 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-07-18 18:51
Reopening. For reference, the revision in which Raymond supposedly fixed
this is r39305.
msg69978 - (view) Author: Karen Tracey (kmtracey) Date: 2008-07-18 18:58
Thanks for responding.

It had been fixed.  2.4/2.5 behave like so:

Python 2.5.1 (r251:54863, Mar  7 2008, 04:10:12) 
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class F(object):
...     def __iter__(self):
...         yield 23
...     def __len__(self):
...         print 'len called, raising KeyboardInterrupt'
...         raise KeyboardInterrupt
... 
>>> f = F()
>>> list(f)
len called, raising KeyboardInterrupt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 6, in __len__
KeyboardInterrupt
>>> 

It just seems to have regressed in 2.6.
msg69979 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-07-18 19:00
I'm sorry, my use of "supposedly" was wrong here.
msg70093 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-07-20 21:44
The problem is in _PyObject_LengthHint which calls len(o) and masks all
exceptions from it. Its comments says "This function never fails.
Accordingly, it will mask exceptions raised in either method."

Would it be better to at least not mask BaseExceptions?
msg70364 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-07-28 17:08
Raymond, can you comment?
msg70366 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2008-07-28 18:56
Georg's suggestion seems reasonable.  Alternatively, you can just catch 
AttributeError or TypeError.

Lowering the priority to normal and marking as unresolved.  At some 
point, it would be nice to review the whole code base to see if other 
calls to PyErr_Clear() are too aggressive.  In that review, perhaps a 
more clean and general solution with present itself.
msg79873 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-01-14 20:19
Reassigning to Raymond (as per the thread at
http://mail.python.org/pipermail/python-dev/2009-January/085030.html)
msg79891 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-01-15 07:45
Here's a patch that fixes-up length_hint and it's internal callers.

I think there are plenty of other places that also swallow exceptions
but will leave those for someone who wants to look at every instance of
PyErr_Clear() to see if it has been restricted to some small group of
exceptions.
msg79914 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-01-15 19:31
Anyone else want to pick this up from here?
msg79922 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-01-15 22:49
In _PyObject_LengthHint() code, at the line:
    if (ro == NULL && PyErr_Occurred()) {
It seems that PyErr_Occurred() is not useful and is always true when the 
previous PyObject_CallMethodObjArgs() returned NULL.

Otherwise the patch is fine to me.
msg81005 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-02-02 21:54
Thanks for the review.
Fixed in r69227.
History
Date User Action Args
2009-02-02 21:54:50rhettingersetstatus: open -> closed
resolution: fixed
messages: + msg81005
2009-01-27 10:48:12rhettingersetpriority: normal -> high
2009-01-27 05:18:58rhettingersetassignee: rhettinger
2009-01-15 22:49:04amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg79922
2009-01-15 19:31:09rhettingersetmessages: + msg79914
2009-01-15 07:45:05rhettingersetfiles: + lenhint.diff
assignee: rhettinger -> (no value)
messages: + msg79891
keywords: + patch
versions: + Python 3.0, Python 2.7
2009-01-14 20:19:51ncoghlansetassignee: georg.brandl -> rhettinger
messages: + msg79873
nosy: + ncoghlan
2008-07-28 18:56:07rhettingersetpriority: critical -> normal
assignee: rhettinger -> georg.brandl
messages: + msg70366
resolution: fixed -> (no value)
versions: + Python 2.6, Python 3.1
2008-07-28 17:08:17benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg70364
2008-07-20 21:44:40georg.brandlsetmessages: + msg70093
2008-07-18 19:00:01georg.brandlsetmessages: + msg69979
2008-07-18 18:58:14kmtraceysetmessages: + msg69978
2008-07-18 18:51:01georg.brandlsetstatus: closed -> open
nosy: + georg.brandl
messages: + msg69973
priority: normal -> critical
2008-07-18 18:35:22kmtraceysetnosy: + kmtracey
messages: + msg69965
2005-07-21 22:22:37stevea_zopecreate