classification
Title: Inconsistent error types/messages for __len__ (and __nonzero__) between old and new-style classes
Type: behavior Stage:
Components: Interpreter Core Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: loewis, pboddie, r.david.murray, terry.reedy
Priority: normal Keywords:

Created on 2010-02-16 17:36 by pboddie, last changed 2010-08-04 03:56 by terry.reedy. This issue is now closed.

Messages (9)
msg99421 - (view) Author: Paul Boddie (pboddie) Date: 2010-02-16 17:36
As noted here:

http://www.selenic.com/pipermail/mercurial/2010-February/030068.html

This is probably documented somewhere, and there may even be a good reason for the difference, but old-style classes raise TypeError when __len__ returns a non-int, whereas new-style classes raise OverflowError. The latter is probably just as valid, but the message is a bit obscure for debugging purposes.

Maybe this went away after 2.5 - if so, sorry for the noise!

Here's an illustration of the problem:

Python 2.5.4 (r254:67916, Nov  4 2009, 17:59:46)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class C:
...     def __len__(self):
...             return 2**35
...
>>> c = C()
>>> len(c)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __len__() should return an int
>>> class C(object):
...     def __len__(self):
...             return 2**35
...
>>> c = C()
>>> len(c)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: long int too large to convert to int
msg99431 - (view) Author: Paul Boddie (pboddie) Date: 2010-02-16 18:13
The following is tangentially related:

http://bugs.python.org/issue2690
msg99432 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-02-16 18:16
I fail to see the bug in this report. What did you expect to happen instead?
msg99448 - (view) Author: Paul Boddie (pboddie) Date: 2010-02-16 23:17
I would have expected a more accurate error message for the new-style class. In the original message which brought this to my attention, the cause was not particularly obvious:

http://www.selenic.com/pipermail/mercurial/2010-February/030066.html

I concede that the different mechanisms in place for new-style classes might make it hard to have a specific error message in such a situation, though.
msg99453 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-02-17 00:50
In the example you give it seems to me that the message for the new style class is more accurate and useful than the message for the old style class.  Looking at your mercurial example, it looks like the problem is the same (len returning a long), and again it seems to me that the new message is more accurate than the old, and that neither one is more confusing than the other in the given context(s).  Or, if anything, that the OverflowError is less confusing as it has more connection with the problem (committing *large* amounts of data).

I don't see this as a bug in any sense, myself.  What would consider to be a "specific" error message?  Perhaps this is a request for an enhancement, instead.
msg99468 - (view) Author: Paul Boddie (pboddie) Date: 2010-02-17 12:16
I don't disagree that OverflowError describes what's happening, but the need to convert to an int in the first place is a detail of the machine - you'd have to know that this is a limitation of whatever internal "protocol" CPython implements - not a description of the cause of the error, which is what the old-style message describes quite clearly.

On the subject of whether __len__ should be able to return long integers, GvR seems to like the idea (from the related bug mentioned earlier):

http://bugs.python.org/issue2690#msg70525

I'll take a closer look at the mechanisms for error reporting around this situation later, but my assertion is that the new-style message isn't as helpful as the old-style one.
msg99469 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-02-17 12:58
Interestingly, this (len returning something larger than ssize_t) has not been fixed in python3.

On the other hand, I still think the new-style message is better.  Yes, it is an implementation detail of CPython, but that is exactly the error being reported.  Another implementation might successfully return the value.
msg99501 - (view) Author: Paul Boddie (pboddie) Date: 2010-02-18 12:55
Actually, in the issue reported, the initial problem occurs in the evaluation of an object in a boolean context (and the subsequent problem occurs with an explicit len invocation):

http://www.selenic.com/pipermail/mercurial/2010-February/030066.html

Presumably (from memory and a brief look at the reference), when "if data:" is evaluated, Python attempts to invoke an instance's __nonzero__ method or its __len__ method. Since the mercurial.httprepo.httpsendfile class only provides a __len__ method, the __len__ method's return value is used to determine truth.

The following demonstrates this particular issue:

>>> class C:
...     def __len__(self):
...             return 2**35
...
>>> c = C()
>>> if c: pass
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __nonzero__ should return an int
>>> class C(object):
...     def __len__(self):
...             return 2**35
...
>>> c = C()
>>> if c: pass
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: long int too large to convert to int

Here, I could actually argue that the message mentioning __nonzero__ is obscure: there isn't such a method defined, and __len__ is the misbehaving method. Still, in the context of boolean evaluation, the OverflowError is less helpful than it could be.
msg112749 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-08-04 03:56
The exceptions cannot change in 2.7.

For 3.1.2
class C:
    def __len__(self):
            return 2**35
c = C()
len(c)
# gives
OverflowError: cannot fit 'int' into an index-sized integer

Maybe (#2690) that will change.

So I do not see any valid bug or feature request issue.
History
Date User Action Args
2010-08-04 03:56:55terry.reedysetstatus: open -> closed
versions: + Python 2.7, - Python 2.5
nosy: + terry.reedy

messages: + msg112749

resolution: not a bug
2010-02-18 12:55:52pboddiesetmessages: + msg99501
title: Inconsistent error types/messages for __len__ between old and new-style classes -> Inconsistent error types/messages for __len__ (and __nonzero__) between old and new-style classes
2010-02-17 12:58:55r.david.murraysetmessages: + msg99469
2010-02-17 12:16:36pboddiesetmessages: + msg99468
2010-02-17 00:50:37r.david.murraysetpriority: normal
nosy: + r.david.murray
messages: + msg99453

2010-02-16 23:17:49pboddiesetmessages: + msg99448
2010-02-16 18:16:10loewissetnosy: + loewis
messages: + msg99432
2010-02-16 18:13:56pboddiesetmessages: + msg99431
2010-02-16 17:36:40pboddiecreate