classification
Title: In str.format an incorrect error message for list, tuple, dict, set
Type: behavior Stage: patch review
Components: Documentation Versions: Python 3.8, Python 3.7, Python 3.6, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: cheryl.sabella, docs@python, eric.smith, ezio.melotti, py.user, r.david.murray, terry.reedy
Priority: normal Keywords: easy, patch

Created on 2012-01-15 05:31 by py.user, last changed 2018-02-26 15:37 by BreamoreBoy.

Files
File name Uploaded Description Edit
i13790.diff terry.reedy, 2012-01-20 23:44 review
i13790b.diff terry.reedy, 2012-01-22 02:16 review
Messages (29)
msg151277 - (view) Author: py.user (py.user) * Date: 2012-01-15 05:31
>>> '{0:d}'.format('a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'str'
>>> '{0:d}'.format(1+1j)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'complex'
>>> '{0:d}'.format([])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'str'
>>>

also strange behavior:
>>> '{0:s}'.format((1, 2, 3))
'(1, 2, 3)'
>>> '{0:10.5s}'.format([1, 2, 3])
'[1, 2     '
>>>
msg151279 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2012-01-15 05:57
I agree it's not the best error message. What's happening is that these types (list, tuple, etc.) do not implement __format__, so object.__format__ is used. It returns str(self). Then the resulting string is formatted with the given format_spec. Since str does not support the 'd' format type, the error you see is raised.

I'm open to suggestions on how to improve this, but I don't see how it's possible given what str.__format__ knows when it generates the error.
msg151300 - (view) Author: py.user (py.user) * Date: 2012-01-15 22:46
also strange(unobvious) behavior:
>>> '{0:.3s}'.format((i for i in (1, 2, 3)))
'<ge'
>>> '{0:.3s}'.format(range(10))
'ran'
>>> '{0:.3s}'.format(None)
'Non'
>>>

it would be better to print an error:
ValueError: Unknown format code 's' for object of type 'generator'

like in this:
>>> '{0:d}'.format(4.5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'float'
>>>

in the documentation there is nothing about it
msg151301 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-01-15 22:54
No, it wouldn't.  I expect

  "{}".format(x)

to produce something for an arbitrary x.  Breaking that would break a fundamental Python contract.

Improving the error message for 'd' is more possible.  Perhaps "the format code 'd' is not implemented by objects of type <type>"?
msg151302 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-01-15 23:00
Oh, and when you say there is nothing in the documentation about the 's' case for arbitrary objects, it is made clear in various places that every object has an str, which defaults to its repr if it has no specific __str__.  Combine that with the description of what happens when you use a fixed field length for 's', and you get the results you see.  There should be nothing surprising about this to anyone who has read the tutorial, I think.  (But specific suggestions for improving the docs are always welcome.)
msg151306 - (view) Author: py.user (py.user) * Date: 2012-01-15 23:26
R. David Murray wrote:
> it is made clear in various places that every object has an str

here:
http://docs.python.org/py3k/library/string.html#format-specification-mini-language

3rd paragraph:
"A general convention is that an empty format string ("") produces the same result as if you had called str() on the value. A non-empty format string typically modifies the result."

"an empty format string ("")" what does it mean ?

"".format(value) or "{}".format(value) or "{0}".format(value) ?
msg151307 - (view) Author: py.user (py.user) * Date: 2012-01-15 23:29
also here:
http://docs.python.org/py3k/library/string.html#format-examples

there is no example with list or tuple to know exactly how they are formatted
msg151308 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-01-15 23:49
"an empty format string" is exactly what I was talking about.  Putting nothing between the {}'s is an empty format string.  I can't think of any way to make that wording clearer.

The format docs should not contains examples of the repr of all possible python objects.  The examples of what tuples and lists and dicts &c look like are shown in the docs for those objects.
msg151316 - (view) Author: py.user (py.user) * Date: 2012-01-16 04:25
R. David Murray wrote:
> Putting nothing between the {}'s is an empty format string.

this is an empty replacement field

here:
http://docs.python.org/py3k/library/string.html#format-string-syntax

the definition of format string:
"Format strings contain “replacement fields” surrounded by curly braces {}. Anything that is not contained in braces is considered literal text, which is copied unchanged to the output."

"The grammar for a replacement field is as follows:"
replacement_field ::=  "{" [field_name] ["!" conversion] [":" format_spec] "}"
msg151355 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-01-16 12:53
Good point.  That should be fixed.  It should be "empty format specification".
msg151378 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2012-01-16 16:19
Changing to a documentation issue.
msg151709 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-01-20 23:44
Doc patch attached to make sure correct. Should {} be quoted?

Eric, do you want to close off the idea of changing :d errors, or switch back after the doc fix?
msg151711 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2012-01-21 00:16
I don't think "{}" is the correct way to document this. These all have an empty format specifier:

"{}".format(foo)
"{:}".format(foo)
"{0}".format(foo)
"{0:}".format(foo)
"{name}".format(name=foo)
format(foo, "")
format(foo)

That is, they all call foo.__format__(""). If foo.__format__ (well, really type(foo).__format__) doesn't exist, then object.__format__(foo, "") gets called. It's object.__format__ that's checking for the empty format string, and if so it returns str(foo).

What would you suggest changing the ':d' error message to, for objects that don't support a format type of 'd'? This makes sense to me:

>>> format('', 'd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'str'

The problem, if there is one, is:
>>> format([], 'd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'str'

The problem is that the str that's producing this error doesn't know that it exists because object.__format__ returned str([]).
msg151720 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-01-21 04:48
Oh, I see.  Yes, that is a problem.

object.__format__ knows the type of the object it was called on, right?  Couldn't it catch the error and re-raise it with the correct type?  (If the type isn't str, of course, we don't want to get into an infinite recursion.)
msg151722 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-01-21 04:50
Oh, never mind that comment about recursion, I wasn't thinking it through.
msg151725 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-01-21 05:56
OK, the example of an empty format spec should be dropped. Let people figure it out ;-).

>>> format([], 'd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'str'

One possibility is to give (str of) the object instead of the type:

ValueError: Unknown format code 'd' for object '[]'

The downside is a long message for long strings. It would need to be limited (as is done in test error reports).
msg151728 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2012-01-21 09:47
While looking at object.__format__, I recall that we've already addressed this, sort of. For a different reason, this is already deprecated in 3.3 and will become an error in 3.4. See issues 9856 and 7994.

$ ./python -Wd
Python 3.3.0a0 (default:40e1be1e0707, Jan 15 2012, 00:58:51) 
[GCC 4.6.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> format([], 'd')
__main__:1: DeprecationWarning: object.__format__ with a non-empty format string is deprecated
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'str'
[67288 refs]
>>> 

We could still have object.__format__ catch and re-throw the ValueError with a better message. I'd have to think it through if we could catch all ValueErrors, or if it's possible for another ValueError to be thrown and we'd only catch and rethrow this specific ValueError.

But since this is deprecated, I'm not sure it's worth the hassle. I'd advocate closing this issue as "won't fix".
msg151730 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-01-21 14:20
So the error is going to be something about the source type not supporting '__format__'?

That change will also address the OP's concern about truncated reprs when a fixed string length is specified, so I agree that the title issue can be closed.  Terry's patch with the ("{}") removed should be committed, though.
msg151738 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2012-01-21 17:35
The error message will be: "non-empty format string passed to object.__format__".

I agree with your comment about Terry's patch.
msg151757 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-01-22 02:16
Looking further, I noticed that 'string' needed to be changed to 'specification' in the following sentence also. Then I decided that the preceding sentence
 
"Most built-in types implement the following options for format specifications, although some of the formatting options are only supported by the numeric types."

should really follow the one about non-empty format specs. This positioning should make it more obvious that most of the options affect the string representation of the object after, not before, the string is produced, and are therefore applicable to all objects and not just string and number objects. I also propose to modify it so it is shorter and no longer contradictory, to read

"Most built-in types implement various options for such modifications, although some are only supported by the numeric types."

Further on, under "The available string presentation types are:"
I think "``'s'`` String format. This is the default type for strings and may be omitted." should have 'and other non-numeric types ' inserted after strings. New patch i13790b.diff attached

The point of these additional changes is to make it clearer that the default formatting of non-number, non-string objects is to call str() and then apply the options to the resulting string. That makes something like
>>> format(range(5), '-^20s') # same with object.__format__(), 3.3.0a0
'----range(0, 5)-----'
predictable and comprehensible.

I agree with not making a temporary change (but see below ;-).

But it seems that the 3.4 message should at least be
"numeric format string passed to object.__format__" or
"format string with number-only options passed to object.__format__" or
"object.__format__ cannot handle number-only options"
as string formats work fine and, I presume, are not deprecated (?).

However, if the new ValueError message did not specify object.__format__ (which could still be confusing, even if more accurate), the change could be make now. For instance
'Numeric option 'd' for non-number object'.
It would not really matter if it is later raised in object.__format__ instead of str.__format__. I believe *all* of the format codes 'unknown' to str (and by extension, by default, to all other non-number types) *are* number codes.
msg164343 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-06-29 18:17
>>> '%d' % ([],)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: %d format: a number is required, not list
msg164345 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2012-06-29 18:29
Serhiy: I'm not sure what you're saying. At the point that str.format() is producing its error message, it doesn't know as much as %-formatting does about the original arguments, so it can't produce a similar message.
msg164373 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-06-30 07:01
> Serhiy: I'm not sure what you're saying. At the point that str.format() is producing its error message, it doesn't know as much as %-formatting does about the original arguments, so it can't produce a similar message.

I'm surprised that the code of the classic and the modern formatting is
so different. Looking deeper, I saw that the issue will go away in 3.4.
I agree with you in msg151728.
msg220399 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-06-13 00:04
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:38:22) [MSC v.1600 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> '{0:d}'.format('a')
Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    '{0:d}'.format('a')
ValueError: Unknown format code 'd' for object of type 'str'

Nothing appears to have changed despite "the issue will go away in 3.4" in msg164373.  What should have happened here?
msg220401 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2014-06-13 00:32
I believe that comment was referring to the subject of this bug:

$ ./python
Python 3.4.1+ (3.4:bec6f18dd636, Jun 12 2014, 20:23:30)
[GCC 4.8.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> format([], 'd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: non-empty format string passed to object.__format__
>>> format((), 'd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: non-empty format string passed to object.__format__
>>> format({}, 'd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: non-empty format string passed to object.__format__
>>> format(set(), 'd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: non-empty format string passed to object.__format__

With the possible exception of listing the type in this error message, I think these are all fine.


I'm not sure what you'd expect format('a', 'd') to produce other than the error you're seeing. 'd' is in fact an unknown "format code" for str.

>>> format('a', 'd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'str'
msg220411 - (view) Author: py.user (py.user) * Date: 2014-06-13 01:21
Python 2.7.7 is still printing.

>>> format([], 'd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'd' for object of type 'str'
>>>
msg220412 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-06-13 01:24
Yes, the deprecation in 3.3 did not apply to 2.7.
msg312821 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2018-02-25 17:05
From the examples in msg220401, issue28385 changed it to print the object type in the message. 

>>> format([], 'd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported format string passed to list.__format__
>>> format((), 'd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported format string passed to tuple.__format__


Would the change left on this issue be to create a PR for Terry's documetation patch?

Thanks!
msg312841 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-02-25 19:14
In msg151730, R. David Murry said "Terry's [first] patch with the ("{}") removed should be committed, though."
In msg151738, Eric V. Smith said "I agree with your comment about Terry's patch."

My second patch removed "{}" but also made more text changes, explained in msg151757.  Someone should re-review
History
Date User Action Args
2018-02-26 15:37:03BreamoreBoysetnosy: - BreamoreBoy
2018-02-25 19:14:53terry.reedysetmessages: + msg312841
2018-02-25 17:05:08cheryl.sabellasetnosy: + cheryl.sabella

messages: + msg312821
versions: + Python 3.6, Python 3.7, Python 3.8, - Python 3.4, Python 3.5
2014-06-13 01:24:57terry.reedysetstage: test needed -> patch review
2014-06-13 01:24:04terry.reedysetmessages: + msg220412
versions: + Python 3.4, Python 3.5, - Python 3.2, Python 3.3
2014-06-13 01:21:30py.usersetmessages: + msg220411
2014-06-13 00:32:12eric.smithsetmessages: + msg220401
2014-06-13 00:04:39BreamoreBoysetnosy: + BreamoreBoy
messages: + msg220399
2013-01-07 16:10:40serhiy.storchakasetnosy: - serhiy.storchaka
2012-06-30 07:01:09serhiy.storchakasetmessages: + msg164373
2012-06-29 18:29:26eric.smithsetmessages: + msg164345
2012-06-29 18:17:31serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg164343
2012-05-04 18:29:10ezio.melottisetnosy: + ezio.melotti
2012-05-04 18:25:14eric.smithlinkissue14723 superseder
2012-01-22 02:16:17terry.reedysetfiles: + i13790b.diff
assignee: docs@python -> terry.reedy
messages: + msg151757
2012-01-21 17:35:19eric.smithsetmessages: + msg151738
2012-01-21 14:20:11r.david.murraysetmessages: + msg151730
2012-01-21 09:47:18eric.smithsetmessages: + msg151728
2012-01-21 05:56:24terry.reedysetmessages: + msg151725
2012-01-21 04:50:15r.david.murraysetmessages: + msg151722
2012-01-21 04:48:37r.david.murraysetmessages: + msg151720
2012-01-21 00:16:21eric.smithsetmessages: + msg151711
2012-01-20 23:44:42terry.reedysetfiles: + i13790.diff

nosy: + terry.reedy
messages: + msg151709

keywords: + patch
stage: test needed
2012-01-16 16:19:26eric.smithsetassignee: eric.smith -> docs@python
components: + Documentation, - Interpreter Core
versions: + Python 2.7
keywords: + easy
nosy: + docs@python

messages: + msg151378
2012-01-16 12:53:04r.david.murraysetmessages: + msg151355
2012-01-16 04:25:47py.usersetmessages: + msg151316
2012-01-15 23:49:48r.david.murraysetmessages: + msg151308
2012-01-15 23:29:52py.usersetmessages: + msg151307
2012-01-15 23:26:52py.usersetmessages: + msg151306
2012-01-15 23:00:49r.david.murraysetmessages: + msg151302
2012-01-15 22:54:15r.david.murraysetnosy: + r.david.murray
messages: + msg151301
2012-01-15 22:46:03py.usersetmessages: + msg151300
2012-01-15 05:57:34eric.smithsetversions: + Python 3.2, Python 3.3, - Python 3.1
nosy: + eric.smith

messages: + msg151279

assignee: eric.smith
2012-01-15 05:31:18py.usercreate