classification
Title: Bytes objects should reject all formatting codes with an error message
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: berker.peksag, eric.smith, ned.deily, python-dev, r.david.murray, serhiy.storchaka, socketpair, vstinner
Priority: normal Keywords: easy (C), patch

Created on 2016-10-07 14:28 by socketpair, last changed 2016-11-03 11:36 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
issue28385.diff berker.peksag, 2016-10-08 12:08 review
object-__format__.patch serhiy.storchaka, 2016-10-08 14:06 review
object-__format__-2.patch serhiy.storchaka, 2016-10-18 07:42 review
Messages (20)
msg278244 - (view) Author: Марк Коренберг (socketpair) * Date: 2016-10-07 14:28
$ python3 -c "'{0:s}'.format(b'qwe')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: non-empty format string passed to object.__format__

Spent many hours to detect bug in my code.
msg278249 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-10-07 15:44
Bytes don't support formatting codes, so they are passed through to object, which doesn't support any formatting codes, and produces the error message you see.  Since all other built in types reject invalid codes with a message that mentions their type, I think bytes should too.  This would change the message to:

  ValueError: Unknown format code 's' for object of type 'bytes'

Would that have lessened your confusion sufficiently?
msg278250 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-10-07 15:56
Yes, I think we should implement this for object.__format__. Marking as easy.
msg278260 - (view) Author: Марк Коренберг (socketpair) * Date: 2016-10-07 18:54
Yes, that message will be sufficient. Perfectly.
msg278293 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-10-08 12:08
Here's a patch. Changes in test_builtin might be redundant, but I added them anyway.
msg278301 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-10-08 13:48
I left a comment on Rietveld, but repeating it here because for me those messages often go to spam:

I think you want to make this more general, by modifying the message for the TypeError that follows. Any type that doesn't provide it's own __format__ should produce an error when using a non-empty format string.

For example:

>>> format({}, 'a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: non-empty format string passed to object.__format__

Would be better as:
TypeError: non-empty format string passed to __format__ for object of type "class <'dict'>"

(Or some prettier way to print out the type).
msg278302 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-08 14:06
Sorry, but it looks to me that issue28385.diff solves wrong issue. The issue is not in bytes, but in default __format__. object.__format__ should raise an error message that contains the name of the actual type. There are other issues with object.__format__, following patch tries to solve them.

1. Error message now contains the name of the actual type.

2. Format spec is checked before converting the value to str. Converting bytes and bytearray to str can raise a BytesWarning, and it is better if the type of this error doesn't depend on the -b option.

3. Since format spec is always empty, avoid calling PyObject_Format(). In theory PyObject_Str() can raise a subclass of str with non-trivial __format__, but I think we can ignore this subtle behavior difference.
msg278831 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-10-18 00:30
LGTM, although I'm not so sure about your #3. Maybe it should be a separate issue and raised on python-dev? But I don't feel strongly enough about it to argue the point.

Thanks!
msg278845 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-18 07:42
Removed item 3 from updated patch.
msg278846 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-18 07:53
From msg214162:

> This is basically the definition of object.__format__:
>
> def __format__(self, specifier):
>   if len(specifier) == 0:
>     return str(self)
>   raise TypeError('non-empty format string passed to object.__format__')

This simple algorithm is implemented in object-__format__.patch.

But current implementation uses ``format(str(self), '')`` instead of just ``str(self)``. object-__format__-2.patch keeps this.
msg279736 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-10-30 17:38
New changeset 92cae79fa5d9 by Serhiy Storchaka in branch '3.5':
Issue #28385: An error message when non-empty format spec is passed to
https://hg.python.org/cpython/rev/92cae79fa5d9

New changeset 0a985f7c6731 by Serhiy Storchaka in branch '3.6':
Issue #28385: An error message when non-empty format spec is passed to
https://hg.python.org/cpython/rev/0a985f7c6731

New changeset 6e8183abcc35 by Serhiy Storchaka in branch 'default':
Issue #28385: An error message when non-empty format spec is passed to
https://hg.python.org/cpython/rev/6e8183abcc35
msg279739 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-10-30 18:54
Buildbots are failing:

======================================================================
FAIL: test_errors (test.test_fstring.TestCase) (str="f'{(lambda: 0):x}'")
----------------------------------------------------------------------
TypeError: unsupported format string passed to function.__format__

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/buildarea/3.6.angelico-debian-amd64/build/Lib/test/test_fstring.py", line 20, in assertAllRaise
    eval(str)
AssertionError: "non-empty" does not match "unsupported format string passed to function.__format__"

======================================================================
FAIL: test_errors (test.test_fstring.TestCase) (str="f'{(0,):x}'")
----------------------------------------------------------------------
TypeError: unsupported format string passed to tuple.__format__

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/buildarea/3.6.angelico-debian-amd64/build/Lib/test/test_fstring.py", line 20, in assertAllRaise
    eval(str)
AssertionError: "non-empty" does not match "unsupported format string passed to tuple.__format__"

----------------------------------------------------------------------
Ran 48 tests in 1.389s
msg279742 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-30 19:13
What is better?

1. Replace "unsupported format string passed to" back to "non-empty format string passed to " in generated error message? Note that full error message is changed in any case, this is the purpose of this issue.

2. Change the pattern "non-empty" to "unsupported" in f-string tests. Or to other part of new error message. It may be changed again in future.
msg279756 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-10-31 01:03
I would change the f-string tests. I realize it's a fragile test, but it's the only way to test the strings in the exception.
msg279762 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-10-31 06:13
New changeset f43078ec598c by Serhiy Storchaka in branch '3.6':
Update the f-string test broken in issue #28385.
https://hg.python.org/cpython/rev/f43078ec598c

New changeset 931410a04240 by Serhiy Storchaka in branch 'default':
Update the f-string test broken in issue #28385.
https://hg.python.org/cpython/rev/931410a04240
msg279763 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-31 06:19
Shouldn't the type of error be changed from TypeError to ValueError?
msg279786 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-10-31 12:47
TypeError is documented as "Raised when an operation or function is applied to an object of inappropriate type". I think that fits this case: you're applying some format code to a type that doesn't support it.
msg279787 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-31 12:56
But on other hand, the error depends on the value of format specifier:

>>> format('abc', '')
'abc'
>>> format('abc', 'j')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'j' for object of type 'str'
>>> format([1, 2, 3], '')
'[1, 2, 3]'
>>> format([1, 2, 3], 'j')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported format string passed to list.__format__

If keep TypeError I think it would be better to restore former error message "non-empty format string".
msg279844 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-11-01 00:29
I don't have a strong feeling one way or the other. I'd be surprised if anyone is catching these errors.
msg279986 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-11-03 11:36
The proposition of making default __format__ equivalent to str() will be addressed in separate issue. Python-Dev thread: https://mail.python.org/pipermail/python-dev/2016-October/146765.html.
History
Date User Action Args
2016-11-03 11:36:21serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg279986

stage: patch review -> resolved
2016-11-01 00:29:48eric.smithsetmessages: + msg279844
2016-10-31 12:56:51serhiy.storchakasetmessages: + msg279787
2016-10-31 12:47:06eric.smithsetmessages: + msg279786
2016-10-31 06:19:44serhiy.storchakasetmessages: + msg279763
2016-10-31 06:13:53python-devsetmessages: + msg279762
2016-10-31 01:03:27eric.smithsetmessages: + msg279756
2016-10-30 19:13:25serhiy.storchakasetmessages: + msg279742
2016-10-30 18:54:55ned.deilysetnosy: + ned.deily
messages: + msg279739
2016-10-30 17:38:46python-devsetnosy: + python-dev
messages: + msg279736
2016-10-30 17:26:16serhiy.storchakasetassignee: serhiy.storchaka
2016-10-18 07:53:46serhiy.storchakasetmessages: + msg278846
2016-10-18 07:44:54vstinnersetnosy: + vstinner
2016-10-18 07:42:39serhiy.storchakasetfiles: + object-__format__-2.patch

messages: + msg278845
2016-10-18 00:30:46eric.smithsetmessages: + msg278831
2016-10-08 14:06:52serhiy.storchakasetfiles: + object-__format__.patch
nosy: + serhiy.storchaka
messages: + msg278302

2016-10-08 13:48:52eric.smithsetmessages: + msg278301
2016-10-08 12:08:27berker.peksagsetfiles: + issue28385.diff

nosy: + berker.peksag
messages: + msg278293

keywords: + patch
stage: needs patch -> patch review
2016-10-07 18:54:33socketpairsetmessages: + msg278260
2016-10-07 15:56:19eric.smithsetkeywords: + easy (C)

messages: + msg278250
components: + Interpreter Core, - Library (Lib)
2016-10-07 15:44:20r.david.murraysetnosy: + eric.smith, r.david.murray
title: Non-informative exception while formatting strings. -> Bytes objects should reject all formatting codes with an error message
messages: + msg278249

type: behavior
stage: needs patch
2016-10-07 14:28:40socketpaircreate