classification
Title: PEP 461: Inconsistency between str and bytes formatting of integers
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: ethan.furman, python-dev, serhiy.storchaka, vstinner, wolma
Priority: normal Keywords: patch

Created on 2015-02-15 18:21 by serhiy.storchaka, last changed 2015-04-04 14:31 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
bytes_int_format.patch serhiy.storchaka, 2015-02-16 21:50 review
bytes_format_overflow.patch serhiy.storchaka, 2015-04-03 06:04 review
Messages (18)
msg236056 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-02-15 18:21
PEP 461 says that all numeric bytes formatting codes will work as they do for str. In particular b"%x" % val is equivalent to ("%x" % val).encode("ascii"). But this is wrong with current implementation:

>>> '%x' % 3.14
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: %x format: an integer is required, not float
>>> b'%x' % 3.14
b'3'

The same is for %X, %o and %c.

Raising TypeError on non-integer input to %c, %o, %x, and %X was added in issue19995.
msg236113 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-02-16 21:50
Here is a patch.

The difference between string and bytes formatting is that %c with out of the range integer raises OverflowError for str and TypeError for bytes. May be ValueError is more suitable in both cases.
msg239157 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-24 16:37
Ethan?
msg239191 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-24 22:22
The patch looks good to me, except of a question added on the review.
msg239553 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2015-03-30 01:51
Patch looks good.

Changing the raised exceptions to ValueError  would require deprecation periods.
msg239561 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-30 02:22
> Changing the raised exceptions to ValueError  would require deprecation
periods.

bytes%args is not a new feature of python 3.5? It sounds strange to
deprecate a part of a new feature.
msg239569 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-30 06:22
New changeset cb96fd376baa by Serhiy Storchaka in branch 'default':
Issue #23466: %c, %o, %x, and %X in bytes formatting now raise TypeError on
https://hg.python.org/cpython/rev/cb96fd376baa
msg239922 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2015-04-02 15:20
It's a new feature for 3.5 that is partly responsible for compatibility with 2.7 code.

2.7 raises Overflow error, so 3.5 should also (for out of range values -- wrong value types raise TypeError).
msg239933 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2015-04-02 17:30
b'%c' is still raising a TypeError.  The error message is fine ("%c requires an integer in range(256) or a single byte") but it should be an OverflowError for backwards compatibility.
msg239941 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-02 18:30
OverflowError is for platform limitations (such as the size of machine word or addressed space). When limits are well defined and platform-independent, ValueError or may be TypeError are considered as better types. It would be better to change OverflowError to ValueError or TypeError in formatting.
msg239955 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-03 06:04
Here is a patch that makes bytes formatting raise an OverflowError if integer argument of %c is out of range.
msg239958 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-04-03 08:48
> b'%c' is still raising a TypeError.  The error message is fine ("%c requires an integer in range(256) or a single byte") but it should be an OverflowError for backwards compatibility.

I don't understand why you care so much on the exact exception. It doesn't look right to me to rely on the *exact* exception raised by "%c" % arg. It's an obvious bug in the application.

Sometimes, you may want to be extra safe and catch exception while formating a message. The logging module does this. But the logging doesn't care of the exact exception, it uses a generic "except Except:" in StreamHandler.emit():

    def emit(self, record):
        try:
            msg = self.format(record)
            stream = self.stream
            stream.write(msg)
            stream.write(self.terminator)
            self.flush()
        except Exception:
            self.handleError(record)

IMO b"%c" % int must raise ValueError, not OverflowError, if the value is not in the range 0..255.
msg239962 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-03 09:10
See also issue18184.
msg240011 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2015-04-03 17:01
Looks good, thanks Serhiy.
msg240018 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-03 17:54
New changeset 313fd1c819c5 by Serhiy Storchaka in branch 'default':
Issue #23466: Raised OverflowError if %c argument is out of range.
https://hg.python.org/cpython/rev/313fd1c819c5
msg240070 - (view) Author: Wolfgang Maier (wolma) * Date: 2015-04-04 14:07
the new test:

test_exc('%x', '1', TypeError, "%x format: a number is required, not str")

expects the wrong error message.

python -m unittest -v test.test_format

...
'%x' % '1' works? ... no
Unexpected  <class 'TypeError'> : '%x format: an integer is required, not str'
...

-> it's "an integer", not "a number"
msg240077 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-04 14:30
New changeset 11e6986c794d by Serhiy Storchaka in branch 'default':
Issue #23466: Fixed expected error message in test_format.
https://hg.python.org/cpython/rev/11e6986c794d
msg240078 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-04 14:31
Good catch, Wolfgang!

Definitely we should make test_format more unittest compatible.
History
Date User Action Args
2015-04-04 14:31:51serhiy.storchakasetmessages: + msg240078
2015-04-04 14:30:14python-devsetmessages: + msg240077
2015-04-04 14:07:05wolmasetnosy: + wolma
messages: + msg240070
2015-04-03 17:56:55serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2015-04-03 17:54:11python-devsetmessages: + msg240018
2015-04-03 17:01:15ethan.furmansetmessages: + msg240011
stage: needs patch -> patch review
2015-04-03 09:10:56serhiy.storchakasetmessages: + msg239962
2015-04-03 08:48:40vstinnersetmessages: + msg239958
2015-04-03 06:04:12serhiy.storchakasetfiles: + bytes_format_overflow.patch

messages: + msg239955
2015-04-02 18:30:35serhiy.storchakasetmessages: + msg239941
2015-04-02 17:30:10ethan.furmansetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg239933

stage: resolved -> needs patch
2015-04-02 15:20:00ethan.furmansetmessages: + msg239922
2015-03-31 08:59:05serhiy.storchakasetstatus: open -> closed
assignee: serhiy.storchaka
resolution: fixed
stage: patch review -> resolved
2015-03-30 06:22:04python-devsetnosy: + python-dev
messages: + msg239569
2015-03-30 02:22:24vstinnersetmessages: + msg239561
2015-03-30 01:51:52ethan.furmansetmessages: + msg239553
2015-03-24 22:22:14vstinnersetnosy: + vstinner
messages: + msg239191
2015-03-24 16:37:23serhiy.storchakasetmessages: + msg239157
2015-02-16 21:50:29serhiy.storchakasetfiles: + bytes_int_format.patch
keywords: + patch
messages: + msg236113

stage: patch review
2015-02-15 20:01:30vstinnersettitle: Inconsistency between str and bytes formatting of integers -> PEP 461: Inconsistency between str and bytes formatting of integers
2015-02-15 18:21:46serhiy.storchakacreate