classification
Title: builtin __format__ methods cannot fill with \x00 char
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.smith Nosy List: Gavin.Andresen, davide.rizzo, eric.smith, ezio.melotti, flox, haypo, python-dev
Priority: normal Keywords: patch

Created on 2011-07-13 06:58 by Gavin.Andresen, last changed 2014-05-19 15:15 by eric.smith. This issue is now closed.

Files
File name Uploaded Description Edit
format00.patch davide.rizzo, 2011-07-13 09:07 review
Messages (14)
msg140225 - (view) Author: Gavin Andresen (Gavin.Andresen) Date: 2011-07-13 06:58
This gives me "foo   " instead of expected "foo\x00\x00\x00" :

"{0:\x00<6}".format('foo')
msg140231 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2011-07-13 08:24
\x00 is used as a flag internally meaning "use the default fill character". That's clearly a bug. I'll look at fixing it.

I think there are other places in the built in __format__ functions where special values are used instead of flags. I'll review those as well.

Thanks for the report!
msg140233 - (view) Author: Davide Rizzo (davide.rizzo) Date: 2011-07-13 08:36
This patch removes the special meaning for \x00 and defines the default padding character (' ') in parse_internal_render_format_spec. Test included. Maybe the default padding character should be defined elsewhere?
msg140235 - (view) Author: Davide Rizzo (davide.rizzo) Date: 2011-07-13 08:42
Oops, sorry. Above patch was overly buggy. Please just ignore it.
msg140238 - (view) Author: Davide Rizzo (davide.rizzo) Date: 2011-07-13 09:07
Here's the patch. Same rationale as above (removed the special meaning of '\x00', default specified in parse_internal_render_format_spec). Sorry about the mess again!
msg140242 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2011-07-13 10:01
Patch looks good at first glance. I'll review it some more today and commit it. Thanks!
msg140654 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2011-07-19 10:17
I finally got around to reviewing the patch. A couple of comments:

1. There should be some tests for str.__format__, not just str.format. This is really a bug with str.__format__, after all. I can add those.

2. The bigger issue is that the other built in formatters have this same problem.

>>> format(3, '\x00<6')
'3     '
>>> format(3., '\x00<6')
'3.0   '
>>> format('3', '\x00<6')
'3\x00\x00\x00\x00\x00'
>>> format(3+1j, '\x00<6')
'(3+1j)'
[38654 refs]
>>> format(3+1j, '\x00<10')
'(3+1j)    '

I think the fix is basically the same as str.__format__ (but in format_int_or_long_internal, format_float_internal, and format_complex_internal). I tried that and it worked, but I haven't had time to write tests for them. If you (Davide) can do that, great. Otherwise I'll try and grab some time this week.

Changing the subject to match the wider scope of the problem.
msg215457 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-04-03 16:22
#17705 has been closed as a duplicate of this issue.
msg215793 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-04-09 01:26
The patch looks good to me.
msg216093 - (view) Author: Roundup Robot (python-dev) Date: 2014-04-14 15:25
New changeset 520ce42ba2b8 by Eric V. Smith in branch '2.7':
Issue #12546: Allow \x00 as a fill character for builtin type __format__ methods.
http://hg.python.org/cpython/rev/520ce42ba2b8
msg216105 - (view) Author: Roundup Robot (python-dev) Date: 2014-04-14 16:08
New changeset 7c484551bce1 by Eric V. Smith in branch '3.4':
Issue #12546: Allow \x00 as a fill character for builtin type __format__ methods.
http://hg.python.org/cpython/rev/7c484551bce1

New changeset bd90e68dc81f by Eric V. Smith in branch 'default':
Closes issue #12546: Allow \x00 as a fill character for builtin type __format__ methods.
http://hg.python.org/cpython/rev/bd90e68dc81f
msg216106 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2014-04-14 16:09
Fixed in 2.7, 3.4, 3.5.
msg218782 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-05-19 07:48
I don't understand why it works with "<", "=" or ">":

>>> "{0:\x00<6d}".format(123)
'123\x00\x00\x00'

But not without:

>>> "{0:\x006d}".format(123)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Invalid format specifier

Compare it to:

>>> "{0:6d}".format(123)
'   123'
>>> "{0:06d}".format(123)
'000123'
msg218797 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2014-05-19 15:15
For int, the spec is:
[[fill]align][sign][#][0][width][,][.precision][type]

So, for "06d", "0" is matched as the literal 0, "6" is matched as width, and "d" is matched as type.

For "\x00<6d", "\x00" is matched as fill, "<" as align, "6" as width, and "d" as type.

For "\x006d", there's no align. So "\x00" cannot match as fill. "\x00" doesn't match anything else, so it's an invalid format specifier, thus the exception.
History
Date User Action Args
2014-05-19 15:15:04eric.smithsetmessages: + msg218797
2014-05-19 08:13:20floxsetnosy: + flox
2014-05-19 07:48:07hayposetmessages: + msg218782
2014-04-14 16:09:44eric.smithsetstatus: open -> closed
resolution: fixed
messages: + msg216106

stage: patch review -> resolved
2014-04-14 16:08:28python-devsetmessages: + msg216105
2014-04-14 15:25:25python-devsetnosy: + python-dev
messages: + msg216093
2014-04-14 15:07:22eric.smithsettype: enhancement -> behavior
versions: + Python 2.7, Python 3.4
2014-04-09 01:26:30hayposettype: behavior -> enhancement
messages: + msg215793
versions: - Python 2.7, Python 3.4
2014-04-03 16:58:37eric.smithsetversions: + Python 3.4, Python 3.5, - Python 3.2, Python 3.3
2014-04-03 16:22:16hayposetmessages: + msg215457
2014-04-03 16:15:23skrahlinkissue17705 superseder
2011-07-19 13:53:01hayposetnosy: + haypo
2011-07-19 10:17:31eric.smithsettitle: str.format cannot fill with \x00 char -> builtin __format__ methods cannot fill with \x00 char
messages: + msg140654
stage: commit review -> patch review
2011-07-13 10:01:53eric.smithsetmessages: + msg140242
stage: needs patch -> commit review
2011-07-13 09:07:44davide.rizzosetfiles: + format00.patch

messages: + msg140238
2011-07-13 08:44:17davide.rizzosetfiles: - format00.patch
2011-07-13 08:42:55davide.rizzosetmessages: + msg140235
2011-07-13 08:36:14davide.rizzosetfiles: + format00.patch

nosy: + davide.rizzo
messages: + msg140233

keywords: + patch
2011-07-13 08:24:56eric.smithsetversions: + Python 3.2, Python 3.3
messages: + msg140231

assignee: eric.smith
components: + Interpreter Core, - Library (Lib)
stage: needs patch
2011-07-13 07:08:05ezio.melottisetnosy: + eric.smith, ezio.melotti
2011-07-13 06:58:06Gavin.Andresencreate