classification
Title: string.Formatter doesn't understand the a conversion specifier
Type: behavior Stage: resolved
Components: Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.smith Nosy List: eric.araujo, eric.smith, francismb, ncoghlan, python-dev, r.david.murray, terry.reedy
Priority: normal Keywords: patch

Created on 2011-12-11 07:09 by ncoghlan, last changed 2012-08-19 22:01 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
issue13579_80a50b7ad88f.patch francismb, 2012-02-13 19:59 Changes as proposed by Terry review
issue13579_a99632426af5.patch francismb, 2012-02-15 18:33 review
issue13579_47288880cc9e.patch francismb, 2012-02-18 12:10 review
issue13579_0554183066b5.patch francismb, 2012-03-17 17:42 review
issue13579_910a4b12c796.patch francismb, 2012-04-30 16:25 review
Messages (19)
msg149214 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-12-11 07:09
As the subject line says:

>>> fmt = "{0!a}"
>>> fmt.format(10)
'10'
>>> import string
>>> string.Formatter().format(fmt, 10)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.2/string.py", line 180, in format
    return self.vformat(format_string, args, kwargs)
  File "/usr/lib/python3.2/string.py", line 184, in vformat
    result = self._vformat(format_string, args, kwargs, used_args, 2)
  File "/usr/lib/python3.2/string.py", line 210, in _vformat
    obj = self.convert_field(obj, conversion)
  File "/usr/lib/python3.2/string.py", line 245, in convert_field
    raise ValueError("Unknown conversion specifier {0!s}".format(conversion))
ValueError: Unknown conversion specifier a
msg149645 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-12-16 23:47
At line 237 of string.py:
    def convert_field(self, value, conversion):
        # do any conversion on the resulting object
        if conversion == 'r':
            return repr(value)
        elif conversion == 's':
            return str(value)
        elif conversion is None:
            return value
add
        elif conversion == 'a':
            return ascii(value)

The alternatives should be ordered by decreasing frequency. I suspect that should be None, 's', 'r', 'a'.
msg153291 - (view) Author: Francis MB (francismb) * Date: 2012-02-13 19:59
Hi,
here's a patch with the changes proposed by Terry

Cheers,
francis
msg153389 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-02-15 04:40
Doc/library/string.rst string.Formatter().format(fmt, 10)needs a couple of changes also.

1. "format(format_string, *args, **kwargs) 
format() is the primary API method. It takes a format template string,"

I think 'template' should be removed as we elsewhere (as in the next section) just call them 'format strings'.

Note: I consider this and the preceding sentences

"The built-in string class provides the ability to do complex variable substitutions and value formatting via the format() method described in PEP 3101. The Formatter class in the string module allows you to create and customize your own string formatting behaviors using the same implementation as the built-in format() method."

to imply that this method should accept any format string that str.format does, and that the patch is a bug fix rather than enhancement. There is no mention of any limitation or subseting.

2. "convert_field(value, conversion) 
Converts the value (returned by get_field()) given a conversion type (as in the tuple returned by the parse() method). The default version understands ‘r’ (repr) and ‘s’ (str) conversion types."

Change "‘r’ (repr) and ‘s’ (str)" to "‘s’ (str), ‘r’ (repr), and ‘a’ (ascii)". I consider this a bug in the doc.

---
I discovered that string.Formatter.format has another problem:
it does not auto-number fields

>>> '{0}'.format(10) == '{}'.format(10) == '10'
True
>>> from string import Formatter as F
>>> F().format('{0}', 10)
'10'
>>> F().format('{}', 10)
Traceback (most recent call last):
  File "<pyshell#32>", line 1, in <module>
    F().format('{}', 10)
  File "C:\Programs\Python32\lib\string.py", line 180, in format
    return self.vformat(format_string, args, kwargs)
  File "C:\Programs\Python32\lib\string.py", line 184, in vformat
    result = self._vformat(format_string, args, kwargs, used_args, 2)
  File "C:\Programs\Python32\lib\string.py", line 206, in _vformat
    obj, arg_used = self.get_field(field_name, args, kwargs)
  File "C:\Programs\Python32\lib\string.py", line 267, in get_field
    obj = self.get_value(first, args, kwargs)
  File "C:\Programs\Python32\lib\string.py", line 226, in get_value
    return kwargs[key]
KeyError: ''

So it seems that str.format is either calling something other than _string.formatter_parser or else pre or post processing.

The underlying problem is trying to keep separate codes that do the same thing in sync without testing both with the same set of examples. If examples were in a table of (fmt, args, kwargs, output) tuples, it would be easy to call fmt.format(*args, **kwargs) and F().vformat(fmt, args, kwargs) with each and compare to expected. There are also the %format torture tests in test_format.py. Do % and {} formatting use any of the same code?
msg153393 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-02-15 08:16
Agreed that this is a bug in string.Formatter rather than a new feature.

There's already a separate bug for the autonumbering problem: http://bugs.python.org/issue13598

And I created a new issue about unifying some of the tests: http://bugs.python.org/issue14019

%-style string formatting is its own thing - it doesn't share semantics or code with str.format or string.Formatter.
msg153426 - (view) Author: Francis MB (francismb) * Date: 2012-02-15 18:33
I have updated the patch with the documentation changes proposed and also added a news entry (does a new entry has to be added in some order?).

Thank for the review in advance.
msg153429 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-02-15 19:55
I just noticed two more minor errors that were and remain in "It takes a format string, and an arbitrary set of positional and keyword argument." Remove the comma and make 'argument' plural.

As near as I can tell, the two change block beginning with 
+In less formal terms,
+keyword.  If it's a number,
are strictly re-wrapping and no text changes. Correct?

Francisco - News entries are usually entered at the top of the section, just under the heading. But they are a bit tricky for multi-version patches because the context following is often different for different version. I have not decided which is easier -- resolving a merge conflict with a 3-way merge tool or separately editing Misc/NEWS for each version.

Eric, I near as I can tell, and given Nick's last response, this will be ready to commit with the two minor text changes and movement of the NEWS entry. Francisco is already in Misc/ACKS.
msg153430 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-02-15 20:02
However, I have not actually applied and run test_string.py.
msg153641 - (view) Author: Francis MB (francismb) * Date: 2012-02-18 12:10
Updated. The NEWS line is now at the top of the section.

>As near as I can tell, the two change block beginning with 
>+In less formal terms,
>[...]
>are strictly re-wrapping and no text changes. Correct?

Yes. Just a pydev question here: I now this re-wrapping
adds noise to the patch, but can/should be done?

Please correct me if I'm wrong: one should try to
limit the line width to 80. If one word starts before
and ends behind is not bad but new words should start
in a new line.

Now in the devguide states “To perform a quick sanity check on your path, you can run make patchcheck”

I did that (I'm using the mq approach here) but I get no useful information (AFAIN):
-----------------------------------------------
Modules/Setup.dist is newer than Modules/Setup;
check to make sure you have all the updates you
need in your Modules/Setup file.
Usually, copying Modules/Setup.dist to Modules/Setup will work.
-----------------------------------------------
./python ./Tools/scripts/patchcheck.py
Getting the list of files that have been added/changed ... 0 files
Fixing whitespace ... 0 files
Fixing C file whitespace ... 0 files
Fixing docs whitespace ... 0 files
Docs modified ... NO
Misc/ACKS updated ... NO
Misc/NEWS updated ... NO


Thanks in advance !
msg153646 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-02-18 15:21
On Sat, Feb 18, 2012 at 10:10 PM, Francisco Martín Brugué
<report@bugs.python.org> wrote:
> ./python ./Tools/scripts/patchcheck.py
> Getting the list of files that have been added/changed ... 0 files
> Fixing whitespace ... 0 files
> Fixing C file whitespace ... 0 files
> Fixing docs whitespace ... 0 files
> Docs modified ... NO
> Misc/ACKS updated ... NO
> Misc/NEWS updated ... NO

I suspect mq will confuse make patchcheck (since it looks for
differences between the working copy and the hg branch tip, and there
are no such differences for an applied mq patch).

Cheers,
Nick.
msg153669 - (view) Author: Francis MB (francismb) * Date: 2012-02-19 05:42
> I suspect mq will confuse make patchcheck (since it looks for
> differences between the working copy and the hg branch tip, and there
> are no such differences for an applied mq patch).

Does it makes sense to open a feature request for that?
msg153704 - (view) Author: Francis MB (francismb) * Date: 2012-02-19 15:28
Well, it's done:

http://bugs.python.org/issue14053
msg156153 - (view) Author: Francis MB (francismb) * Date: 2012-03-17 11:11
> However, I have not actually applied and run test_string.py.
I've applied the issue13579_47288880cc9e.patch to the changeset 0554183066b5 and applies without errors for me.
msg156158 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-03-17 13:28
To answer your question about wrapping, it is best not to introduce noise into the patch.  The person who commits it can rewrap (although even then it is nice to do the rewrapping in a separate changeset).
msg156182 - (view) Author: Francis MB (francismb) * Date: 2012-03-17 17:42
Ok, I've updated the patch.

The NEWS entry could be:
Issue #13579: string.Formatter now understands the !a conversion specifier.

(not in the patch because AFAIN it makes the merge easier)

Let me know if that's in the line you want. Thank you!
msg158496 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-04-16 20:03
To be exact, the specifier is "a", without "!" which is a delimiter. :)
msg159705 - (view) Author: Francis MB (francismb) * Date: 2012-04-30 16:25
The patch is updated. Please let me know.

And as Éric noticed the NEWS entry could be: 

Issue #13579: string.Formatter now understands the "a" conversion specifier.

Thanks!
msg168597 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-19 21:59
New changeset 24b449a77e88 by R David Murray in branch '3.2':
#13579: teach string.Formatter about 'a'.
http://hg.python.org/cpython/rev/24b449a77e88

New changeset 4feb10457c13 by R David Murray in branch 'default':
Merge #13579: teach string.Formatter about 'a'.
http://hg.python.org/cpython/rev/4feb10457c13

New changeset c793d62cdecc by R David Murray in branch '2.7':
#13579: minimize code base drift for 'a' string.Formatter change.
http://hg.python.org/cpython/rev/c793d62cdecc
msg168598 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-08-19 22:01
I've reviewed the patch and applied it.

2.7 doesn't support !a or 'ascii()', but I did backport the doc changes and the reordering of the clauses in order to minimize code base drift.

Thanks, Francisco.
History
Date User Action Args
2012-08-19 22:01:19r.david.murraysetstatus: open -> closed
resolution: fixed
messages: + msg168598

stage: commit review -> resolved
2012-08-19 21:59:08python-devsetnosy: + python-dev
messages: + msg168597
2012-04-30 16:25:16francismbsetfiles: + issue13579_910a4b12c796.patch

messages: + msg159705
2012-04-16 20:03:02eric.araujosetnosy: + eric.araujo

messages: + msg158496
title: string.Formatter doesn't understand the !a conversion specifier -> string.Formatter doesn't understand the a conversion specifier
2012-03-17 17:42:44francismbsetfiles: + issue13579_0554183066b5.patch

messages: + msg156182
2012-03-17 13:28:19r.david.murraysetnosy: + r.david.murray
messages: + msg156158
2012-03-17 11:11:53francismbsetmessages: + msg156153
2012-02-19 15:28:13francismbsetmessages: + msg153704
2012-02-19 05:43:00francismbsetmessages: + msg153669
2012-02-18 15:21:07ncoghlansetmessages: + msg153646
2012-02-18 12:10:57francismbsetfiles: + issue13579_47288880cc9e.patch

messages: + msg153641
2012-02-15 20:02:58terry.reedysetmessages: + msg153430
2012-02-15 19:55:25terry.reedysetmessages: + msg153429
stage: patch review -> commit review
2012-02-15 18:33:26francismbsetfiles: + issue13579_a99632426af5.patch

messages: + msg153426
2012-02-15 08:16:51ncoghlansetmessages: + msg153393
2012-02-15 04:40:51terry.reedysetstage: needs patch -> patch review
2012-02-15 04:40:24terry.reedysetmessages: + msg153389
2012-02-13 19:59:30francismbsetfiles: + issue13579_80a50b7ad88f.patch

nosy: + francismb
messages: + msg153291

keywords: + patch
2011-12-16 23:50:03terry.reedysetstage: needs patch
type: behavior
versions: + Python 2.7, Python 3.2, Python 3.3
2011-12-16 23:47:31terry.reedysetnosy: + terry.reedy
messages: + msg149645
2011-12-11 16:26:02eric.smithsetassignee: eric.smith

nosy: + eric.smith
2011-12-11 07:09:49ncoghlancreate