New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
string.Formatter returns str for empty unicode template #60155
Comments
Expected behavior of string.Formatter() is to return unicode strings for unicode templates, and "byte" strings for str templates. Which is exactly what it does, with one frustrating exception: for empty unicode string it returns byte str. Test follows: import string
template = u""
result = string.Formatter().format(template)
assert isinstance(result, unicode)
# AssertionError |
Adding failing test. Patch coming next. |
Here are some related failing cases that I found: >>> f = string.Formatter()
>>> f.format(u"{0}", "")
''
>>> f.format(u"{0}", 1)
'1'
>>> f.format(u"{0}", "a")
'a'
>>> f.format(u"{0}{1}", "a", "b")
'ab'
>>> f.format("{0}", u"a")
u'a' Note that PEP-3101 says the following: "In all cases, the type of the format string dominates - that |
Actually, I'm going to defer on creating a patch because this covers more scenarios than I originally thought and so may require more time. |
What about cases like this? >>> f.format(u'{0}', '\xe9')
'\xe9' It seems fixing this issue for non-empty strings would cause formerly running cases like this to raise UnicodeDecodeError. >>> unicode('\xe9')
...
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 0: ordinal not in range(128) Would that be acceptable? |
Note that I didn't say it was correct, I just said it was consistent :) And no, breaking stuff that current works is a non-starter for 2.7. |
I filed bpo-15952 for the behavior difference between format(value) and value.__format__() and the related lack of documentation re: unicode format strings. Given that the expected behavior for the current issue doesn't seem to be documented (aside from PEP-3101, which is probably too late to follow), we should probably agree on what the behavior should be (as well as documenting it) before or while addressing this issue. |
Attached is a proposed patch. Some explanation behind the patch that stems from the above comments: The following is an example of Formatter.format() returning str in the current implementation that would break if we made Formatter.format() return unicode whenever format_string is unicode: >>> f.format(u"{0}", "\xc3\xa9") # UTF-8 encoded "e-acute".
'\xc3\xa9' (It would break with a UnicodeDecodeError because 'ascii' is the default encoding.) Since we can't change Formatter.format(format_string) to return unicode whenever format_string is unicode without breaking existing code, I believe the best we can do is to document the departure from PEP-3101. Since the caller has to handle return values of type str anyways, I don't think it helps to ensure that more return values are unicode. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: