Title: locale.format() and locale.format_string() cast Decimals to float
Type: enhancement Stage: resolved
Components: Interpreter Core, Library (Lib) Versions: Python 3.10
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ced, eric.smith, holdenweb, jemerton, matrixise, methane
Priority: normal Keywords: patch

Created on 2018-08-01 20:22 by jemerton, last changed 2021-04-12 12:20 by matrixise. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 15275 merged ced, 2019-08-14 11:25
Messages (10)
msg322885 - (view) Author: James Emerton (jemerton) * Date: 2018-08-01 20:22
We use locale.format('%.2f', x, True) to convert Decimal values to strings for display. Unfortunately, the locale module is using %-formatting to generate the initial string before applying locale specific formatting. As a result, any value which cannot be accurately represented as a float will produce incorrect results.

I've built some formatting that uses new-style string formatting (and some internal locale functions) which corrects the problem.

Unfortunately, making this change in the locale module would require converting the input format string to the new syntax, so '%.2f' would become '{:.2f}'.

See also #33731
msg322903 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2018-08-02 00:40
Would my suggestion in #33731 of adding another letter in the format spec for float and decimal.Decimal solve your problem? I guess if you're using monetary=True you'd need two additional letters: like 'f' but locale aware, and like 'f' but locale aware and monetary=True.  Maybe 'l' and 'L' for these? In this case, there would be no changes to the locale module.

I don't see any good way of using new-style formatting without changing float.__format__ and decimal.Decimal.__format__.
msg322904 - (view) Author: James Emerton (jemerton) * Date: 2018-08-02 00:48
Certainly adding another letter to the format spec would solve my issue and would in fact be somewhat preferable to using local.format directly.

I think this could be fixed in the locale module by transforming the format spec and using new-style formatting, but I'm not familiar enough with the corner cases to know if its practical to cover all the possible cases; particularly those coming from format_string().
msg323120 - (view) Author: James Emerton (jemerton) * Date: 2018-08-04 22:14
It looks like a bot got a bit excited when I mentioned this issue in the PR for bpo-33731. I unlinked the PR but this issue still got flagged for review.
msg349683 - (view) Author: Cédric Krier (ced) * Date: 2019-08-14 11:28
I think we can solve this issue like I solved issue13918 by providing a locale.localize() method which does the formatting as locale.format_string does but using the already formatted string.

I created PR-15275 which implements it and also use the new format in locale.currency as it is highly probable that currency will be used with Decimal.
msg359705 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-01-10 06:44
Does the name "locale.localize" have some origin?

I feel the name doesn't represent well about it is for formatting numeric based on LC_NUMERIC.
msg359711 - (view) Author: Cédric Krier (ced) * Date: 2020-01-10 08:26
For me, the name was natural as it is the reverse operation of the existing delocalize method.
msg359790 - (view) Author: Steve Holden (holdenweb) * (Python committer) Date: 2020-01-11 10:40
Verified. Methododology:

1. Copied from the PR into a master checkout.
2. Added a null locale.localize.
3. Verified that all new tests failed.

.. code-block::

  Ran 64 tests in 0.023s

  FAILED (errors=4, skipped=4)
  (base) blockhead:cpython sholden$ vi Lib/
  (base) blockhead:cpython sholden$ ./python.exe -m test.test_locale
  ...................ssss....F...................testing with ('tr_TR', 'ISO8859-9') .....testing with 'en_US.UTF-8'... .testing with 'en_US.UTF-8'... .testing with 'en_US.UTF-8'... .testing with 'en_US.UTF-8'... .testing with 'en_US.UTF-8'... .testing with 'en_US.UTF-8'... .testing with 'en_US.UTF-8'... .testing with 'en_US.UTF-8'... ....F
  FAIL: test_localize_invalid_format (__main__.TestEnUSLocalize)
  Traceback (most recent call last):
    File "/Users/sholden/cpython/Lib/test/", line 613, in test_localize_invalid_format
  AssertionError: ValueError not raised

  FAIL: test_localize (__main__.TestfrFRLocalize)
  Traceback (most recent call last):
    File "/Users/sholden/cpython/Lib/test/", line 625, in test_localize
      self._test_localize('50000.00', '50000,00')
    File "/Users/sholden/cpython/Lib/test/", line 601, in _test_localize
      self.assertEqual(locale.localize(value, grouping=grouping), out)
  AssertionError: '50000.00' != '50000,00'
  - 50000.00
  ?      ^
  + 50000,00
  ?      ^

  Ran 64 tests in 0.024s

  FAILED (failures=2, skipped=4)

4. Checked out cedk/locale_format branch.
5. Observed that all locale tests now pass.

Seems to me like this one should be good to go, so I've changed the stage to "commit review" and await the application of some core developer's commit bit.
msg390842 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2021-04-12 12:17
New changeset e126547c070fbc080562abb08e16a2c93a8a805d by Cédric Krier in branch 'master':
bpo-34311: Add locale.localize (GH-15275)
msg390844 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2021-04-12 12:20
I have merged the PR, thank you to Cédric for the PR, and thank you to Steve for his review.
Date User Action Args
2021-04-12 12:20:49matrixisesetstatus: open -> closed
resolution: fixed
messages: + msg390844

stage: commit review -> resolved
2021-04-12 12:17:48matrixisesetnosy: + matrixise
messages: + msg390842
2021-04-12 10:14:24matrixisesetversions: + Python 3.10, - Python 3.8
2020-01-11 10:40:34holdenwebsetnosy: + holdenweb

messages: + msg359790
stage: patch review -> commit review
2020-01-10 08:26:58cedsetmessages: + msg359711
2020-01-10 06:44:02methanesetnosy: + methane
messages: + msg359705
2019-08-14 11:28:50cedsetnosy: + ced
messages: + msg349683
2019-08-14 11:25:09cedsetpull_requests: + pull_request14997
2018-08-04 22:14:21jemertonsetmessages: + msg323120
2018-08-04 22:12:52jemertonsetpull_requests: - pull_request8166
2018-08-04 22:07:37jemertonsetkeywords: + patch
stage: patch review
pull_requests: + pull_request8166
2018-08-02 00:48:39jemertonsetmessages: + msg322904
2018-08-02 00:40:06eric.smithsettype: enhancement
messages: + msg322903
components: + Interpreter Core
versions: - Python 3.6, Python 3.7
2018-08-02 00:29:02eric.smithsetnosy: + eric.smith
2018-08-01 20:22:51jemertoncreate