Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

locale.format() and locale.format_string() cast Decimals to float #78492

Closed
james-emerton mannequin opened this issue Aug 1, 2018 · 10 comments
Closed

locale.format() and locale.format_string() cast Decimals to float #78492

james-emerton mannequin opened this issue Aug 1, 2018 · 10 comments
Labels
3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@james-emerton
Copy link
Mannequin

james-emerton mannequin commented Aug 1, 2018

BPO 34311
Nosy @ericvsmith, @cedk, @methane, @matrixise, @james-emerton
PRs
  • bpo-34311: Add locale.localize #15275
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-04-12.12:20:49.444>
    created_at = <Date 2018-08-01.20:22:51.666>
    labels = ['interpreter-core', 'type-feature', 'library', '3.10']
    title = 'locale.format() and locale.format_string() cast Decimals to float'
    updated_at = <Date 2021-04-12.12:20:49.443>
    user = 'https://github.com/james-emerton'

    bugs.python.org fields:

    activity = <Date 2021-04-12.12:20:49.443>
    actor = 'matrixise'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-04-12.12:20:49.444>
    closer = 'matrixise'
    components = ['Interpreter Core', 'Library (Lib)']
    creation = <Date 2018-08-01.20:22:51.666>
    creator = 'jemerton'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 34311
    keywords = ['patch']
    message_count = 10.0
    messages = ['322885', '322903', '322904', '323120', '349683', '359705', '359711', '359790', '390842', '390844']
    nosy_count = 6.0
    nosy_names = ['holdenweb', 'eric.smith', 'ced', 'methane', 'matrixise', 'jemerton']
    pr_nums = ['15275']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue34311'
    versions = ['Python 3.10']

    @james-emerton
    Copy link
    Mannequin Author

    james-emerton mannequin commented Aug 1, 2018

    We use locale.format('%.2f', x, True) to convert Decimal values to strings for display. Unfortunately, the locale module is using %-formatting to generate the initial string before applying locale specific formatting. As a result, any value which cannot be accurately represented as a float will produce incorrect results.

    I've built some formatting that uses new-style string formatting (and some internal locale functions) which corrects the problem.

    Unfortunately, making this change in the locale module would require converting the input format string to the new syntax, so '%.2f' would become '{:.2f}'.

    See also bpo-33731

    @james-emerton james-emerton mannequin added 3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir labels Aug 1, 2018
    @ericvsmith
    Copy link
    Member

    Would my suggestion in bpo-33731 of adding another letter in the format spec for float and decimal.Decimal solve your problem? I guess if you're using monetary=True you'd need two additional letters: like 'f' but locale aware, and like 'f' but locale aware and monetary=True. Maybe 'l' and 'L' for these? In this case, there would be no changes to the locale module.

    I don't see any good way of using new-style formatting without changing float.__format__ and decimal.Decimal.__format__.

    @ericvsmith ericvsmith added interpreter-core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement and removed 3.7 (EOL) end of life labels Aug 2, 2018
    @james-emerton
    Copy link
    Mannequin Author

    james-emerton mannequin commented Aug 2, 2018

    Certainly adding another letter to the format spec would solve my issue and would in fact be somewhat preferable to using local.format directly.

    I think this could be fixed in the locale module by transforming the format spec and using new-style formatting, but I'm not familiar enough with the corner cases to know if its practical to cover all the possible cases; particularly those coming from format_string().

    @james-emerton
    Copy link
    Mannequin Author

    james-emerton mannequin commented Aug 4, 2018

    It looks like a bot got a bit excited when I mentioned this issue in the PR for bpo-33731. I unlinked the PR but this issue still got flagged for review.

    @cedk
    Copy link
    Mannequin

    cedk mannequin commented Aug 14, 2019

    I think we can solve this issue like I solved bpo-13918 by providing a locale.localize() method which does the formatting as locale.format_string does but using the already formatted string.

    I created PR-15275 which implements it and also use the new format in locale.currency as it is highly probable that currency will be used with Decimal.

    @methane
    Copy link
    Member

    methane commented Jan 10, 2020

    Does the name "locale.localize" have some origin?

    I feel the name doesn't represent well about it is for formatting numeric based on LC_NUMERIC.

    @cedk
    Copy link
    Mannequin

    cedk mannequin commented Jan 10, 2020

    For me, the name was natural as it is the reverse operation of the existing delocalize method.

    @holdenweb
    Copy link
    Member

    Verified. Methododology:

    1. Copied test_localise.py from the PR into a master checkout.
    2. Added a null locale.localize.
    3. Verified that all new tests failed.

    .. code-block::

    Ran 64 tests in 0.023s

    FAILED (errors=4, skipped=4)
    (base) blockhead:cpython sholden$ vi Lib/locale.py
    (base) blockhead:cpython sholden$ ./python.exe -m test.test_locale
    ...................ssss....F...................testing with ('tr_TR', 'ISO8859-9') .....testing with 'en_US.UTF-8'... .testing with 'en_US.UTF-8'... .testing with 'en_US.UTF-8'... .testing with 'en_US.UTF-8'... .testing with 'en_US.UTF-8'... .testing with 'en_US.UTF-8'... .testing with 'en_US.UTF-8'... .testing with 'en_US.UTF-8'... ....F
    ======================================================================
    FAIL: test_localize_invalid_format (main.TestEnUSLocalize)
    ----------------------------------------------------------------------

      Traceback (most recent call last):
        File "/Users/sholden/cpython/Lib/test/test_locale.py", line 613, in test_localize_invalid_format
          locale.localize('foo')
      AssertionError: ValueError not raised

    ======================================================================
    FAIL: test_localize (main.TestfrFRLocalize)
    ----------------------------------------------------------------------

      Traceback (most recent call last):
        File "/Users/sholden/cpython/Lib/test/test_locale.py", line 625, in test_localize
          self._test_localize('50000.00', '50000,00')
        File "/Users/sholden/cpython/Lib/test/test_locale.py", line 601, in _test_localize
          self.assertEqual(locale.localize(value, grouping=grouping), out)
      AssertionError: '50000.00' != '50000,00'
      - 50000.00
      ?      ^
      + 50000,00
      ?      ^

    ----------------------------------------------------------------------
    Ran 64 tests in 0.024s

    FAILED (failures=2, skipped=4)

    1. Checked out cedk/locale_format branch.
    2. Observed that all locale tests now pass.

    Seems to me like this one should be good to go, so I've changed the stage to "commit review" and await the application of some core developer's commit bit.

    @matrixise matrixise added 3.10 only security fixes and removed 3.8 only security fixes labels Apr 12, 2021
    @matrixise
    Copy link
    Member

    New changeset e126547 by Cédric Krier in branch 'master':
    bpo-34311: Add locale.localize (GH-15275)
    e126547

    @matrixise
    Copy link
    Member

    I have merged the PR, thank you to Cédric for the PR, and thank you to Steve for his review.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants