Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testFsum failure caused by constant folding of a float expression #83173

Closed
xdegaye mannequin opened this issue Dec 7, 2019 · 8 comments
Closed

testFsum failure caused by constant folding of a float expression #83173

xdegaye mannequin opened this issue Dec 7, 2019 · 8 comments
Labels
3.9 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@xdegaye
Copy link
Mannequin

xdegaye mannequin commented Dec 7, 2019

BPO 38992
Nosy @malemburg, @tim-one, @rhettinger, @mdickinson, @vstinner, @xdegaye, @pablogsal, @isidentical
PRs
  • bpo-38992: avoid fsum test failure from constant-folding #17513
  • [3.8] bpo-38992: avoid fsum test failure from constant-folding (GH-17513) #17530
  • Files
  • foo.x86_64
  • foo.arm64
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-12-09.18:22:49.217>
    created_at = <Date 2019-12-07.11:32:08.267>
    labels = ['type-bug', 'tests', '3.9']
    title = 'testFsum failure caused by constant folding of a float expression'
    updated_at = <Date 2019-12-09.18:22:49.212>
    user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2019-12-09.18:22:49.212>
    actor = 'mark.dickinson'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-12-09.18:22:49.217>
    closer = 'mark.dickinson'
    components = ['Tests']
    creation = <Date 2019-12-07.11:32:08.267>
    creator = 'xdegaye'
    dependencies = []
    files = ['48762', '48763']
    hgrepos = []
    issue_num = 38992
    keywords = ['patch']
    message_count = 8.0
    messages = ['357969', '358024', '358025', '358031', '358069', '358103', '358120', '358123']
    nosy_count = 9.0
    nosy_names = ['lemburg', 'tim.peters', 'rhettinger', 'mark.dickinson', 'vstinner', 'stutzbach', 'xdegaye', 'pablogsal', 'BTaskaya']
    pr_nums = ['17513', '17530']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue38992'
    versions = ['Python 3.9']

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 7, 2019

    Title: testFsum failure caused by constant folding of a float expression

    Description:
    ------------
    Python (Python 3.9.0a1+ heads/master-dirty:ea9835c5d1) is built on a Linux x86_64. This native interpreter is used to cross-compile Python (using the same source) to Android API 24. Next the installation is done locally to DESTDIR by running 'make install' with the env var DESTDIR set and the standard library modules are compiled by the native interpreter during this process. The content of DESTDIR is then copied to an arm64 android device (Huawei FWIW). The test_math.MathTests.testFsum test fails on the android device with:

    AssertionError: -4.309103330548428e+214 != -1.0

    This occurs when testing '([1.7**(i+1)-1.7**i for i in range(1000)] + [-1.7**1000], -1.0)' in test_values.

    Next the test_math.py file is touched on the android device to force recompilation of the module and testFsum becomes surprisingly successful.

    Investigation:
    --------------
    The hexadecimal representation of 1.7**n on x86_64 and arm64 are:

    • different for n in (10, 100, 1000)
    • equal for n in [0, 9] or 11
    on x86_64:
    >>> 1.7**10
    201.59939004489993
    >>> (1.7**10).hex()
    '0x1.9332e34080c95p+7'
    
    on arm64:
    >>> 1.7**10
    201.59939004489996
    >>> (1.7**10).hex()
    '0x1.9332e34080c96p+7'

    The output of the following foo.py module that has been run on x86_64 and arm64 are attached to this issue:

    #######################
    import math, dis
    
    def test_fsum():
        x = [1.7**(i+1)-1.7**i for i in range(10)] + [-1.7**10]
        return x
    
    y = test_fsum()
    print(y)
    print(math.fsum(y))
    dis.dis(test_fsum)
    #######################

    The only difference between both dissasembly of test_fsum() is at bytecode 16 that loads the folded constant 1.7**10.

    Conclusion:
    -----------
    The compilation of the expression '[1.7**(i+1)-1.7**i for i in range(1000)] + [-1.7**1000]' on x86_64 folds '1.7**1000' to 2.8113918290273277e+230 When the list comprehension (the first term of the expression) is executed on arm64, then 1.7**1000 is evaluated as 2.8113918290273273e+230. On arm64 1.7**1000 - 2.8113918290273277e+230 = -4.309103330548428e+214, hence the AssertionError above.

    This is confirmed by changing testFsum to prevent constant folding by replacing 1000 in the testFsum expression with a variable whose value is 1000. In that case the test_math module compiled on x86_64 is successful on arm64. This could be a fix for this issue unless this fix would be hiding another problem such as .pyc files portability across different platforms and my knowledge of IEEE 754 is too superficial to answer that point.

    @xdegaye xdegaye mannequin added 3.9 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Dec 7, 2019
    @mdickinson
    Copy link
    Member

    So if I'm understanding correctly, the cause of the issue is that the value 1.7**(i+1) computed in the last iteration (i=999) of the list comprehension doesn't exactly match the -1.7**1000 value, because the former is computed at runtime using the libm's pow, while the latter is constant-folded and likely uses something more accurate than pow.

    I think it should be easy to rewrite the test so that it precomputes the powers of 1.7, and then makes sure to use those computed values (i.e., so that we're only computing 1.7**1000 once rather than twice, eliminating the possibility of getting different results).

    @mdickinson
    Copy link
    Member

    Note that the exact values of 1.7**i don't matter here - some sloppiness in the pow results should not cause a test failure. The key point is that under fairly mild assumptions about IEEE 754 conformance, the subtractions 1.7**(i+1) - 1.7**i are always performed exactly, thanks to Sterbenz's lemma.

    @mdickinson
    Copy link
    Member

    @xdegaye Please could you test whether the PR #61715 fixes the issue for you?

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 9, 2019

    Yes PR #61715 does fix the problem.
    Thanks Mark.

    @mdickinson
    Copy link
    Member

    New changeset bba873e by Mark Dickinson in branch 'master':
    bpo-38992: avoid fsum test failure from constant-folding (GH-17513)
    bba873e

    @mdickinson
    Copy link
    Member

    New changeset 3c5feaf by Mark Dickinson (Miss Islington (bot)) in branch '3.8':
    bpo-38992: avoid fsum test failure from constant-folding (GH-17513) (GH-17530)
    3c5feaf

    @mdickinson
    Copy link
    Member

    Fixed in master and 3.8. Not sure this is worth backporting to 3.7.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant