classification
Title: test_math.testFsum failure on release30-maint
Type: behavior Stage: needs patch
Components: Extension Modules Versions: Python 3.0, Python 3.1, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: mark.dickinson Nosy List: mark.dickinson, pitrou
Priority: normal Keywords:

Created on 2009-03-29 01:07 by pitrou, last changed 2009-04-24 16:42 by mark.dickinson. This issue is now closed.

Messages (14)
msg84354 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-29 01:07
I started getting this in release30-maint (not in py3k).

======================================================================
FAIL: testFsum (test.test_math.MathTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antoine/py3k/30/Lib/test/test_math.py", line 443, in testFsum
    self.assertEqual(actual, expected)
AssertionError: 1.1102230246251565e-16 != 0.0

----------------------------------------------------------------------
msg84355 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-29 01:13
It only seems to happen on a 32-bit build on a 64-bit system.
msg84356 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-29 01:16
And it actually also happens in py3k (but only in 32-bit mode, too).
msg84378 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-03-29 11:56
Hmm.  I can't reproduce this.

Is this a failure that just started happening recently on this
particular platform, or is this the first time you ran the
math test with this setup?  I can't see any recent checkins
that could have precipitated this.

Could you attach the output of the configure script (just the
stdout output, not the config.log)?
msg84381 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-29 12:18
Yikes. I can't reproduce it anymore. Perhaps "make distclean" is really
necessary when switching a working copy from a 64-bit to a 32-bit
build... Sorry for the noise.
msg84383 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-03-29 12:55
> Sorry for the noise.

Not noise.  I'd still be interested in understanding where this is coming 
from;  I seem to recall someone else having exactly the same experience 
(reported bug, then found that it disappeared after a clean compile).

My best guess is that you somehow ended up in a situation where the math 
module was using the x87 FPU for floating-point, while the interpreter 
core was using SSE2.  Is this possible?

Explanation: the x87 FPU has problems with double rounding (because it 
uses 80-bit extended precision registers internally) while SSE2 doesn't.  fsum is a bit broken on systems with double rounding problems.  So there's 
a pair of lines in testFsum that look like:

if 1e16+2.0 != 1e16+2.9999:
    return

These lines are supposed to skip all these tests on platforms with the 
double rounding problem.  But if the core is using SSE2 then the test will 
fail and all the fsum tests will be executed, which is a problem if fsum 
is using x87.
msg84386 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-03-29 12:58
Found the other report of this:  see issue 3421.
msg84389 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-03-29 13:02
> My best guess is that you somehow ended up in a situation where the math 
> module was using the x87 FPU for floating-point, while the interpreter 
> core was using SSE2.  Is this possible?

I should also have said that this would fit with the 32-bit/64-bit
stuff:  I *think* it's true that for gcc on Linux, in the absence of 
compiler flags, a 64-bit build defaults to using SSE2 while a 32-bit build 
defaults to x87.
msg84391 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-29 13:09
> My best guess is that you somehow ended up in a situation where the math 
> module was using the x87 FPU for floating-point, while the interpreter 
> core was using SSE2.  Is this possible?
> 

That would be the reverse, since this occurred on a 32-bit build, i.e.
the interpreter core was using x87. But I don't understand how a 64-bit
module could be loaded by a 32-bit executable (I did check that
sys.maxsize was 2**31 - 1).

> fsum is a bit broken on systems with double rounding problems.  So
there's 
> a pair of lines in testFsum that look like:
> 
> if 1e16+2.0 != 1e16+2.9999:
>     return


Wouldn't it be a problem with stale pyc files then? The result of each
addition is stored as a constant when the code is compiled. Note how the
constants arrays differ:

64-bit:

>>> def f():
...   return 1e16+2.9999
... 
>>> dis.dis(f)
  2           0 LOAD_CONST               3 (10000000000000002.0) 
              3 RETURN_VALUE         
>>> zlib.crc32(marshal.dumps(f.__code__.co_consts))
2292868100

32-bit:

>>> def f():    
...   return 1e16+2.9999
... 
>>> dis.dis(f)  
  2           0 LOAD_CONST               3 (10000000000000004.0) 
              3 RETURN_VALUE         
>>> zlib.crc32(marshal.dumps(f.__code__.co_consts))
103113703
msg84394 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-03-29 13:20
> Wouldn't it be a problem with stale pyc files then?

Hah!  Yes!  That seems entirely likely.

So what sequence of moves does one have to go through
to reproduce this?
msg84396 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-29 13:36
> > Wouldn't it be a problem with stale pyc files then?
> 
> Hah!  Yes!  That seems entirely likely.
> 
> So what sequence of moves does one have to go through
> to reproduce this?

I suppose: first run "-m test.regrtest -v test_math" in 64-bit mode,
then rebuild in 32-bit mode (*) without doing "make (dist)clean", then
run "-m test.regrtest -v test_math" again (in 32-bit mode).

(*) CC="gcc -m32" ./configure
msg84401 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-03-29 14:11
Thanks, Antoine!  Yes, I can now reproduce the testFsum failure on my 
MacBook Pro (OS X 10.5.6/x86_64, with Apple's gcc 4.0.1) using the 
following sequence of commands:  (I tested this for the trunk, but I 
py3k should be just the same).

make distclean
CC="gcc -arch x86_64" ./configure && make
./python.exe -m test.regrtest -v test_math
rm Parser/*.o
CC="gcc -mfpmath=387" ./configure && make
./python.exe -m test.regrtest -v test_math

(the rm is necessary to avoid a 'wrong architecture' build failure).

It might be worth making the tests a bit more robust here;  I'll
take a look.  On the other hand, there are plans to replace fsum
with a double-rounding-friendly version.
msg84402 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-29 14:15
I think it would be sufficient to invoke the addition through a helper
function, that is:

def add(x, y):
    return x + y

if add(1e16, 2.0) != add(1e16, 2.9999):
    return

Also, instead of "return", you might use the new "raise
unittest.SkipTest('some message')".
msg86414 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-04-24 16:42
Test fixed for 2.7, 3.1 in r71837, r71839
History
Date User Action Args
2009-04-24 16:42:21mark.dickinsonsetstatus: open -> closed
resolution: fixed
messages: + msg86414
2009-03-29 14:15:46pitrousetmessages: + msg84402
2009-03-29 14:11:18mark.dickinsonsetversions: + Python 2.6, Python 2.7
messages: + msg84401

assignee: mark.dickinson
components: + Extension Modules, - Interpreter Core
stage: needs patch
2009-03-29 13:36:48pitrousetmessages: + msg84396
2009-03-29 13:20:41mark.dickinsonsetmessages: + msg84394
2009-03-29 13:09:13pitrousetstatus: closed -> open
resolution: not a bug -> (no value)
messages: + msg84391
2009-03-29 13:02:55mark.dickinsonsetmessages: + msg84389
2009-03-29 12:58:48mark.dickinsonsetmessages: + msg84386
2009-03-29 12:55:34mark.dickinsonsetmessages: + msg84383
2009-03-29 12:18:34pitrousetstatus: open -> closed
resolution: not a bug
messages: + msg84381
2009-03-29 11:56:47mark.dickinsonsetmessages: + msg84378
2009-03-29 01:16:47pitrousetmessages: + msg84356
versions: + Python 3.1
2009-03-29 01:14:03pitrousetmessages: + msg84355
2009-03-29 01:07:40pitroucreate