This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: cPickle cannot unpickle subnormal floats on some machines
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.0, Python 3.1, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: mark.dickinson Nosy List: georg.brandl, loewis, mark.dickinson, oneg, pitrou, sgala
Priority: normal Keywords: patch

Created on 2007-03-02 11:10 by oneg, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
issue1672332.patch mark.dickinson, 2009-01-16 10:35
Messages (14)
msg31399 - (view) Author: Bartlomiej Ogryczak (oneg) Date: 2007-03-02 11:10
Seems that on 64-bit machines cPickle is able to pickle floats smaller then DBL_MIN, but it's unable to unpickle them. 

64-bit machine (Solaris 10 on Sparc)
Python 2.4.4 (#1, Oct 26 2006, 10:01:37)
[GCC 3.3.4] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import cPickle
>>> cPickle.dumps(1e-310)
'F9.9999999999999694e-311\n.'
>>> cPickle.loads(_)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
ValueError: could not convert string to float

Same thing works fine on 32-bit machine (P4 with MS Windows)
Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import cPickle
>>> cPickle.dumps(1e-310)
'F0\n.'
>>>
msg31400 - (view) Author: Santiago Gala (sgala) Date: 2007-03-05 16:38
FYI: it doesn't happen for me in python 2.4.3, 2.5svn and 2.6svn built on linux (x86_64) with gcc-4.1.2, so it looks compiler version or hw platform specific.
msg31401 - (view) Author: Bartlomiej Ogryczak (oneg) Date: 2007-03-05 16:49
I forgot to specify, it's 32-bit Python binary. I'm guessing yours is 64-bit binary on 64-bit system?
Some additional info:
gcc --version
gcc (GCC) 3.3.4
$ file `which python`
ELF 32-bit MSB executable SPARC Version 1, dynamically linked, not stripped
$ ldd `which python`
        libresolv.so.2 =>        /usr/lib/libresolv.so.2
        libsocket.so.1 =>        /usr/lib/libsocket.so.1
        libnsl.so.1 =>   /usr/lib/libnsl.so.1
        librt.so.1 =>    /usr/lib/librt.so.1
        libdl.so.1 =>    /usr/lib/libdl.so.1
        libstdc++.so.5 =>        /usr/local/lib/libstdc++.so.5
        libm.so.2 =>     /usr/lib/libm.so.2
        libgcc_s.so.1 =>         /usr/local/lib/libgcc_s.so.1
        libc.so.1 =>     /usr/lib/libc.so.1
        libmp.so.2 =>    /usr/lib/libmp.so.2
        libmd5.so.1 =>   /usr/lib/libmd5.so.1
        libscf.so.1 =>   /usr/lib/libscf.so.1
        libaio.so.1 =>   /usr/lib/libaio.so.1
        libdoor.so.1 =>  /usr/lib/libdoor.so.1
        libuutil.so.1 =>         /usr/lib/libuutil.so.1
        /platform/SUNW,Sun-Fire-15000/lib/libc_psr.so.1
        /platform/SUNW,Sun-Fire-15000/lib/libmd5_psr.so.1
msg31402 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2007-03-05 21:46
I seem to remember that float pickling has been overhauled for 2.5, could you try with a 2.5 Python and report the results?
msg31403 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2007-03-10 17:09
If python is a 32-bit binary, it should be completely irrelevant that the processor is a 64-bit processor. Python can't tell the difference.
msg79134 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-01-05 12:22
I'm seeing this too on OS X, with Python 2.6.

Python 2.6.1+ (release26-maint:68182M, Jan  2 2009, 23:13:43) 
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cPickle
>>> cPickle.dumps(1e-307)
'F9.9999999999999991e-308\n.'
>>> cPickle.loads(_)
9.9999999999999991e-308
>>> cPickle.dumps(1e-308)
'F9.9999999999999991e-309\n.'
>>> cPickle.loads(_)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: could not convert string to float

Same results on trunk, and (using pickle instead of cPickle, and 
specifying protocol 0 explicitly) for 3.x.

I'll see if I can figure out where this is coming from.
msg79150 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-01-05 14:07
The problem comes down to the platform strtod:  on some systems,
strtod sets errno to ERANGE on underflow.  The load_float function in
Modules/cPickle.c calls PyOS_ascii_strtod and then raises ValueError
if that call sets errno.

I suggest replacing the call to PyOS_ascii_strtod with a call to
PyFloat_FromString instead.  This would make cPickle behave identically
to pickle, and would also fix the problem on Windows where 1e-310 is
dumped as 0.0.  (PyFloat_FromString goes out of its way to use atof 
instead of strtod to deal with this case.)
msg79154 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-01-05 15:18
Change title;  this has nothing to do with 64 bit.
msg79949 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-01-16 10:35
Here's a patch, against the trunk, that simply amends load_float to ignore 
errno on underflow (while retaining it on overflow).  This fixes the problem on my machine.

The added tests may be a little severe.  There's no *good* reason why 
dumping and then loading a float on the same machine shouldn't produce 
exactly the same value, but there are likely many bad reasons.  I propose 
to checkin the change and watch the buildbots carefully;  if there are 
failures I'll weaken the tests to check for approximate equality instead 
of exact equality.
msg80464 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-01-24 17:00
Fixed in the trunk in r68903.  If the buildbots are happy with the new 
tests, I'll port to 2.6, 3.0 and 3.1.
msg80472 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-01-24 19:07
Hmm.  After checking in this patch, the gentoo x86 trunk buildbot
has a new failure in test_pickletools, with the following output:

test_pickletools
**********************************************************************
File "/home/buildslave/python-trunk/trunk.norwitz-x86/build/Lib/pickletools.py", line ?, in 
pickletools.__test__.disassembler_test
Failed example:
    dis(pickle.dumps(random.random, 0))
Expected:
        0: c    GLOBAL     'random random'
       15: p    PUT        0
       18: .    STOP
    highest protocol among opcodes = 0
Got:
        0: c    GLOBAL     'bsddb.test.test_thread random'
       31: p    PUT        0
       34: .    STOP
    highest protocol among opcodes = 0
**********************************************************************
1 items had failures:
   1 of  25 in pickletools.__test__.disassembler_test
***Test Failed*** 1 failures.
test test_pickletools failed -- 1 of 95 doctests failed

I'm having some difficulty figuring out how this could be
related to the checkin of this patch.

Anyone have any ideas where this might be coming from?
msg80475 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-24 19:54
This is not a new failure but an erratic one, and caused by a weakness
in pickle's handling of functions. See #3657.
msg80476 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-01-24 19:59
Thanks, Antoine!
msg80484 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-01-24 21:49
Fix merged in r68907 (2.6), r68908 (3.1) and r68909 (3.0).
History
Date User Action Args
2022-04-11 14:56:22adminsetgithub: 44644
2009-01-24 21:49:48mark.dickinsonsetstatus: open -> closed
resolution: fixed
messages: + msg80484
2009-01-24 19:59:06mark.dickinsonsetmessages: + msg80476
2009-01-24 19:54:18pitrousetnosy: + pitrou
messages: + msg80475
2009-01-24 19:07:02mark.dickinsonsetmessages: + msg80472
2009-01-24 17:00:27mark.dickinsonsetmessages: + msg80464
versions: - Python 2.7
2009-01-16 10:35:04mark.dickinsonsetfiles: + issue1672332.patch
keywords: + patch
messages: + msg79949
2009-01-05 15:19:00mark.dickinsonsetmessages: + msg79154
title: cPickle can pickle, but cannot unpickle on 64bit machines -> cPickle cannot unpickle subnormal floats on some machines
2009-01-05 14:07:15mark.dickinsonsetmessages: + msg79150
2009-01-05 12:22:10mark.dickinsonsetassignee: mark.dickinson
versions: + Python 2.6, Python 3.0, Python 3.1, Python 2.7
type: behavior
messages: + msg79134
nosy: + mark.dickinson
2007-03-02 11:10:33onegcreate