msg31399 - (view) |
Author: Bartlomiej Ogryczak (oneg) |
Date: 2007-03-02 11:10 |
Seems that on 64-bit machines cPickle is able to pickle floats smaller then DBL_MIN, but it's unable to unpickle them.
64-bit machine (Solaris 10 on Sparc)
Python 2.4.4 (#1, Oct 26 2006, 10:01:37)
[GCC 3.3.4] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import cPickle
>>> cPickle.dumps(1e-310)
'F9.9999999999999694e-311\n.'
>>> cPickle.loads(_)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ValueError: could not convert string to float
Same thing works fine on 32-bit machine (P4 with MS Windows)
Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import cPickle
>>> cPickle.dumps(1e-310)
'F0\n.'
>>>
|
msg31400 - (view) |
Author: Santiago Gala (sgala) |
Date: 2007-03-05 16:38 |
FYI: it doesn't happen for me in python 2.4.3, 2.5svn and 2.6svn built on linux (x86_64) with gcc-4.1.2, so it looks compiler version or hw platform specific.
|
msg31401 - (view) |
Author: Bartlomiej Ogryczak (oneg) |
Date: 2007-03-05 16:49 |
I forgot to specify, it's 32-bit Python binary. I'm guessing yours is 64-bit binary on 64-bit system?
Some additional info:
gcc --version
gcc (GCC) 3.3.4
$ file `which python`
ELF 32-bit MSB executable SPARC Version 1, dynamically linked, not stripped
$ ldd `which python`
libresolv.so.2 => /usr/lib/libresolv.so.2
libsocket.so.1 => /usr/lib/libsocket.so.1
libnsl.so.1 => /usr/lib/libnsl.so.1
librt.so.1 => /usr/lib/librt.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libstdc++.so.5 => /usr/local/lib/libstdc++.so.5
libm.so.2 => /usr/lib/libm.so.2
libgcc_s.so.1 => /usr/local/lib/libgcc_s.so.1
libc.so.1 => /usr/lib/libc.so.1
libmp.so.2 => /usr/lib/libmp.so.2
libmd5.so.1 => /usr/lib/libmd5.so.1
libscf.so.1 => /usr/lib/libscf.so.1
libaio.so.1 => /usr/lib/libaio.so.1
libdoor.so.1 => /usr/lib/libdoor.so.1
libuutil.so.1 => /usr/lib/libuutil.so.1
/platform/SUNW,Sun-Fire-15000/lib/libc_psr.so.1
/platform/SUNW,Sun-Fire-15000/lib/libmd5_psr.so.1
|
msg31402 - (view) |
Author: Georg Brandl (georg.brandl) * |
Date: 2007-03-05 21:46 |
I seem to remember that float pickling has been overhauled for 2.5, could you try with a 2.5 Python and report the results?
|
msg31403 - (view) |
Author: Martin v. Löwis (loewis) * |
Date: 2007-03-10 17:09 |
If python is a 32-bit binary, it should be completely irrelevant that the processor is a 64-bit processor. Python can't tell the difference.
|
msg79134 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-05 12:22 |
I'm seeing this too on OS X, with Python 2.6.
Python 2.6.1+ (release26-maint:68182M, Jan 2 2009, 23:13:43)
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cPickle
>>> cPickle.dumps(1e-307)
'F9.9999999999999991e-308\n.'
>>> cPickle.loads(_)
9.9999999999999991e-308
>>> cPickle.dumps(1e-308)
'F9.9999999999999991e-309\n.'
>>> cPickle.loads(_)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: could not convert string to float
Same results on trunk, and (using pickle instead of cPickle, and
specifying protocol 0 explicitly) for 3.x.
I'll see if I can figure out where this is coming from.
|
msg79150 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-05 14:07 |
The problem comes down to the platform strtod: on some systems,
strtod sets errno to ERANGE on underflow. The load_float function in
Modules/cPickle.c calls PyOS_ascii_strtod and then raises ValueError
if that call sets errno.
I suggest replacing the call to PyOS_ascii_strtod with a call to
PyFloat_FromString instead. This would make cPickle behave identically
to pickle, and would also fix the problem on Windows where 1e-310 is
dumped as 0.0. (PyFloat_FromString goes out of its way to use atof
instead of strtod to deal with this case.)
|
msg79154 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-05 15:18 |
Change title; this has nothing to do with 64 bit.
|
msg79949 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-16 10:35 |
Here's a patch, against the trunk, that simply amends load_float to ignore
errno on underflow (while retaining it on overflow). This fixes the problem on my machine.
The added tests may be a little severe. There's no *good* reason why
dumping and then loading a float on the same machine shouldn't produce
exactly the same value, but there are likely many bad reasons. I propose
to checkin the change and watch the buildbots carefully; if there are
failures I'll weaken the tests to check for approximate equality instead
of exact equality.
|
msg80464 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-24 17:00 |
Fixed in the trunk in r68903. If the buildbots are happy with the new
tests, I'll port to 2.6, 3.0 and 3.1.
|
msg80472 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-24 19:07 |
Hmm. After checking in this patch, the gentoo x86 trunk buildbot
has a new failure in test_pickletools, with the following output:
test_pickletools
**********************************************************************
File "/home/buildslave/python-trunk/trunk.norwitz-x86/build/Lib/pickletools.py", line ?, in
pickletools.__test__.disassembler_test
Failed example:
dis(pickle.dumps(random.random, 0))
Expected:
0: c GLOBAL 'random random'
15: p PUT 0
18: . STOP
highest protocol among opcodes = 0
Got:
0: c GLOBAL 'bsddb.test.test_thread random'
31: p PUT 0
34: . STOP
highest protocol among opcodes = 0
**********************************************************************
1 items had failures:
1 of 25 in pickletools.__test__.disassembler_test
***Test Failed*** 1 failures.
test test_pickletools failed -- 1 of 95 doctests failed
I'm having some difficulty figuring out how this could be
related to the checkin of this patch.
Anyone have any ideas where this might be coming from?
|
msg80475 - (view) |
Author: Antoine Pitrou (pitrou) * |
Date: 2009-01-24 19:54 |
This is not a new failure but an erratic one, and caused by a weakness
in pickle's handling of functions. See #3657.
|
msg80476 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-24 19:59 |
Thanks, Antoine!
|
msg80484 - (view) |
Author: Mark Dickinson (mark.dickinson) * |
Date: 2009-01-24 21:49 |
Fix merged in r68907 (2.6), r68908 (3.1) and r68909 (3.0).
|
|
Date |
User |
Action |
Args |
2022-04-11 14:56:22 | admin | set | github: 44644 |
2009-01-24 21:49:48 | mark.dickinson | set | status: open -> closed resolution: fixed messages:
+ msg80484 |
2009-01-24 19:59:06 | mark.dickinson | set | messages:
+ msg80476 |
2009-01-24 19:54:18 | pitrou | set | nosy:
+ pitrou messages:
+ msg80475 |
2009-01-24 19:07:02 | mark.dickinson | set | messages:
+ msg80472 |
2009-01-24 17:00:27 | mark.dickinson | set | messages:
+ msg80464 versions:
- Python 2.7 |
2009-01-16 10:35:04 | mark.dickinson | set | files:
+ issue1672332.patch keywords:
+ patch messages:
+ msg79949 |
2009-01-05 15:19:00 | mark.dickinson | set | messages:
+ msg79154 title: cPickle can pickle, but cannot unpickle on 64bit machines -> cPickle cannot unpickle subnormal floats on some machines |
2009-01-05 14:07:15 | mark.dickinson | set | messages:
+ msg79150 |
2009-01-05 12:22:10 | mark.dickinson | set | assignee: mark.dickinson versions:
+ Python 2.6, Python 3.0, Python 3.1, Python 2.7 type: behavior messages:
+ msg79134 nosy:
+ mark.dickinson |
2007-03-02 11:10:33 | oneg | create | |