classification
Title: 64/32-bit issue when unpickling random.Random
Type: Stage:
Components: Library (Lib) Versions: Python 2.5
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: loewis Nosy List: loewis, melicertes, pm67nz, sligocki
Priority: normal Keywords: patch

Created on 2007-05-29 17:08 by melicertes, last changed 2007-12-03 19:22 by loewis. This issue is now closed.

Files
File name Uploaded Description Edit
python_head_random.patch sligocki, 2007-09-17 10:13
Messages (7)
msg32146 - (view) Author: Charles (melicertes) Date: 2007-05-29 17:08
If an object containing a random.Random instance is pickled on a 64-bit platform, unpickling the result on a 32-bit platform can cause an OverflowError:

  File "/usr/local/lib/python2.4/random.py", line 140, in __setstate__
    self.setstate(state)
  File "/usr/local/lib/python2.4/random.py", line 125, in setstate
    super(Random, self).setstate(internalstate)
OverflowError: long int too large to convert to int

I wasn't able to find any other tickets referring specifically to this.  This is in Python 2.4.1 for sure, may be in later versions.
msg55938 - (view) Author: Shawn Ligocki (sligocki) Date: 2007-09-16 05:14
I've had this same problem with 2.5.1

Pickling random.getstate() on 64bit and then loading it back with
random.setstate() on 32bit does not work (crashes because of trying cast
64bit ints to 32bit).

The other way around is even worse. It loads but is in a faulty state,
where for example:
>>> random.randrange(1000)
49801980494

This seems like a bug to me.
msg55944 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2007-09-16 16:05
Would anybody of you like to work on a patch?
msg55959 - (view) Author: Shawn Ligocki (sligocki) Date: 2007-09-17 10:13
I've got a patch! The problem was that the state was being cast from a
C-type unsigned long to a long.

On 32-bit machines this makes large 32-bit longs negative.
On 64-bit machines this preserves the sign of 32-bit values (because
they are stored in 64-bit longs).

My patch returns the values with PyLong_FromUnsignedLong() instead of
PyInt_FromLong(), therefore there is no casting to long and both 32-bit
and 64-bit machines produce the same result.

I added code to read states from the old (buggy) version and decypher it
appropriately (from either 32-bit or 64-bit source!). In other words,
old pickles can now be opened on either architecture with the new patch.

This patch is taken from the svn head, but also works on Python 2.5.1 .

I haven't tested this patch fully on 64-bit machine yet. I'll let you
know when I have.

Cheers,
-Shawn
msg55989 - (view) Author: Shawn Ligocki (sligocki) Date: 2007-09-18 05:41
Yep, tested it on a 64-bit machine and 2 32-bit machines and back and
forth between them. It seems to resolve the problem.
msg57298 - (view) Author: Peter Maxwell (pm67nz) Date: 2007-11-09 07:54
For the record, and to prevent dilution of the count of times this bug has 
been encountered: this issue is a duplicate of issue1472695, which was 
later marked "won't fix" for no apparent reason. sligocki's patch is more 
thorough than mine was and I hope it has a better fate too.
msg58144 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2007-12-03 19:22
Thanks for the patch. Committed as r59295.

Because of the version change, the patch cannot be applied to 2.5.x
History
Date User Action Args
2007-12-03 19:22:07loewissetstatus: open -> closed
resolution: accepted
messages: + msg58144
2007-11-09 07:54:34pm67nzsetnosy: + pm67nz
messages: + msg57298
2007-09-23 08:32:57loewissetassignee: loewis
severity: normal -> major
2007-09-18 05:41:26sligockisetmessages: + msg55989
2007-09-17 21:47:45loewissetkeywords: + patch
2007-09-17 10:13:41sligockisetfiles: + python_head_random.patch
messages: + msg55959
2007-09-16 16:05:31loewissetnosy: + loewis
messages: + msg55944
2007-09-16 05:14:41sligockisetnosy: + sligocki
messages: + msg55938
2007-09-16 05:08:44sligockisetversions: + Python 2.5, - Python 2.4
2007-05-29 17:08:37melicertescreate