msg56130 - (view) |
Author: Armin Rigo (arigo) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2007-09-25 11:31 |
The functions zlib.crc32() and zlib.adler32() return a signed value
in the range(-2147483648, 2147483648) on 32-bit platforms, but an
unsigned value in the range(0, 4294967296) on 64-bit platforms. This
means that half the possible answers are numerically different on these
two kinds of platforms.
Ideally, this should be fixed by having them always return unsigned
numbers (their C return type is unsigned too). It's unclear if we can
do this without breaking existing code, though :-(
|
msg56146 - (view) |
Author: Guido van Rossum (gvanrossum) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2007-09-26 16:51 |
Since it's basically a magic cookie, not a meaningful numeric value, I'd
propose sticking with backwards compatibility and fixing the 64-bit
version to return a signed version.
return x - ((x & 0x80000000) <<1)
anyone?
|
msg58336 - (view) |
Author: Tim Lesher (tlesher) * |
Date: 2007-12-10 01:45 |
Both CRC-32 and ADLER32 are standards (described in ISO 3309 and RFC
1950 respectively); whatever fix implemented should make sure that the
output complies.
ISO 3309 isn't available online as far as I can see, but CRC-32
reference code is published in RFC 1952; RFC 1950 contains reference
code for ADLER32.
|
msg58344 - (view) |
Author: Armin Rigo (arigo) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2007-12-10 12:29 |
The C reference code in rfc1950 for Adler-32 and in rfc1952 for CRC-32
compute with and return "unsigned long" values. From this point of
view, returning negative values on 32-bit machines from CPython's zlib
module can be considered a bug. That only leaves open the question of
backward compatibility.
|
msg58362 - (view) |
Author: Guido van Rossum (gvanrossum) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2007-12-10 19:40 |
How about, in Python 2.6 we make the 64-bit version return a signed
value for better compatibility with the 32-bit version, and in Python
3.0 we make both versions return the signed value for better compliance
with the standard?
|
msg58375 - (view) |
Author: Armin Rigo (arigo) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2007-12-10 22:23 |
Obscure but reasonable. (I suspect you meant to say that py3k should
return the *unsigned* value for better compliance with the standard.)
|
msg58376 - (view) |
Author: Guido van Rossum (gvanrossum) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2007-12-10 22:25 |
> Obscure but reasonable. (I suspect you meant to say that py3k should
> return the *unsigned* value for better compliance with the standard.)
Yes. :)
|
msg63664 - (view) |
Author: Gregory P. Smith (gregory.p.smith) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2008-03-17 16:27 |
working on this now. foo = 'abcdefghijklmnop'
2.x 32-bit long: zlib.crc32(foo) returns -1808088941
2.x 64-bit long: zlib.crc32(foo) returns 2486878355
This is because PyInt_FromLong() happily fits the value into a signed
long internally to the integer object on 64-bit platforms. They are
both the same number if considered with & 0xffffffff.
I'm doing as guido suggests and leaving this slightly odd behavior for
2.x so that crc32 and adler32 always return an integer object. in 3.0
they'll always return an unsigned value.
|
msg63666 - (view) |
Author: Gregory P. Smith (gregory.p.smith) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2008-03-17 16:33 |
question: should I also make 64-bit 2.x return a signed value as well to
be consistent with 32bit python 2.x?
Consistency in case someone ever pickles the value and sends it to
another python instance of a different word length would be good...
|
msg63668 - (view) |
Author: Guido van Rossum (gvanrossum) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2008-03-17 16:35 |
Sure. (Though sending pickles to 3.0 would still cause problems.
Consumers of pickled checksums would do wise to *always* take the CRC
mod 2**32 before doing comparisons.)
|
msg63676 - (view) |
Author: Jesús Cea Avión (jcea) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2008-03-17 17:05 |
I store CRC in reed-solomon schema of mine. I compare with equality, so,
I think we should enforce "CRC(python 32 bits) == CRC(python 64 bits)".
I will need to touch my code in python 3.0, but that will be inevitable
anyway...
|
msg63756 - (view) |
Author: Gregory P. Smith (gregory.p.smith) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2008-03-17 20:25 |
fixed.
3.0 always returns unsigned.
2.6 always returns signed, 2**31...2**31-1 come back as negative integers.
trunk r61449
branches/py3k r61459
|
msg64416 - (view) |
Author: Michael Becker (mbecker) |
Date: 2008-03-24 14:58 |
In case it isn't obvious the work around for pre 3.0 to get the right
sum is something like:
x=zlib.adler32(str)
if x < 0:
x=(long(x) + 4294967296L) # 2^32, long may or may not be needed here
|
msg64420 - (view) |
Author: Gregory P. Smith (gregory.p.smith) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2008-03-24 16:55 |
The workaround I prefer to use is:
x = zlib.adler32(mystr) & 0xffffffffL
|
msg74380 - (view) |
Author: Facundo Batista (facundobatista) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2008-10-06 16:10 |
Let me reopen this, I think we have an issue with this fix.
The conclusion of this discussion so far is that in 3.0 the crc32 will
behave like the standard, which is a good thing (tm), but in 2.6 it will
not: it should return a signed integer. I agree with this outcome!
The documentation for 2.6, the commit message for the fix and what it's
said here states that: "2.6 always returns signed, 2**31...2**31-1 come
back as negative integers."
This is *not* actually happening:
>>> s = "*"*100000
>>> print zlib.crc32(s) # 2.6, 32 bits
-872059092
>>> print zlib.crc32(s) # 2.6, 64 bits
3422908204
The problem in the code is, IMHO, that the "32b rounding" is being
forced by assigning the result to an int (Modules/zlibmodule.c, line
929), but this "rounding" does not actually work for 64b (because the
int has 64 bit, and even as it's signed, the number stays big and positive).
Thank you!
|
msg74384 - (view) |
Author: Gregory P. Smith (gregory.p.smith) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2008-10-06 17:53 |
An int is 32-bits on all popular platforms. Anyways i'll double check
things. What platforms did you run your test on?
|
msg79408 - (view) |
Author: J. David Ibáñez (jdavid.ibp@gmail.com) |
Date: 2009-01-08 12:13 |
I believe I have hit this bug. With Python 2.6.1 in a Gentoo Linux
64 bits.
This code:
from zipfile import ZipFile
inzip = ZipFile('Document.odt')
outzip = ZipFile('out.zip', 'w')
for f in inzip.infolist():
if f.filename != 'content.xml':
print f.filename, '(CRC: %s)' % f.CRC
outzip.writestr(f, inzip.read(f.filename))
outzip.close()
Produces this output:
...
styles.xml (CRC: 3930581528)
test_odt.py:10: DeprecationWarning: 'l' format requires -2147483648 <=
number <= 2147483647
outzip.writestr(f, inzip.read(f.filename))
...
The CRC is not a 32bits signed, and then the module struct complains,
here:
zipfile.py:1098
self.fp.write(struct.pack("<lLL", zinfo.CRC, zinfo.compress_size,
Apparently 'struct.pack' expects 'zinfo.CRC' to be a 32 signed it,
which is not.
I can attach the 'Document.odt' file if you want.
|
msg79527 - (view) |
Author: Gregory P. Smith (gregory.p.smith) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2009-01-10 03:09 |
seems there are bugs with it not staying signed as it should on some
64bit platforms. i'll be looking into this shortly. its a good
candidate bug for 2.6.x and 3.0.x releases.
|
msg79528 - (view) |
Author: Gregory P. Smith (gregory.p.smith) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2009-01-10 03:10 |
err not 3.0.x, 3.0 is always unsigned like anyone sane would want. :)
|
msg85557 - (view) |
Author: Gregory P. Smith (gregory.p.smith) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2009-04-05 19:13 |
J. David Ibáñez - do you still happen to have that Document.odt laying
around?
|
msg85626 - (view) |
Author: J. David Ibáñez (jdavid.ibp@gmail.com) |
Date: 2009-04-06 09:42 |
There it is.
|
msg86192 - (view) |
Author: Gaëtan de Menten (gdementen) |
Date: 2009-04-20 09:13 |
Regarding the issue J. David Ibáñez has, I have a few comments to add:
- It's also present on 32bit.
- AFAICT:
* it's present in both 2.6 branch & trunk (as of 68886),
* it's a problem with line 1110 (in 2.6 branch), or line 1122 in
trunk, which should read "<LLL" instead of "<lLL",
* it is an omission from changeset 61591:
http://svn.python.org/view/python/trunk/Lib/zipfile.py?view=diff&r1=61590&r2=61591
which changed line 964 (at the time)
http://svn.python.org/view/python/trunk/Lib/zipfile.py?annotate=61591#l964
but not the corresponding one in writestr.
I'm not sure whether or not a separate bug report should be opened for
this issue.
|
msg89723 - (view) |
Author: Gregory P. Smith (gregory.p.smith) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2009-06-26 08:19 |
fix for J. David's issue submitted to trunk r73565 and py3k r73566 just in
time for the 3.1 release.
release30-maint r73567
release26-maint r73568
|
|
Date |
User |
Action |
Args |
2022-04-11 14:56:27 | admin | set | github: 45543 |
2009-06-26 08:20:10 | gregory.p.smith | set | resolution: fixed |
2009-06-26 08:19:28 | gregory.p.smith | set | status: open -> closed
messages:
+ msg89723 versions:
+ Python 3.0, Python 3.1 |
2009-04-20 09:13:46 | gdementen | set | nosy:
+ gdementen messages:
+ msg86192
|
2009-04-06 09:42:23 | jdavid.ibp@gmail.com | set | files:
+ Document.odt
messages:
+ msg85626 |
2009-04-05 19:13:36 | gregory.p.smith | set | messages:
+ msg85557 |
2009-01-10 03:10:01 | gregory.p.smith | set | messages:
+ msg79528 versions:
- Python 3.0 |
2009-01-10 03:09:17 | gregory.p.smith | set | keywords:
- easy messages:
+ msg79527 |
2009-01-10 03:06:59 | gregory.p.smith | link | issue4903 dependencies |
2009-01-08 12:13:47 | jdavid.ibp@gmail.com | set | nosy:
+ jdavid.ibp@gmail.com messages:
+ msg79408 |
2008-10-06 17:53:06 | gregory.p.smith | set | messages:
+ msg74384 |
2008-10-06 16:10:43 | facundobatista | set | status: closed -> open nosy:
+ facundobatista resolution: fixed -> (no value) messages:
+ msg74380 |
2008-03-24 16:55:18 | gregory.p.smith | set | messages:
+ msg64420 |
2008-03-24 14:58:55 | mbecker | set | nosy:
+ mbecker messages:
+ msg64416 |
2008-03-17 20:25:17 | gregory.p.smith | set | status: open -> closed resolution: fixed messages:
+ msg63756 |
2008-03-17 17:05:03 | jcea | set | messages:
+ msg63676 |
2008-03-17 16:35:36 | gvanrossum | set | messages:
+ msg63668 |
2008-03-17 16:33:13 | gregory.p.smith | set | messages:
+ msg63666 |
2008-03-17 16:27:11 | gregory.p.smith | set | messages:
+ msg63664 |
2008-01-23 08:34:23 | gregory.p.smith | set | priority: normal assignee: gregory.p.smith versions:
+ Python 2.6, Python 3.0 keywords:
+ 64bit, easy nosy:
+ gregory.p.smith |
2008-01-16 02:28:45 | jcea | set | nosy:
+ jcea |
2007-12-10 22:25:48 | gvanrossum | set | messages:
+ msg58376 |
2007-12-10 22:23:52 | arigo | set | messages:
+ msg58375 |
2007-12-10 19:40:40 | gvanrossum | set | messages:
+ msg58362 |
2007-12-10 12:29:55 | arigo | set | messages:
+ msg58344 |
2007-12-10 01:45:23 | tlesher | set | nosy:
+ tlesher messages:
+ msg58336 |
2007-09-26 16:51:11 | gvanrossum | set | nosy:
+ gvanrossum messages:
+ msg56146 |
2007-09-25 11:31:11 | arigo | create | |