Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

binascii.crc32 is not 64-bit clean when USE_ZLIB_CRC32 #82437

Closed
zmk mannequin opened this issue Sep 23, 2019 · 10 comments
Closed

binascii.crc32 is not 64-bit clean when USE_ZLIB_CRC32 #82437

zmk mannequin opened this issue Sep 23, 2019 · 10 comments
Assignees
@gpshead
Labels
3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@zmk
Copy link
Mannequin

zmk mannequin commented Sep 23, 2019

BPO 38256
Nosy @gpshead, @Savier
PRs
  • bpo-38256: Fix binascii.crc32() when inputs are 4+GiB #32000
  • [3.10] bpo-38256: Fix binascii.crc32 large input. #32013
  • [3.9] bpo-38256: Fix binascii.crc32 large input. #32015
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/gpshead'
closed_at = <Date 2022-03-21.06:37:04.336>
created_at = <Date 2019-09-23.07:48:20.996>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.11', '3.7', 'library']
title = 'binascii.crc32 is not 64-bit clean when USE_ZLIB_CRC32'
updated_at = <Date 2022-03-21.06:37:04.335>
user = 'https://bugs.python.org/zmk'

    bugs.python.org fields:

    activity = <Date 2022-03-21.06:37:04.335>
actor = 'gregory.p.smith'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2022-03-21.06:37:04.336>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2019-09-23.07:48:20.996>
creator = 'zmk'
dependencies = []
files = []
hgrepos = []
issue_num = 38256
keywords = ['patch']
message_count = 10.0
messages = ['352992', '352993', '352994', '352995', '415577', '415578', '415628', '415636', '415667', '415668']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'nadeem.vawda', 'zmk', 'iamsav']
pr_nums = ['32000', '32013', '32015']
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38256'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']
    @zmk
    Copy link
    Mannequin Author

    zmk mannequin commented Sep 23, 2019

    Following code:
    ----------------------------

    import binascii, zlib
bigdata=memoryview(bytearray((1<<32) + 100))

print(binascii.crc32(bigdata))
crc = binascii.crc32(bigdata[:1000])
crc = binascii.crc32(bigdata[1000:], crc)
print(crc)

print(zlib.crc32(bigdata))
crc = zlib.crc32(bigdata[:1000])
crc = zlib.crc32(bigdata[1000:], crc)
print(crc)

    Outputs with python3.7

    $ python3.7 x.py 
2575877834
2838121701
2838121701
2838121701

    @zmk zmk mannequin added 3.7 (EOL) end of life stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Sep 23, 2019
    @Savier
    Copy link
    Mannequin

    Savier mannequin commented Sep 23, 2019

    I've got
    Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)]
    2838121701
    2838121701
    2838121701
    2838121701

    @zmk
    Copy link
    Mannequin Author

    zmk mannequin commented Sep 23, 2019

    Looking at the code, the bug is dependant on USE_ZLIB_CRC32 define and it's unfixed in master. Cause is zlib crc32 that takes length as integer, zlibmodule has workaround, binascii has not.

    @zmk
    Copy link
    Mannequin Author

    zmk mannequin commented Sep 23, 2019

    Found zlibmodule fix: https://bugs.python.org/issue10276

    @zmk zmk mannequin added 3.8 only security fixes 3.9 only security fixes labels Sep 23, 2019
    @gpshead
    Copy link
    Member

    gpshead commented Mar 19, 2022 

    $ python3.8
Python 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii, zlib
>>> bigdata=memoryview(bytearray((1<<32) + 100))
>>> 
>>> print(binascii.crc32(bigdata))
2838121701
>>> crc = binascii.crc32(bigdata[:1000])
>>> crc = binascii.crc32(bigdata[1000:], crc)
>>> print(crc)
2838121701
>>> 
>>> print(zlib.crc32(bigdata))
2838121701
>>> crc = zlib.crc32(bigdata[:1000])
>>> crc = zlib.crc32(bigdata[1000:], crc)
>>> print(crc)
2838121701

    @gpshead gpshead removed 3.8 only security fixes 3.9 only security fixes labels Mar 19, 2022
    @gpshead gpshead closed this as completed Mar 19, 2022
    @gpshead
    Copy link
    Member

    gpshead commented Mar 19, 2022

    it depends on the build. USE_ZLIB_CRC32 causes it due to zlib's 32-bitness as noted my marko.

    $ ./python 
Python 3.11.0a6+ (heads/main-dirty:b3f2d4c8ba, Mar 19 2022, 15:32:04) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import binascii, zlib
>>> bigdata=memoryview(bytearray((1<<32) + 100))
>>> 
>>> print(binascii.crc32(bigdata))
2575877834
>>> crc = binascii.crc32(bigdata[:1000])
>>> crc = binascii.crc32(bigdata[1000:], crc)
>>> print(crc)
2838121701
>>> 
>>> print(zlib.crc32(bigdata))
2838121701
>>> crc = zlib.crc32(bigdata[:1000])
>>> crc = zlib.crc32(bigdata[1000:], crc)
>>> print(crc)
2838121701
>>>

    @gpshead gpshead added 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes labels Mar 19, 2022
    @gpshead gpshead reopened this Mar 19, 2022
    @gpshead gpshead changed the title binascii.crc32 is not 64-bit clean binascii.crc32 is not 64-bit clean when USE_ZLIB_CRC32 Mar 19, 2022
    @gpshead gpshead self-assigned this Mar 19, 2022
    @gpshead
    Copy link
    Member

    gpshead commented Mar 20, 2022

    New changeset 9d1c4d6 by Gregory P. Smith in branch 'main':
    bpo-38256: Fix binascii.crc32() when inputs are 4+GiB (GH-32000)
    9d1c4d6

    @gpshead
    Copy link
    Member

    gpshead commented Mar 20, 2022

    New changeset 4c989e1 by Gregory P. Smith in branch '3.10':
    [3.10] bpo-38256: Fix binascii.crc32 large input. (GH-32000) (GH-32013)
    4c989e1

    @gpshead
    Copy link
    Member

    gpshead commented Mar 21, 2022

    New changeset 58a7e13 by Gregory P. Smith in branch '3.9':
    bpo-38256: Fix binascii.crc32 large input. (GH-32000) (GH-32013) (GH-32015)
    58a7e13

    @gpshead
    Copy link
    Member

    gpshead commented Mar 21, 2022

    3.8 and older are in security fix only mode. 3.9+ have been fixed. realistically this was a rare issue with multiple workarounds.

    The 3.9 and 3.10 fixes were strictly bugfix only. The 3.11 fix included some performance improvements.

    @gpshead gpshead closed this as completed Mar 21, 2022
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @ned-deily ned-deily mentioned this issue Jun 22, 2023
    Closed
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @gpshead gpshead

    Labels
    3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @gpshead