Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

integer overflow in hashlib causes wrong results for cryptographic hash functions [was: mmap broken with large files on 64bit system] #47276

Closed
donut mannequin opened this issue Jun 2, 2008 · 8 comments
Closed

integer overflow in hashlib causes wrong results for cryptographic hash functions [was: mmap broken with large files on 64bit system] #47276

donut mannequin opened this issue Jun 2, 2008 · 8 comments
Labels
extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error

Comments

@donut
Copy link
Mannequin

donut mannequin commented Jun 2, 2008

BPO 3026
Nosy @loewis
Files
  • testbigfile.py: test script
  • large_digest_update.diff: patch against svn r64953
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2008-09-18.12:06:26.205>
created_at = <Date 2008-06-02.02:24:56.963>
labels = ['extension-modules', 'type-bug']
title = 'integer overflow in hashlib causes wrong results for cryptographic hash functions [was: mmap broken with large files on 64bit system]'
updated_at = <Date 2008-09-18.12:06:26.203>
user = 'https://bugs.python.org/donut'

    bugs.python.org fields:

    activity = <Date 2008-09-18.12:06:26.203>
actor = 'loewis'
assignee = 'none'
closed = True
closed_date = <Date 2008-09-18.12:06:26.205>
closer = 'loewis'
components = ['Extension Modules']
creation = <Date 2008-06-02.02:24:56.963>
creator = 'donut'
dependencies = []
files = ['10500', '10895']
hgrepos = []
issue_num = 3026
keywords = ['patch']
message_count = 8.0
messages = ['67623', '67624', '67701', '67775', '69642', '69664', '73373', '73375']
nosy_count = 3.0
nosy_names = ['loewis', 'donut', 'schmir']
pr_nums = []
priority = 'critical'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue3026'
versions = ['Python 2.6', 'Python 3.0']
    @donut
    Copy link
    Mannequin Author

    donut mannequin commented Jun 2, 2008

    mmap on large files on 64 bit platforms in python >=2.5 returns some
    sort of garbage. In 2.4 it would just throw an exception. Now I get
    something like this (script runs md5.md5 on mmap object, and then runs
    os.system md5sum for comparison):

    This is python2.5 from Ubuntu 8.04 AMD64
    /tmp$ python2.5 testbigfile.py
    python mmap md5: 1230552d39b7c1751f86bae5205ec0c8
    abe59e28c9a3f11b883f62c80a3833a5 *bigfile

    This is python svn as of 20080601, compiled the on same system.
    /tmp$ python2.6 testbigfile.py
    testbigfile.py:5: DeprecationWarning: the md5 module is deprecated; use
    hashlib instead
    import md5
    python mmap md5: 1230552d39b7c1751f86bae5205ec0c8
    abe59e28c9a3f11b883f62c80a3833a5 *bigfile

    Also note how the python md5 call returns immediately, not something you
    would expect when md5ing 4GB of data.

    @donut donut mannequin added extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error labels Jun 2, 2008
    @donut
    Copy link
    Mannequin Author

    donut mannequin commented Jun 2, 2008

    Actually, I just realized that this might be a problem with md5 module
    instead. Either way, something is busted.

    @schmir
    Copy link
    Mannequin

    schmir mannequin commented Jun 4, 2008

    I tested this with python 2.6 and can confirm the issue.
    The problem is that unsigned int isn't big enough to hold the size of
    the objects, but the size is downcasted to an unsigned int at several
    places in _hashopenssl.c. All of these occurences of Py_SAFE_DOWNCAST
    seem problematic to me (Py_SAFE_DOWNCAST(len, Py_ssize_t, unsigned int))

    @schmir
    Copy link
    Mannequin

    schmir mannequin commented Jun 6, 2008

    the same bug also occurs when computing the md5 of a string larger than
    2**32

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Jul 14, 2008

    So would anybody like to contribute a patch?

    @schmir
    Copy link
    Mannequin

    schmir mannequin commented Jul 14, 2008

    this patch adds a digest_update function.
    digest_update calls EVP_DigestUpdate(..) with chunks of 16 MB size and
    also checks for signals.
    I didn't write any tests (as they will most probably annoy many people
    cause they would need much memory).

    testbigfile.py however now works.

    @schmir schmir mannequin changed the title mmap broken with large files on 64bit system integer overflow in hashlib causes wrong results for cryptographic hash functions [was: mmap broken with large files on 64bit system] Aug 5, 2008
    @schmir
    Copy link
    Mannequin

    schmir mannequin commented Sep 18, 2008

    same issue in http://bugs.python.org/issue3886.
    it's sad that no one took a look at the patch...
    now, it should probably be closed...

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Sep 18, 2008

    Ok, closing. Thanks for the patch, anyway.

    @loewis loewis mannequin closed this as completed Sep 18, 2008
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    0 participants