Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hashlib.algorithms_available lists algorithms that are not available in OpenSSL 3.0 default provider #91257

Open
tiran opened this issue Mar 23, 2022 · 8 comments
Labels
3.9 only security fixes 3.10 only security fixes 3.11 only security fixes extension-modules C modules in the Modules dir stdlib Python modules in the Lib dir topic-SSL type-bug An unexpected behavior, bug, or error

Comments

@tiran
Copy link
Member

tiran commented Mar 23, 2022

BPO 47101
Nosy @tiran, @miss-islington
PRs
  • bpo-47101: list only activated algorithms in hashlib.algorithms_available #32076
  • [3.9] bpo-47101: list only activated algorithms in hashlib.algorithms_available (GH-32076) #32084
  • [3.10] bpo-47101: list only activated algorithms in hashlib.algorithms_available (GH-32076) (GH-32085) #32085
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2022-03-23.14:28:49.732>
    labels = ['type-bug', '3.9', '3.10', '3.11', 'extension-modules', 'library']
    title = 'hashlib.algorithms_available lists algorithms that are not available in OpenSSL 3.0 default provider'
    updated_at = <Date 2022-03-23.21:15:34.062>
    user = 'https://github.com/tiran'

    bugs.python.org fields:

    activity = <Date 2022-03-23.21:15:34.062>
    actor = 'christian.heimes'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Extension Modules', 'Library (Lib)']
    creation = <Date 2022-03-23.14:28:49.732>
    creator = 'christian.heimes'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 47101
    keywords = ['patch']
    message_count = 6.0
    messages = ['415877', '415878', '415880', '415907', '415909', '415912']
    nosy_count = 2.0
    nosy_names = ['christian.heimes', 'miss-islington']
    pr_nums = ['32076', '32084', '32085']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue47101'
    versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    @tiran
    Copy link
    Member Author

    tiran commented Mar 23, 2022

    Hubert Kario wrote in https://bugzilla.redhat.com/show_bug.cgi?id=2054702

    Description of problem:
    The hashlib.algorithms_available set includes algorithms like ripemd160 and whirlpool, those algorithms are not usable unless openssl legacy provider is loaded. Since it's not loaded, and the hashlib module won't load it, any attempt to use them fails.

    Version-Release number of selected component (if applicable):
    python3-3.9.10-1.el9.x86_64
    openssl-3.0.1-5.el9.x86_64

    How reproducible:
    always

    Steps to Reproduce:
    0. start python3

    1. from hashlib import algorithms_available
    2. algorithms_available
    3. import hashlib
    4. a = {(name, hashlib.new(name).digest_size) for name in algorithms_available}

    Actual results:
    {'sha3_384', 'blake2s', 'sha384', 'sha512_224', 'md5', 'sha3_512', 'md5-sha1', 'sha3_256', 'shake_128', 'sm3', 'sha256', 'sha512', 'sha1', 'shake_256', 'blake2b', 'whirlpool', 'sha512_256', 'sha3_224', 'sha224', 'ripemd160', 'md4'}

    Traceback (most recent call last):
      File "/usr/lib64/python3.9/hashlib.py", line 164, in __hash_new
        return _hashlib.new(name, data, **kwargs)
    ValueError: [digital envelope routines] unsupported
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "<stdin>", line 1, in <setcomp>
      File "/usr/lib64/python3.9/hashlib.py", line 170, in __hash_new
        return __get_builtin_constructor(name)(data)
      File "/usr/lib64/python3.9/hashlib.py", line 127, in __get_builtin_constructor
        raise ValueError('unsupported hash type ' + name)
    ValueError: unsupported hash type whirlpool

    Expected results:
    {'sha3_384', 'blake2s', 'sha384', 'sha512_224', 'md5', 'sha3_512', 'md5-sha1', 'sha3_256', 'shake_128', 'sm3', 'sha256', 'sha512', 'sha1', 'shake_256', 'blake2b', 'sha512_256', 'sha3_224', 'sha224'}

    {('blake2b', 64), ('sha512', 64), ('md5-sha1', 36), ('sha3_512', 64), ('md5', 16), ('sha224', 28), ('shake_128', 0), ('sm3', 32), ('blake2s', 32), ('sha1', 20), ('shake_256', 0), ('sha512_256', 32), ('sha3_224', 28), ('sha3_256', 32), ('sha3_384', 48), ('sha384', 48), ('sha256', 32), ('sha512_224', 28)}

    Additional info:
    If the legacy provider is loaded, then the algorithms should be listed and should work.

    It may be caused by Python using the deprecated EVP_MD_do_all() method instead of the EVP_MD_do_all_provided() method

    @tiran tiran added 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes extension-modules C modules in the Modules dir stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Mar 23, 2022
    @tiran
    Copy link
    Member Author

    tiran commented Mar 23, 2022

    Hubert's suggested solution EVP_MD_do_all_provided() worked almost straight forward. The function signature is a bit different and I got "undefined" in the result set. Filtering out NID_undef got right of it.

    @tiran
    Copy link
    Member Author

    tiran commented Mar 23, 2022

    $ ./python Tools/ssl/multissltests.py --openssl 3.0.2 --steps modules
    $ ./python -c "import hashlib; print(hashlib.algorithms_available)"
    {'blake2b', 'sha512', 'sm3', 'shake_128', 'md5', 'sha3_256', 'sha224', 'sha512_224', 'sha3_384', 'sha384', 'md5-sha1', 'sha3_224', 'shake_256', 'sha3_512', 'sha512_256', 'sha1', 'sha256', 'blake2s'}

    @miss-islington
    Copy link
    Contributor

    New changeset 48e2010 by Christian Heimes in branch 'main':
    bpo-47101: list only activated algorithms in hashlib.algorithms_available (GH-32076)
    48e2010

    @miss-islington
    Copy link
    Contributor

    New changeset ec3589f by Miss Islington (bot) in branch '3.9':
    bpo-47101: list only activated algorithms in hashlib.algorithms_available (GH-32076)
    ec3589f

    @tiran
    Copy link
    Member Author

    tiran commented Mar 23, 2022

    New changeset 1b6acaa by Christian Heimes in branch '3.10':
    [3.10] bpo-47101: list only activated algorithms in hashlib.algorithms_available (GH-32076) (GH-32085)
    1b6acaa

    @mhsmith
    Copy link
    Member

    mhsmith commented Sep 5, 2022

    @tiran: It looks like this has been fixed: should the issue be closed?

    @h-vetinari
    Copy link

    It looks like this has been fixed: should the issue be closed?

    Hashlib still needs to load the legacy provider to achieve parity with 1.1.1 builds, but that's tracked in #92876.

    thesamesam added a commit to thesamesam/portage that referenced this issue Mar 20, 2023
    thesamesam added a commit to thesamesam/portage that referenced this issue Mar 20, 2023
    thesamesam added a commit to thesamesam/portage that referenced this issue Mar 20, 2023
    thesamesam added a commit to thesamesam/portage that referenced this issue Mar 21, 2023
    gentoo-bot pushed a commit to gentoo/portage that referenced this issue Mar 21, 2023
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes 3.10 only security fixes 3.11 only security fixes extension-modules C modules in the Modules dir stdlib Python modules in the Lib dir topic-SSL type-bug An unexpected behavior, bug, or error
    Projects
    Status: Todo
    Development

    No branches or pull requests

    5 participants