Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prefer libb2 over vendored copy of blake2 #91251

Closed
tiran opened this issue Mar 22, 2022 · 6 comments
Closed

Prefer libb2 over vendored copy of blake2 #91251

tiran opened this issue Mar 22, 2022 · 6 comments
Labels
3.11 only security fixes extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error

Comments

@tiran
Copy link
Member

tiran commented Mar 22, 2022

BPO 47095
Nosy @gpshead, @tiran
PRs
  • bpo-47095: Use libb2 to provide blake2 implementation (GH-32059) #32059
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2022-03-26.20:05:29.265>
created_at = <Date 2022-03-22.19:51:17.380>
labels = ['extension-modules', 'type-bug', '3.11']
title = 'Prefer libb2 over vendored copy of blake2'
updated_at = <Date 2022-03-26.20:05:29.265>
user = 'https://github.com/tiran'

    bugs.python.org fields:

    activity = <Date 2022-03-26.20:05:29.265>
actor = 'christian.heimes'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-26.20:05:29.265>
closer = 'christian.heimes'
components = ['Extension Modules']
creation = <Date 2022-03-22.19:51:17.380>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 47095
keywords = ['patch']
message_count = 3.0
messages = ['415807', '415833', '416085']
nosy_count = 2.0
nosy_names = ['gregory.p.smith', 'christian.heimes']
pr_nums = ['32059']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue47095'
versions = ['Python 3.11']
    @tiran
    Copy link
    Member Author

    tiran commented Mar 22, 2022

    Python's blake2 implementation provides hashing, MAC (key, salt, personalization), variable length output, and tree hashing [1]. All features except for tree hashing are provided by OpenSSL 3.0.0 and newer [2]. It is unlikely that OpenSSL will get tree hashing any time soon, if all. [3]

    I would like to remove our vendored copy of blake2 eventually and just rely on OpenSSL. Therefore I propose to deprecate tree hashing feature so we can drop it in Python 3.13. The tree hashing parameters are: fanout, depth, leaf_size, node_offset, node_depth, inner_size, last_node

    Note: OpenSSL 3.0 might impose additional restrictions on the parameter. It might be possible that OpenSSL does not support salt and personalization (OSSL_MAC_PARAM_CUSTOM) without a MAC key.

    Alternatively we could replace our copy of blake2 and depend on libb2 from https://blake2.net/. libb2 is available in Fedora.

    [1] https://docs.python.org/3/library/hashlib.html#hashlib.blake2b
    [2] https://www.openssl.org/docs/manmaster/man7/EVP_MAC-BLAKE2.html
    [3] openssl/openssl#980

    @tiran tiran added 3.11 only security fixes extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error labels Mar 22, 2022
    @gpshead
    Copy link
    Member

    gpshead commented Mar 22, 2022

    In the short term we should prefer libb2 linkage when available.

    As for deprecation, it'd be useful to research how often the options going away are used in code in PyPI packages and in Github repos to understand the deprecation impact.

    The PyPI landscape for blake2 modules is not great because we've had it in hashlib for a while. One of those, or a new one, would need to be created by someone who needs the non openssl features.

    ultimate goal: simplify what's in Modules/_blake2/impl/ if it cannot be removed. Use an external library for the implementation when possible (and in all our binary releases. Those are better maintained to take advantage of performance or hw features over time.

    @tiran tiran changed the title Deprecate blake2's tree hashing feature Prefer libb2 over vendored copy of blake2 Mar 23, 2022
    @tiran tiran changed the title Deprecate blake2's tree hashing feature Prefer libb2 over vendored copy of blake2 Mar 23, 2022
    @tiran
    Copy link
    Member Author

    tiran commented Mar 26, 2022

    New changeset b16b6bb by Christian Heimes in branch 'main':
    bpo-47095: Use libb2 to provide blake2 implementation (GH-32059)
    b16b6bb

    @tiran tiran closed this as completed Mar 26, 2022
    @tiran tiran closed this as completed Mar 26, 2022
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @zooba
    Copy link
    Member

    zooba commented May 13, 2022

    Is there a configure option to prefer the vendored copy? Or static linking? (This change has broken some of my semi-portable builds, because libb2 isn't installed by default but it gets pulled in by the build tools, so it's on the build machines.)

    @tiran
    Copy link
    Member Author

    tiran commented May 13, 2022

    Untested: You should be able to statically link with libb2 using something like LIBB2_LIBS="-L/path/to/libdir -l:libb2.a -Wl,--exclude-libs,libb2.a

    @zooba
    Copy link
    Member

    zooba commented May 13, 2022

    Thanks! I used LIBB2_LIBS="-L$(pkg-config libb2 --variable=libdir) -l:libb2.a -Wl,--exclude-libs,libb2.a" and it's worked just fine (well, with one targeted test, going to be another hour or so before I can do a "real" test).

    DaanDeMeyer pushed a commit to DaanDeMeyer/python-rpm that referenced this issue Nov 17, 2022
    @hroncok
    Build Python with the optimized Blake2 library libb2
    17d458f 
    See python/cpython#91251

> Distributors are encouraged to build Python with the optimized Blake2
> library libb2.
    @gpshead gpshead mentioned this issue Apr 19, 2023
    Open
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.11 only security fixes extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @gpshead @tiran @zooba