Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Port _hashlib to OpenSSL 3.0.0 #84659

Closed
tiran opened this issue May 3, 2020 · 18 comments
Closed

Port _hashlib to OpenSSL 3.0.0 #84659

tiran opened this issue May 3, 2020 · 18 comments
Assignees
@tiran
Labels
3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes topic-SSL type-feature A feature request or enhancement

Comments

@tiran
Copy link
Member

tiran commented May 3, 2020

BPO 40479
Nosy @gpshead, @tiran, @ZackerySpytz, @miss-islington
PRs
  • bpo-40479: Port hashlib to OpenSSL 3.0.0 #19878
  • bpo-40479: Fix hashlib issue with OpenSSL 3.0.0 #20107
  • bpo-40479: Test with latest OpenSSL versions #20108
  • [3.8] bpo-40479: Test with latest OpenSSL versions (GH-20108) #20111
  • [3.7] bpo-40479: Test with latest OpenSSL versions (GH-20108) #20112
  • bpo-40479: Fix hashlib's usedforsecurity for OpenSSL 3.0.0 (GH-30455) #30455
  • [3.10] bpo-40479: Fix hashlib's usedforsecurity for OpenSSL 3.0.0 (GH-30455) #30573
  • [3.9] bpo-40479: Fix hashlib's usedforsecurity for OpenSSL 3.0.0 (GH-30455) (GH-30574) #30574
  • bpo-40479: Fix typo, flag must be set for OpenSSL < 3.0.0 (GH-30584) #30584
  • [3.10] bpo-40479: Fix typo, flag must be set for OpenSSL < 3.0.0 (GH-30584) #30585
  • [3.9] bpo-40479: Fix typo, flag must be set for OpenSSL < 3.0.0 (GH-30584) #30586
  • bpo-40479: Fix undefined behavior in Modules/_hashopenssl.c #31153
  • [3.10] bpo-40479: Fix undefined behavior in Modules/_hashopenssl.c (GH-31153) #31183
  • [3.9] bpo-40479: Fix undefined behavior in Modules/_hashopenssl.c (GH-31153) #31184
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2020-05-03.09:23:23.143>
labels = ['expert-SSL', '3.8', 'type-feature', '3.7', '3.9']
title = 'Port _hashlib to OpenSSL 3.0.0'
updated_at = <Date 2022-02-07.08:08:13.986>
user = 'https://github.com/tiran'

    bugs.python.org fields:

    activity = <Date 2022-02-07.08:08:13.986>
actor = 'miss-islington'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['SSL']
creation = <Date 2020-05-03.09:23:23.143>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 40479
keywords = ['patch']
message_count = 16.0
messages = ['367962', '368957', '368958', '368961', '368962', '410097', '410471', '410479', '410481', '410523', '410525', '410526', '412614', '412715', '412716', '412717']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'christian.heimes', 'ZackerySpytz', 'miss-islington']
pr_nums = ['19878', '20107', '20108', '20111', '20112', '30455', '30573', '30574', '30584', '30585', '30586', '31153', '31183', '31184']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40479'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']
    @tiran
    Copy link
    Member Author

    tiran commented May 3, 2020

    OpenSSL 3.0.0-alpha1 was releases about a week ago. OpenSSL 1.1.x APIs are still functional. However some APIs have been deprecated and FIPS (usedforsecurity flag) is no longer functional.

    • One shot HMAC() is deprecated and should be replaced with EVP_MAC API calls
    • ERR_func_error_string() is deprecated
    • OpenSSL has introduced a new concept of crypto providers (OSSL_PROVIDER), library context (OPENSSL_CTX) and additional flags. A new function EVP_MD_fetch() has been introduced.
    • FIPS support has been rewritten and is now shipped with OpenSSL 3.0.0. EVP_MD_CTX_FLAG_NON_FIPS_ALLOW is no longer supported. FIPS state is no longer part of EVP_MD_CTX but of EVP_MD.

    See https://wiki.openssl.org/index.php/OpenSSL_3.0

    @tiran tiran added 3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes labels May 3, 2020
    @tiran tiran self-assigned this May 3, 2020
    @tiran tiran added topic-SSL type-feature A feature request or enhancement labels May 3, 2020
    @miss-islington
    Copy link
    Contributor

    New changeset 16d4e6f by Christian Heimes in branch 'master':
    bpo-40479: Fix hashlib issue with OpenSSL 3.0.0 (GH-20107)
    16d4e6f

    @miss-islington
    Copy link
    Contributor

    New changeset 62d618c by Christian Heimes in branch 'master':
    bpo-40479: Test with latest OpenSSL versions (GH-20108)
    62d618c

    @miss-islington
    Copy link
    Contributor

    New changeset 5e6b491 by Miss Islington (bot) in branch '3.7':
    bpo-40479: Test with latest OpenSSL versions (GH-20108)
    5e6b491

    @miss-islington
    Copy link
    Contributor

    New changeset 5a06cf0 by Miss Islington (bot) in branch '3.8':
    bpo-40479: Test with latest OpenSSL versions (GH-20108)
    5a06cf0

    @tiran
    Copy link
    Member Author

    tiran commented Jan 8, 2022

    Simple benchmark with openssl-3.0.0-5.el9.x86_64

    With custom EVP_MD cache:
    $ ./python -m timeit -s "from hashlib import md5" "md5(b'12345678', usedforsecurity=False).digest()"500000 loops, best of 5: 520 nsec per loop
    $ ./python -m timeit -s "from hashlib import sha512" "sha512(b'12345678', usedforsecurity=False).digest()"
    500000 loops, best of 5: 730 nsec per loop

    Without EVP_MD cache:
    $ ./python -m timeit -s "from hashlib import md5" "md5(b'', usedforsecurity=False).digest()"
    500000 loops, best of 5: 807 nsec per loop
    $ ./python -m timeit -s "from hashlib import sha512" "sha512(b'12345678', usedforsecurity=False).digest()"
    200000 loops, best of 5: 1.03 usec per loop

    @tiran
    Copy link
    Member Author

    tiran commented Jan 13, 2022

    New changeset 443b308 by Christian Heimes in branch 'main':
    bpo-40479: Fix hashlib's usedforsecurity for OpenSSL 3.0.0 (GH-30455)
    443b308

    @tiran
    Copy link
    Member Author

    tiran commented Jan 13, 2022

    Something went wrong with Miss Islington.
    3ce6945 / #74758 is backport to 3.10.

    @tiran
    Copy link
    Member Author

    tiran commented Jan 13, 2022

    New changeset 4ddd5da by Christian Heimes in branch '3.9':
    [3.9] bpo-40479: Fix hashlib's usedforsecurity for OpenSSL 3.0.0 (GH-30455) (GH-30574)
    4ddd5da

    @tiran
    Copy link
    Member Author

    tiran commented Jan 13, 2022

    New changeset 276c234 by Christian Heimes in branch 'main':
    bpo-40479: Fix typo, flag must be set for OpenSSL < 3.0.0 (GH-30584)
    276c234

    @miss-islington
    Copy link
    Contributor

    New changeset 47422a8 by Miss Islington (bot) in branch '3.10':
    [3.10] bpo-40479: Fix typo, flag must be set for OpenSSL < 3.0.0 (GH-30584) (GH-30585)
    47422a8

    @miss-islington
    Copy link
    Contributor

    New changeset 537f16a by Miss Islington (bot) in branch '3.9':
    bpo-40479: Fix typo, flag must be set for OpenSSL < 3.0.0 (GH-30584)
    537f16a

    @ZackerySpytz
    Copy link
    Mannequin

    ZackerySpytz mannequin commented Feb 6, 2022

    There is a missing call to va_end() in 443b308 (which is undefined behavior). PR 31153 fixes this.

    @miss-islington
    Copy link
    Contributor

    New changeset 59e004a by Zackery Spytz in branch 'main':
    bpo-40479: Fix undefined behavior in Modules/_hashopenssl.c (GH-31153)
    59e004a

    @miss-islington
    Copy link
    Contributor

    New changeset 3ceff99 by Miss Islington (bot) in branch '3.10':
    bpo-40479: Fix undefined behavior in Modules/_hashopenssl.c (GH-31153)
    3ceff99

    @miss-islington
    Copy link
    Contributor

    New changeset 0892a0e by Miss Islington (bot) in branch '3.9':
    bpo-40479: Fix undefined behavior in Modules/_hashopenssl.c (GH-31153)
    0892a0e

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @kumaraditya303
    Copy link
    Contributor

    @tiran Can this be closed now ?

    @DavidVadnais DavidVadnais mentioned this issue Jan 5, 2023
    Closed
    @arhadthedev arhadthedev mentioned this issue Apr 22, 2023
    Closed
    @gpshead
    Copy link
    Member

    gpshead commented May 20, 2023

    closing as the bulk of this appears done but it isn't clear what 100% is. If there are remaining deprecated APIs in use we can open individual issues for those and clean them up as needed going forwards.

    @gpshead gpshead closed this as completed May 20, 2023
    @Bo98 Bo98 mentioned this issue Oct 8, 2023
    Merged
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @tiran tiran

    Labels
    3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes topic-SSL type-feature A feature request or enhancement
    Projects
    OpenSSL 3
    Status: Done
    Milestone
    No milestone
    Development

    No branches or pull requests
    4 participants
    @gpshead @tiran @miss-islington @kumaraditya303