classification
Title: Investigate replacing SHA3 code with OpenSSL
Type: behavior Stage: patch review
Components: Extension Modules Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: christian.heimes, cstratak, gregory.p.smith, miss-islington
Priority: normal Keywords: patch

Created on 2019-07-19 11:30 by christian.heimes, last changed 2020-05-23 19:39 by christian.heimes.

Pull Requests
URL Status Linked Edit
PR 16049 merged christian.heimes, 2019-09-12 13:42
PR 20154 merged christian.heimes, 2020-05-17 15:43
Messages (6)
msg348165 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-07-19 11:30
Recent OpenSSL comes with SHA3. Now that Python is going to drop support for old OpenSSL, we can consider to use OpenSSL's SHA3 and drop the reference implementation from Python.

For variable length SHAKE API, OpenSSL added EVP_MD_CTRL_XOF_LEN and EVP_DigestFinalXOF().
msg369066 - (view) Author: miss-islington (miss-islington) Date: 2020-05-16 20:27
New changeset d5b3f6b7f9fc74438009af63f1de01bd77be9385 by Christian Heimes in branch 'master':
bpo-37630: Use SHA3 and SHAKE XOF from OpenSSL (GH-16049)
https://github.com/python/cpython/commit/d5b3f6b7f9fc74438009af63f1de01bd77be9385
msg369067 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-05-16 20:40
I'll add a whatsnew later.
msg369138 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-05-17 16:32
New changeset 62ecd8a8f908282726d2f019c93efa1cf2e9e784 by Christian Heimes in branch 'master':
bpo-37630: Fix spelling shake128 -> shake_128 (GH-20154)
https://github.com/python/cpython/commit/62ecd8a8f908282726d2f019c93efa1cf2e9e784
msg369715 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-05-23 11:21
OpenSSL's SHA-3 implementation is a tiny bit faster than our builtin copy of SHA-3.

builtin SHA-3 with PGO

$ python3 -m timeit -s "from _sha3 import sha3_256; d = b'12345678' * 1000" "sha3_256(d)"
10000 loops, best of 5: 20.3 usec per loop

builtin SHA-3 without PGO

$ ./python -m timeit -s "from _sha3 import sha3_256; d = b'12345678' * 1000" "sha3_256(d)"
10000 loops, best of 5: 21.1 usec per loop

OpenSSL SHA-3

$ ./python -m timeit -s "from _hashlib import openssl_sha3_256 as sha3_256; d = b'12345678' * 1000" "sha3_256(d)"
20000 loops, best of 5: 19.1 usec per loop


OpenSSL's Blake2 implementation is also a tiny bit faster. (b.copy().update() because the _hashlib module doesn't have fast constructor yet)

$ python3 -m timeit -s "from _blake2 import blake2b; b = blake2b(); d = b'12345678' * 1000" "b.copy().update(d)"
50000 loops, best of 5: 9.67 usec per loop
$ python3 -m timeit -s "from _hashlib import new; b = new('blake2b512'); d = b'12345678' * 1000" "b.copy().update(d)"
50000 loops, best of 5: 8.87 usec per loop
msg369741 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-05-23 19:39
LibreSSL does neither include SHA3/SHAKE family nor Blake2. Feature requests have been open for 1.5 to almost four years. The first reply on each feature request don't come as a surprise to me...

https://github.com/libressl-portable/portable/issues/199
https://github.com/libressl-portable/portable/issues/455
History
Date User Action Args
2020-05-23 19:39:24christian.heimessetmessages: + msg369741
2020-05-23 11:21:50christian.heimessetmessages: + msg369715
2020-05-17 16:32:53christian.heimessetmessages: + msg369138
2020-05-17 15:43:46christian.heimessetpull_requests: + pull_request19457
2020-05-16 20:40:19christian.heimessetmessages: + msg369067
versions: - Python 3.7, Python 3.8
2020-05-16 20:27:10miss-islingtonsetnosy: + miss-islington
messages: + msg369066
2019-09-12 13:42:29christian.heimessetkeywords: + patch
stage: patch review
pull_requests: + pull_request15671
2019-07-22 12:00:23cstrataksetnosy: + cstratak
2019-07-19 11:30:56christian.heimescreate