This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Add SHA-3 and SHAKE (Keccak) support
Type: enhancement Stage: resolved
Components: Extension Modules Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, asvetlov, bjornedstrom, christian.heimes, dstufft, ezio.melotti, gregory.p.smith, haakon, habnabit, jcea, larry, loewis, markk, mgorny, pitrou, python-dev, rhettinger, sbt, spatz, tim.peters, vstinner
Priority: normal Keywords: patch

Created on 2012-10-03 03:10 by christian.heimes, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
521e85a613bf.diff christian.heimes, 2012-10-05 21:07 review
remove_sha3.patch christian.heimes, 2014-01-02 20:52 review
SHA3-and-SHAKE-support-for-Python.patch christian.heimes, 2016-05-06 21:11 review
SHA3-and-SHAKE-support-for-Python-2.patch christian.heimes, 2016-06-02 18:51 review
SHA3-and-SHAKE-support-for-Python-3.patch christian.heimes, 2016-08-20 23:35 review
Messages (80)
msg171848 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-03 03:10
Today the latest crypto hash function was announced by NIST [1]. I suggest that we include the new hash algorithm in 3.4 once it lands in OpenSSL.

The Keccak site also has a reference implementation in C and Assembler [2]. It may take some effort to integrate the reference implementation as it contains several optimized backends for X86, X86_64, SIMD and various ARM platforms.

[1] http://www.nist.gov/itl/csd/sha-100212.cfm
[2] http://keccak.noekeon.org/
msg171882 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2012-10-03 13:28
We have MD5, SHA1, sha256, sha512 implemented, to use when openssl is not available. Can we do the same with sha-3?. I would suggest to adopt the reference implementation without extensive optimizations, since we will have them when openssl has them.

So we might implement SHA-3 now and integrate OpenSSL implementation later, when available. This is interesting, for instance, because many users of Python 3.4 will have a non "up to date" OpenSSL system library.
msg171898 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-03 17:06
I've done some experiments with the reference implementation and adopted code of sha1module.c for sha3: https://bitbucket.org/tiran/pykeccak

So far the code just compiles (64bit only) but doesn't work properly yet. I may need to move away from the NIST interface and use the sponge interface directly.
msg171913 - (view) Author: Björn Edström (bjornedstrom) Date: 2012-10-03 22:34
For what it's worth, I've built a working C-based sha3-module that is available here: https://github.com/bjornedstrom/python-sha3

Note that I've only tested this on Python 2, for Python 3 YMMV.

Best regards
Björn
msg171929 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-04 10:16
Hello Björn,

thanks for the information. Your package didn't turn up on Google when I started with my experiment. Perhaps it's too new?

Your code and mine have lots of similarities. I was amused when I saw that you had the same issue with the block size attribute. At first I set it to 200 (1600 / 8) but eventually I didn't implement it.

My code does everything in C with a separate constructor for each flavor of SHA-3. It's compatible to Python 2.6 to 3.4 and uses the optimized code for 32 and 64bit platforms.

Oh, and my code is now working properly. Feel free to review the module. I'll upload the test code later.
msg171963 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-04 15:31
Release 0.1 of pysha3 [1] is out. I've tweaked the C module to make it compatible with Python 2.6 to 3.4. The module and its tests run successfully under Linux and Windows. So far I've tested Linux X84_64 (2.7, 3.2, 3.3, 3.4), Windows X86 (2.6, 2.7, 3.2, 3.3) and Windows X86_64 (2.6, 2.7, 3.2, 3.3).

Please review Modules/sha3module.c and ignore all version specific #if blocks. For Python 3.4 I'm going to remove all blocks for Python < 3.3.

[1] http://pypi.python.org/pypi/pysha3/0.1
msg171964 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-10-04 15:32
> Please review Modules/sha3module.c

Can't you post a patch here?
msg171968 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-04 16:19
How about a sandbox repos?
msg171971 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-10-04 16:29
Good, you can click the "create patch" button when it's ready :)
msg171983 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-04 18:05
Antoine pointed out that the code contains C++ comments and exports a lot of functions. The latest patch has all // comments replaced, marks all functions and globals as static and #includes the C files directly.
msg171995 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-04 21:02
Please review the latest patch. 

I've included Gregory as he is the creator of hashlib.
msg172070 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-05 10:47
The hightlights of the next patch are

* release the GIL
* more test vectors
* remove bgr_endian.h
* move typedef UINT64 to sha3module
* declare more globals as static
msg172100 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-05 16:59
I've documented the optimization options of Keccak. The block also contains a summarization of my modifications of the reference code.

http://hg.python.org/sandbox/cheimes/file/57948df78dbd/Modules/_sha3/sha3module.c#l22
msg172144 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-05 21:10
New patch. I've removed the dependency on uint64 types. On platforms without a uint64 type the module is using the 32bit implementation with interleave tables.

By the way the SSE / SIMD instructions aren't useful. They are two to four times slower.
msg172152 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-10-05 21:40
don't worry about optimization settings in python itself for now.  the canonical optimized version will be in a future openssl version.  now that it has been declared the standard it will get a *lot* more attention in the next few years.

as it is, we _may_ want to replace this reference implementation with one from libtomcrypt in the future when it gets around to implementing it just so that the code for all of our bundled hash functions comes from the same place.
msg172157 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-10-06 00:36
New changeset 11c9a894680e by Christian Heimes in branch 'default':
Issue #16113: integrade SHA-3 (Keccak) patch from http://hg.python.org/sandbox/cheimes
http://hg.python.org/cpython/rev/11c9a894680e
msg172158 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-06 00:37
The code has landed in default. Let's see how the build bots like my patch and the reference implementation.
msg172313 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-10-07 16:15
_sha3 is not being built on Windows, so importing hashlib fails

>>> import hashlib
ERROR:root:code for hash sha3_224 was not found.
Traceback (most recent call last):
  File "C:\Repos\cpython-dirty\lib\hashlib.py", line 109, in __get_openssl_constructor
    f = getattr(_hashlib, 'openssl_' + name)
AttributeError: 'module' object has no attribute 'openssl_sha3_224'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Repos\cpython-dirty\lib\hashlib.py", line 154, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "C:\Repos\cpython-dirty\lib\hashlib.py", line 116, in __get_openssl_constructor
    return __get_builtin_constructor(name)
  File "C:\Repos\cpython-dirty\lib\hashlib.py", line 104, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha3_224
...
msg172314 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-07 16:16
I've pushed a fix about 5 minutes ago. The module wasn't compiled in debug builds due to an error in the project file. Please update your copy and try again.
msg172316 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-07 16:24
6cf6b8265e57 and 8172cc8bfa6d have fixed the issue on my VM. I didn't noticed the issue as I only tested hashlib with the release builds, not the debug builds. Sorry for that.
msg172319 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-10-07 16:43
> 6cf6b8265e57 and 8172cc8bfa6d have fixed the issue on my VM. I didn't 
> noticed the issue as I only tested hashlib with the release builds, not 
> the debug builds. Sorry for that.

Ah.  I did not even notice there was _sha3.vcxproj.

Is there any particular reason for not making it part of python3.dll like _sha1, _sha256, _sha512 are?  (I thought it was only modules with special link requirements that became separate DLLs.)
msg172324 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-10-07 17:14
The module is rather large (about 190 KB) because the optimized SHA-3 implementation isn't optimized for size. For this reason I like to keep the module out of the main binary for now.
msg183129 - (view) Author: (englabenny) Date: 2013-02-27 10:13
Please do not go forward until NIST publishes its SHA-3 specification document. We don't know yet what parameters they will finally choose when making Keccak SHA-3.
msg190303 - (view) Author: (englabenny) Date: 2013-05-29 12:49
NIST has published a tentative schedule for SHA-3 standardization. They expect to publish in the second quarter of 2014.

See http://csrc.nist.gov/groups/ST/hash/sha-3/timeline_fips.html  

and http://csrc.nist.gov/groups/ST/hash/sha-3/sha-3_standardization.html
msg191931 - (view) Author: Aaron Gallagher (habnabit) Date: 2013-06-27 05:56
As long as the reference Keccak code is going to live in the python stdlib anyway, I would /greatly/ appreciate it if the Keccak sponge function was directly exposed instead of just the fixed parameters used for SHA-3. 

A Keccak sponge can have a much wider range of rates/capacities, and after absorption can have any number of bytes squeezed out. The ability to get an unbounded number of bytes out is very useful and I've written some code that uses that behavior. I ended up having to write my own Keccak python library since none of the other SHA-3 libraries exposed this either.
msg191940 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-06-27 10:32
Hi Aaron,

it's a tempting idea but I have to decline. The API is deliberately limited to the NIST interface. Once OpenSSL gains SHA-3 support we are going to use it in favor for the reference implementation. I don't expect OpenSSL to provide the full sponge API.

I also like to keep all options open so I can switch to a different and perhaps smaller implementation in the future. The reference implementation is huge and the binary is more than 400 KB. For comparison the SHA-2 384 + 512 module's binary is just about 60 KB on a 64bit Linux system.

Once a a new API has been introduced it's going to take at least two minor Python release and about four to five years to remove it.

But I could add a more flexible interface to Keccak's sponge to my standalone sha3 module https://pypi.python.org/pypi/pysha3 ...
msg191971 - (view) Author: Aaron Gallagher (habnabit) Date: 2013-06-27 23:41
https://pypi.python.org/pypi/cykeccak/ is what I've written to do this, for reference. 

Honestly I hope that the Keccak sponge is directly exposed in openssl (or any other SHA-3 implementation) because of its utility beyond SHA-3. If the source of some other implementation is going to be bundled with python anyway, it shouldn't be difficult to expose the sponge bits.
msg201078 - (view) Author: (haakon) Date: 2013-10-23 22:48
Please make sure that the currently committed code is not released as part of Python 3.4. SHA-3 is not standardised yet, and NIST has said that they intend to make some changes to the Keccak SHA-3 submission before standardisation as a FIPS.

The links englabenny posted have a good overview of the SHA-3 timeline and the proposed changes.

It would be very confusing if hashlib in Python 3.4 came with a "sha3" that was incompatible with the final standard.
msg201079 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2013-10-23 22:54
Victor: a "new feature" is not a "release blocker".
msg201080 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-23 22:59
I'm tracking the SHA-3 progress closely. I'm prepared to pull the plug if there is any doubt about the final version of SHA-3 before beta 2 is released on Jan 5th.

Larry:
I have marked this new feature as release blocker because I may have to remove it and reschedule its addition for 3.5. I'd like to remove it after you have branched off the 3.4 branch.
msg201081 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-10-23 23:01
Larry:
I have marked this new feature as release blocker because I may have to remove it and reschedule its addition for 3.5. I'd like to remove it after you have branched off the 3.4 branch.
msg201082 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2013-10-23 23:15
"release blocker" means "the release cannot go out until this issue is solved".  Adding SHA-3, while nice, is simply not something I am going to hold up 3.4 for, full stop.

Please stop marking this issue as a "release blocker".
msg201083 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2013-10-23 23:19
Here "release blocker" would mean that if SHA-3 specification is not finished, then "the release cannot go out until SHA-3 is deleted".
msg201084 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2013-10-23 23:23
You guys are making me cranky.  Please stop adding me to this issue.
msg201085 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2013-10-23 23:40
@Larry, you seem to be misreading this.  They're not saying 3.4 can't be released until this feature is added.  It's _already_ been added.  They're saying 3.4 possibly can't be released until this feature is _removed_ - but whether it needs to be removed is outside of our control, and is not yet known.

> "release blocker" means "the release cannot go out until this
> issue is solved"

Yes - and this issue has not been solved yet.  It should indeed be solved before 3.4 is released, so "release blocker" is spot on.
msg201086 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2013-10-23 23:48
*sigh* fine.  But the title of the issue is no longer accurate.

And, Christian, I generate the 3.4 maintenance branch during the release process, not before.  So if you have to remove sha3 you're going to have to remove it from trunk.
msg201092 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-24 06:22
> I'm prepared to pull the plug if there is any doubt about the final version of SHA-3 before beta 2 is released on Jan 5th.

Shouldn't it be removed before beta1? The usual rule of feature freeze applies here.
msg201096 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2013-10-24 07:58
This strikes me as a rather unusual case.  How about discuss it on python-dev, come to an agreement and document the process for this type of issue somewhere for future reference?  Or is that simply OTT?
msg201928 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-11-01 18:08
New information on NIST's hash forum strongly suggest that NIST is going to standardize SHA-3 according to the original proposal with c=2n and Sakura padding as well as two SHAKEs with variable length output.

SHA3-224 with c=448
SHA3-256 with c=512
SHA3-384 with c=768
SHA3-512 with c=1024

SHAKE128 with c=256
SHAKE256 with c=512
msg207169 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-01-02 16:38
I'm going to remove sha3 from the trunk tomorrow unless I hear otherwise. Python shouldn't implement something called "sha3" until SHA-3 actually is a standard. According to the current NIST timeline, the comment period on the draft FIPS should have ended by now, but AFAICT, the draft FIPS that starts the 90 day comment period hasn't even been published yet.
msg207170 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-01-02 16:40
Will it be possible/easy to maintain a sha3 module on PyPI? It would be nice to have to for Python 2.6-3.4.

@Christian: Are you interested to do that?
msg207171 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-01-02 16:44
Either that, or we call it something else than "sha3"?
msg207184 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-01-02 20:17
I would not bother pulling this out until the week before RC1 if the standard has not yet been declared final.

Otherwise, -1 on keeping it under another name.  The only hashes we bundle should be standard ones as those are the only ones people will want to use in the long run.  We'd be saddled with carrying along a non-standard likely not widely used algorithm implementation forever otherwise.

even if sha3 isn't declared before 3.4rc1, people building 3.4 against a sufficiently modern version of openssl that includes sha3 (as i'm sure some version will) will still have access to the algorithm.

otherwise i'm sure someone will package this as a module on pypi for older pythons regardless.
msg207187 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2014-01-02 20:52
I have created a backport of the sha3 for Python 2.6 to 3.3 about an year ago. It's on PyPI: https://pypi.python.org/pypi/pysha3 . I'm planing to update the code with SHAKE256 and SHAKE512 support soonish, too.

I have very high confidence that NIST is neither going to change the parameters or padding for SHA3 nor is NIST going to deviate from the original Keccak proposal. In case you still prefer to remove SHA3 I suggest that we stick to GPS' plan and wait until RC1.

The attached patch removes all code and documentation for SHA3.
msg207188 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-01-02 21:13
Ok, this this remains a release blocker. I'm still +1 for removing it, and I'm -0 for removing it just before the release candidate. AFAICT, there is *zero* (.000000001) chance that it actually becomes a NIST standard before the Python release is made. According to the current timeline:

 http://csrc.nist.gov/groups/ST/hash/sha-3/timeline_fips.html

the *submission to the secretary* (of commerce) was scheduled for Q2. With the current delay, this must become Q3, so the publication as a standard might happen in Q4 (not sure how long the Secretary of Commerce needs to study the specification of a hash algorithm).

What might happen is that a draft is published by the time the RC is made. I'd then still be -1 on including something in Python that only implements a draft standard. So we could just as well remove it right away.
msg207189 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-01-02 21:16
I agree with Martin that it should be removed right now. It's not really reasonable to call something SHA-3 if it's not SHA-3, even in beta versions.
msg207190 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-01-02 21:46
OpenSSL doesn't implement SHA-3 yet, it's strange to have SHA-3 in
Python but not in OpenSSL. If the standard is still a draft, I agree
to remove the code right now.
msg207191 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-01-02 21:52
Given the likely delay in the standard Martin cites, I've change my mind: agreed.  Go ahead and remove it for 3.4.

We'll have an official sha3 in Python 3.5.  Early adopters can live with PyPI.
msg207192 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-01-02 22:07
I just looked at the hash-forum archives (*)

http://cio.nist.gov/esd/emaildir/lists/hash-forum/msg02809.html

which says that they plan to publish the draft "soon after Christmas".
They also indicate how the padding open issue might get resolved (append 1111 for SHAKE, 1101 for the SHA-2 drop-ins). Not sure whether this is what Christian has already implemented.

(*) See http://crypto.stackexchange.com/questions/10645/are-nists-changes-to-keccak-sha-3-problematic for the password
msg207225 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-01-03 13:11
New changeset 52350d325b41 by Martin v. Löwis in branch 'default':
* Issue #16113: Remove sha3 module again.
http://hg.python.org/cpython/rev/52350d325b41
msg207226 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-01-03 13:13
I have now removed the aha code.
msg207228 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-01-03 14:48
@Martin: It looks like the _overlapped module is not more compiled on Windows.

http://buildbot.python.org/all/builders/x86%20Windows%20Server%202008%20%5BSB%5D%203.x/builds/2032/steps/test/logs/stdio

test test_asyncio crashed -- Traceback (most recent call last):
  File "E:\home\cpython\buildslave\x86\3.x.snakebite-win2k8r2sp1-x86\build\lib\asyncio\__init__.py", line 16, in <module>
    from . import _overlapped
ImportError: cannot import name '_overlapped'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "../lib/test/regrtest.py", line 1278, in runtest_inner
    test_runner()
  File "E:\home\cpython\buildslave\x86\3.x.snakebite-win2k8r2sp1-x86\build\lib\test\test_asyncio\__init__.py", line 31, in test_main
    run_unittest(suite())
  File "E:\home\cpython\buildslave\x86\3.x.snakebite-win2k8r2sp1-x86\build\lib\test\test_asyncio\__init__.py", line 21, in suite
    __import__(mod_name)
  File "E:\home\cpython\buildslave\x86\3.x.snakebite-win2k8r2sp1-x86\build\lib\test\test_asyncio\test_base_events.py", line 11, in <module>
    from asyncio import base_events
  File "E:\home\cpython\buildslave\x86\3.x.snakebite-win2k8r2sp1-x86\build\lib\asyncio\__init__.py", line 18, in <module>
    import _overlapped  # Will also be exported.
ImportError: No module named '_overlapped'
msg207229 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-01-03 14:54
Thanks for the report. Restored in 8a3718f31188
msg231838 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-11-28 23:22
New changeset 21257f916668 by Ned Deily in branch '3.4':
Issue #16113: Also remove test_case_sha3_224_huge
https://hg.python.org/cpython/rev/21257f916668

New changeset bd97eab25c70 by Ned Deily in branch 'default':
Issue #16113: Also remove test_case_sha3_224_huge
https://hg.python.org/cpython/rev/bd97eab25c70
msg253023 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2015-10-15 00:56
SHA-3 is released.

http://www.nist.gov/manuscript-publication-search.cfm?pub_id=919061

Should we add it to Python 2.7.11, 3.4.4, 3.5.1 or just for 18 months away Python 3.6?. Since this is a security sensitive issue, I would vote for adding it to the maintained releases.
msg253025 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-10-15 01:16
Describing this as a "security sensitive issue" is being facile.  It's more appropriate to describe this as a "new feature", aka, something that does not go in after x.y.0 final.

Please only check this in to "default", aka 3.6.
msg253028 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-10-15 05:48
I agree with Larry. If this feature is super useful, we can consider adding
it to stable versions later.
msg253029 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2015-10-15 05:50
I should clarify, I don't speak for 2.7.  The rules there are a little different and it's up to Benjamin to decide.  But please don't add new features to 3.4 and 3.5.
msg253174 - (view) Author: Björn Edström (bjornedstrom) Date: 2015-10-19 09:47
Remember that FIPS202 slightly change some parts of the Keccak that won the competition, so test results are different. I updated my stand alone SHA3 module, for anyone who is interested in using this now in Python 2 and 3.

https://github.com/bjornedstrom/python-sha3
msg264029 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-04-22 19:00
The authors of Keccak have released a new version of the Keccak Code Package, http://keccak.noekeon.org/reorganized_code.html . The new package makes it much easier to integrate Keccak in Python. I'm working on a new patch with SHA3 and SHAKE support.
msg265033 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-05-06 21:11
This patch implements SHA-3 and SHAKE for Python 3.6. The algorithm is provided by a slightly modified copy of the Keccak Code Package. I had to replace C++ comments and perform some minor cleanups.
msg265059 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2016-05-07 11:17
Is there any guidance or recommendation on how to use the SHAKE variants?
msg265066 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2016-05-07 13:15
Christian: any interest in proposing this for 2.7?  We could ask Benjamin.  It could still make 2.7.11--rc1 should be tagged in about a month.
msg265088 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2016-05-07 19:05
I'd there any good reason 2.7 needs this? They are available via pypi as
extensions. (Read: I vote no)

On Sat, May 7, 2016, 3:15 AM Larry Hastings <report@bugs.python.org> wrote:

>
> Larry Hastings added the comment:
>
> Christian: any interest in proposing this for 2.7?  We could ask
> Benjamin.  It could still make 2.7.11--rc1 should be tagged in about a
> month.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue16113>
> _______________________________________
>
msg265125 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-05-08 10:49
Larry,
I'm with Gregory. There is no good reason to add SHA3 to Python 2.7. The SHA-2 family is still safe. Besides I'd rather add BLAKE2 to Python 2.7. It's much faster and more versatile than SHA3.

Antoine,
SHAKEs are XOF (extensible output function). NIST has standardized the XOFs but not yet approved them as replacement for other constructs. They are useful for signatures or as a simple stream cipher. The SHAKEs were low hanging fruits to implement, so I included them.
msg266911 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-06-02 18:51
New patch:

- I moved the test vectors out of the repos. They are currently hosted on github. I'll move them to pythontest infra later.
msg266974 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2016-06-02 22:18
comments added to the code review.
msg273252 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-08-20 23:35
Patch 3 addresses GPS' code review.
msg273328 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2016-08-22 02:18
> The SHAKEs were low hanging fruits to implement, so I included them.

I don't think this is sufficient motivation.  Each new API is a permanent maintenance and documentation burden.  It is also a burden to every new user seeing the module and trying to decide which offering to use.  We should provide tools that we know people need and error on the side of economy.  I asked a room full of network engineers about SHAKE and not a single one of them had heard of it, so I think it would be premature to add to the standard library.
msg273330 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2016-08-22 02:23
> I asked a room full of network engineers about SHAKE and not a single one of them had heard of it

Why would a network engineer know about a new variable length hashing algorithm? It's not really within their problem domain.
msg273331 - (view) Author: Aaron Gallagher (habnabit) Date: 2016-08-22 02:26
I'm not sure why one would pick and choose here—SHAKE is part of the NIST
SHA-3 standard.
msg273363 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-08-22 12:07
The maintenance burden is minimal. All six algorithms are just variants of the same KeccakP-1600 sponge construction with different initialization parameters for rate, capacity, delimiter and output size. SHAKEs have no default output len and another delimiter as SHA3s. https://github.com/gvanas/KeccakCodePackage/blob/master/Modes/KeccakHash.h#L34
msg274786 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-07 09:58
New changeset f8700ee4aef0 by Christian Heimes in branch 'default':
Issue #16113: Add SHA-3 and SHAKE support to hashlib module.
https://hg.python.org/cpython/rev/f8700ee4aef0
msg274789 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-07 10:42
New changeset 4971ca2960c7 by Christian Heimes in branch 'default':
Issue #16113: KeccakP-1600-opt64 does not support big endian platforms yet.
https://hg.python.org/cpython/rev/4971ca2960c7
msg274790 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-07 10:52
New changeset e8884dcace9f by Christian Heimes in branch 'default':
Issue #16113: compile the module on Windows, too.
https://hg.python.org/cpython/rev/e8884dcace9f
msg274791 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-07 11:01
New changeset 68df416e94ba by Christian Heimes in branch 'default':
Issue #16113: take 2 on big endian machines.
https://hg.python.org/cpython/rev/68df416e94ba
msg274793 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-09-07 11:05
A buildbot is complaining about strict aliasing:

In file included from /buildbot/buildarea/3.x.ware-gentoo-x86.installed/build/Modules/_sha3/sha3module.c:113:0:
/buildbot/buildarea/3.x.ware-gentoo-x86.installed/build/Modules/_sha3/kcp/KeccakP-1600-inplace32BI.c: In function ‘_PySHA3_KeccakP1600_SetBytesInLaneToZero’:
/buildbot/buildarea/3.x.ware-gentoo-x86.installed/build/Modules/_sha3/kcp/KeccakP-1600-inplace32BI.c:97:5: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
     low = *((UINT32*)(laneAsBytes+0));
     ^
In file included from /buildbot/buildarea/3.x.ware-gentoo-x86.installed/build/Modules/_sha3/sha3module.c:113:0:
/buildbot/buildarea/3.x.ware-gentoo-x86.installed/build/Modules/_sha3/kcp/KeccakP-1600-inplace32BI.c: In function ‘_PySHA3_KeccakP1600_AddBytesInLane’:
/buildbot/buildarea/3.x.ware-gentoo-x86.installed/build/Modules/_sha3/kcp/KeccakP-1600-inplace32BI.c:152:5: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
     low = *((UINT32*)(laneAsBytes+0));
     ^
/buildbot/buildarea/3.x.ware-gentoo-x86.installed/build/Modules/_sha3/kcp/KeccakP-1600-inplace32BI.c: In function ‘_PySHA3_KeccakP1600_ExtractBytesInLane’:
/buildbot/buildarea/3.x.ware-gentoo-x86.installed/build/Modules/_sha3/kcp/KeccakP-1600-inplace32BI.c:294:5: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
     *((UINT32*)(laneAsBytes+0)) = low;
     ^
/buildbot/buildarea/3.x.ware-gentoo-x86.installed/build/Modules/_sha3/kcp/KeccakP-1600-inplace32BI.c: In function ‘_PySHA3_KeccakP1600_ExtractAndAddBytesInLane’:
/buildbot/buildarea/3.x.ware-gentoo-x86.installed/build/Modules/_sha3/kcp/KeccakP-1600-inplace32BI.c:367:5: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
     *((UINT32*)(laneAsBytes+0)) = low;
msg274797 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-07 11:18
New changeset ddc95a9bc2e0 by Christian Heimes in branch 'default':
Issue #16113: one more C90 violation in big endian code.
https://hg.python.org/cpython/rev/ddc95a9bc2e0
msg275000 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-08 11:35
New changeset e5871ffe9ac0 by Christian Heimes in branch 'default':
Issue #16113: SHA3: allocate extra memory for lane extraction and check return value of PyModule_Create()
https://hg.python.org/cpython/rev/e5871ffe9ac0
msg288706 - (view) Author: Michał Górny (mgorny) * Date: 2017-02-28 11:18
Christian, since the code is now integrated in Python 3.6+ (with some bugfixes AFAICS), could you consider updating your bitbucket package to match it? It would be helpful as a backport package for older Python versions.
History
Date User Action Args
2022-04-11 14:57:36adminsetgithub: 60317
2017-02-28 11:18:51mgornysetnosy: + mgorny
messages: + msg288706
2016-09-13 07:17:48christian.heimeslinkissue28117 dependencies
2016-09-08 11:35:17python-devsetmessages: + msg275000
2016-09-08 09:51:00christian.heimessetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2016-09-07 11:18:56python-devsetmessages: + msg274797
2016-09-07 11:05:21christian.heimessetmessages: + msg274793
2016-09-07 11:01:57python-devsetmessages: + msg274791
2016-09-07 10:52:05python-devsetmessages: + msg274790
2016-09-07 10:42:56python-devsetmessages: + msg274789
2016-09-07 10:11:38christian.heimessetstage: patch review -> commit review
2016-09-07 09:58:40python-devsetmessages: + msg274786
2016-08-22 12:07:57christian.heimessetmessages: + msg273363
2016-08-22 02:26:52habnabitsetmessages: + msg273331
2016-08-22 02:23:35dstufftsetmessages: + msg273330
2016-08-22 02:18:06rhettingersetnosy: + rhettinger
messages: + msg273328
2016-08-20 23:36:08christian.heimessetfiles: + SHA3-and-SHAKE-support-for-Python-3.patch

messages: + msg273252
2016-06-12 11:21:18christian.heimessetassignee: christian.heimes ->
2016-06-02 22:18:00gregory.p.smithsetmessages: + msg266974
2016-06-02 18:51:15christian.heimessetfiles: + SHA3-and-SHAKE-support-for-Python-2.patch

messages: + msg266911
2016-05-09 13:20:13englabennysetnosy: - englabenny
2016-05-08 10:49:02christian.heimessetmessages: + msg265125
2016-05-07 19:05:19gregory.p.smithsetmessages: + msg265088
2016-05-07 13:15:42larrysetmessages: + msg265066
2016-05-07 11:17:08pitrousetmessages: + msg265059
2016-05-06 21:12:14christian.heimessetfiles: + SHA3-and-SHAKE-support-for-Python.patch

messages: + msg265033
stage: needs patch -> patch review
2016-05-06 21:08:50christian.heimessethgrepos: - hgrepo152
2016-04-22 19:00:52christian.heimessettitle: Add SHA-3 (Keccak) support -> Add SHA-3 and SHAKE (Keccak) support
messages: + msg264029
stage: needs patch
2016-03-06 20:13:50BreamoreBoysetnosy: - BreamoreBoy
2016-03-06 20:04:59spatzsetnosy: + spatz
2015-10-19 09:47:04bjornedstromsetmessages: + msg253174
2015-10-15 06:27:36Arfreversetstatus: closed -> open
versions: + Python 3.6, - Python 3.4
title: SHA-3 (Keccak) support may need to be removed before 3.4 -> Add SHA-3 (Keccak) support
priority: release blocker -> normal
resolution: fixed -> (no value)
stage: commit review -> (no value)
2015-10-15 05:50:57larrysetmessages: + msg253029
2015-10-15 05:48:01vstinnersetmessages: + msg253028
2015-10-15 01:16:42larrysetmessages: + msg253025
2015-10-15 00:57:00jceasetmessages: + msg253023
2014-11-28 23:22:50python-devsetmessages: + msg231838
2014-01-03 14:54:24loewissetstatus: open -> closed
resolution: fixed
messages: + msg207229
2014-01-03 14:48:09vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg207228
2014-01-03 13:13:21loewissetstatus: open -> closed
resolution: fixed
messages: + msg207226
2014-01-03 13:11:02python-devsetmessages: + msg207225
2014-01-02 22:07:35loewissetmessages: + msg207192
2014-01-02 21:52:53gregory.p.smithsetmessages: + msg207191
2014-01-02 21:46:43vstinnersetmessages: + msg207190
2014-01-02 21:16:03pitrousetmessages: + msg207189
2014-01-02 21:13:34loewissetmessages: + msg207188
2014-01-02 20:52:23christian.heimessetfiles: + remove_sha3.patch

messages: + msg207187
2014-01-02 20:17:04gregory.p.smithsetmessages: + msg207184
2014-01-02 16:44:00pitrousetmessages: + msg207171
2014-01-02 16:40:21vstinnersetnosy: + vstinner
messages: + msg207170
2014-01-02 16:38:06loewissetnosy: + loewis
messages: + msg207169
2013-11-01 18:08:23christian.heimessetmessages: + msg201928
2013-10-24 08:07:22makersetnosy: - maker
2013-10-24 07:58:07BreamoreBoysetnosy: + BreamoreBoy
messages: + msg201096
2013-10-24 06:22:52pitrousetmessages: + msg201092
2013-10-23 23:52:58tim.peterssetpriority: normal -> release blocker
title: Add SHA-3 (Keccak) support -> SHA-3 (Keccak) support may need to be removed before 3.4
2013-10-23 23:48:52larrysetmessages: + msg201086
2013-10-23 23:40:34tim.peterssetnosy: + tim.peters
messages: + msg201085
2013-10-23 23:23:17larrysetnosy: gregory.p.smith, jcea, pitrou, larry, christian.heimes, habnabit, ezio.melotti, Arfrever, asvetlov, englabenny, maker, python-dev, sbt, bjornedstrom, dstufft, markk, haakon
messages: + msg201084
2013-10-23 23:19:40Arfreversetnosy: + larry
messages: + msg201083
2013-10-23 23:16:09larrysetnosy: - larry
2013-10-23 23:15:29larrysetmessages: + msg201082
2013-10-23 23:01:35christian.heimessetnosy: + larry
messages: + msg201081
2013-10-23 22:59:28christian.heimessetmessages: + msg201080
2013-10-23 22:54:42larrysetnosy: - larry
2013-10-23 22:54:01larrysetpriority: release blocker -> normal

messages: + msg201079
2013-10-23 22:51:33vstinnersetpriority: normal -> release blocker
nosy: + larry
2013-10-23 22:48:50haakonsetnosy: + haakon
messages: + msg201078
2013-08-24 22:47:21dstufftsetnosy: + dstufft
2013-07-03 12:43:12markksetnosy: + markk
2013-06-27 23:41:20habnabitsetmessages: + msg191971
2013-06-27 10:32:07christian.heimessetmessages: + msg191940
2013-06-27 05:56:59habnabitsetnosy: + habnabit
messages: + msg191931
2013-05-29 12:49:35englabennysetmessages: + msg190303
2013-02-27 10:13:48englabennysetnosy: + englabenny
messages: + msg183129
2012-10-08 18:30:46pitrouunlinkissue16166 dependencies
2012-10-08 14:33:55christian.heimeslinkissue16166 dependencies
2012-10-07 17:14:33christian.heimessetmessages: + msg172324
2012-10-07 16:43:58sbtsetmessages: + msg172319
2012-10-07 16:43:30asvetlovsetnosy: + asvetlov
2012-10-07 16:24:40christian.heimessetmessages: + msg172316
2012-10-07 16:16:55christian.heimessetmessages: + msg172314
2012-10-07 16:15:16sbtsetnosy: + sbt
messages: + msg172313
2012-10-06 17:56:51Arfreversetnosy: + Arfrever
2012-10-06 14:24:48makersetnosy: + maker
2012-10-06 00:37:33christian.heimessetmessages: + msg172158
stage: patch review -> commit review
2012-10-06 00:36:25python-devsetnosy: + python-dev
messages: + msg172157
2012-10-05 21:40:14gregory.p.smithsetmessages: + msg172152
2012-10-05 21:10:06christian.heimessetmessages: + msg172144
2012-10-05 21:07:24christian.heimessetfiles: + 521e85a613bf.diff
2012-10-05 21:06:34christian.heimessetfiles: - 622009fb6192.diff
2012-10-05 21:06:27christian.heimessetfiles: - 49a949116245.diff
2012-10-05 21:06:16christian.heimessetfiles: - 44920b1d9db1.diff
2012-10-05 18:58:16christian.heimessetfiles: + 622009fb6192.diff
2012-10-05 16:59:22christian.heimessetmessages: + msg172100
2012-10-05 10:47:49christian.heimessetmessages: + msg172070
2012-10-05 10:45:06christian.heimessetfiles: + 49a949116245.diff
2012-10-04 21:02:25christian.heimessetnosy: + gregory.p.smith

messages: + msg171995
stage: needs patch -> patch review
2012-10-04 20:59:03christian.heimessetfiles: + 44920b1d9db1.diff
2012-10-04 20:57:27christian.heimessetfiles: - 79e3fb1838ce.diff
2012-10-04 18:05:59christian.heimessetfiles: + 79e3fb1838ce.diff
2012-10-04 18:05:38christian.heimessetmessages: + msg171983
2012-10-04 17:59:52christian.heimessetfiles: - 4509ef9b28a0.diff
2012-10-04 16:29:43pitrousetmessages: + msg171971
2012-10-04 16:29:17christian.heimessetfiles: + 4509ef9b28a0.diff
keywords: + patch
2012-10-04 16:19:43christian.heimessethgrepos: + hgrepo152
messages: + msg171968
2012-10-04 15:32:42pitrousetmessages: + msg171964
2012-10-04 15:31:43christian.heimessetmessages: + msg171963
2012-10-04 15:26:14pitrousetnosy: + pitrou
2012-10-04 10:16:47christian.heimessetmessages: + msg171929
2012-10-03 22:34:49bjornedstromsetnosy: + bjornedstrom
messages: + msg171913
2012-10-03 17:06:49christian.heimessetassignee: christian.heimes
messages: + msg171898
2012-10-03 13:28:21jceasetmessages: + msg171882
2012-10-03 13:18:40jceasetnosy: + jcea
2012-10-03 12:30:49brett.cannonlinkissue16118 superseder
2012-10-03 03:16:47ezio.melottisetnosy: + ezio.melotti

stage: needs patch
2012-10-03 03:10:04christian.heimescreate