Title: ssl: get CPU cap flags for AESNI and PCLMULQDQ
Type: enhancement Stage: resolved
Components: Extension Modules, SSL Versions: Python 3.6
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: alex, christian.heimes, dstufft, gregory.p.smith, janssen, pitrou
Priority: normal Keywords:

Created on 2016-08-15 13:55 by christian.heimes, last changed 2017-09-05 23:37 by pitrou. This issue is now closed.

Messages (13)
msg272763 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-08-15 13:55
OpenSSL has a function called OPENSSL_ia32cap_loc() to get the processor's capability vector in X86 and X86_64 systems. The information is useful to decide which cipher suite to prefer. For example on machines without AES-NI and CLMUL CPU instructions, ChaCha20 should be prefered over AES-GCM.

#27766 just exposes the plain OPENSSL_ia32cap_loc(). A richer API should parse the bit field and expose the bits as structure.
msg272845 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2016-08-16 07:14
This is very architecture specific and focused on low level information that you can also already get from places like /proc/cpuinfo on Linux.

Also, regardless of what capability bits a CPU exposes that has nothing to do with what accelerations the underlying library implementation itself actually supports.

For your example purpose of choosing which algorithm to use, testing the actual performance of each option in your code at startup time seems more foolproof in any application running long enough for performance to be an issue.

I expect in most common situations you can just use ctypes to call this function from openssl if you feel you must use it.

I'm not convinced it belongs in the stdlib.
msg273359 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2016-08-22 11:40
Agreed with Gregory.

I'm also surprised you consider selecting a cipher suite based on performance considerations rather than security. I'm not sure that's something we want to promote.

(btw, AES speed is usually not a critical factor except in very specific situations)
msg273360 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2016-08-22 11:41
In this case, performance is security. Both AES-GCM and ChaCha20-Poly1305 are secure. Modulo one thing: GCM in software is hard to implement in constant-time, so it's strongly preferable to use it only when there's a hardware implementation. It works out nicely that in addition to being constant-time, the hardware implementation of GCM is faster.
msg273361 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2016-08-22 11:49
I still think it's a slippery slope to include such specialized APIs that most people don't know what to make of.  This reminds me of `RAND_egd()`.

If ChaCha20-Poly1305 is always /at least/ as secure as AES-GCM (and sometimes more), then we should simply prioritize it in the cipher list, regardless of a potentially poorer performance that probably never has an actual impact on the Python-written application (correct me if I'm wrong here).
msg273633 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-08-25 07:36
GPS, sure it is simple enough under Linux. But what about other operating systems? OPENSSL_ia32cap_loc() works under Windows, too.

Antoine, AES-GCM is still faster and performs better than ChaCha20 Poly1305. NSS and Mozilla's recommended cipher suite list prefers AES-GCM over ChaCha20, too.

It's reasonable and simple to provide the best cipher suite that matches the systems' capabilities. As Alex stated, performance is security.

To provide the CPU capabilities to the ssl module and 3rd party authors (e.g. Cory asked on behalf of requests), let's keep ssl._ia32cap() a private function and just add two constants: HAVE_AESNI = True/False/None, HAVE_PCLMULQDQ = True/False/None (None: ia32cap is not available on the system). Is that ok with you?
msg273635 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2016-08-25 07:49
The constants would have to be private, too. We really don't want to encourage other users to rely on them.
msg273636 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-08-25 07:54
Why? Other libraries like requests like to use the information, too.
msg273637 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2016-08-25 08:01
If we wanted to expose such information, we would need to do it in a dedicated module (or in `os`) and provide something that doesn't restrict itself to two particular x86 flags.

requests can rely on a private attribute if they really want to.
msg273676 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2016-08-25 20:19
The processor capabilities vector is not the API you want.

What you want is an API to ask the ssl library about characteristics of algorithm implementations it will be using.

Those are not the same thing.

If we want to expose processor capabilities as a library function it should not be done in the ssl module.  That information belongs in the platform module.  And doing so should not require a third party library such as OpenSSL.  Querying capabilities is a very simple instruction and every platform architecture specific C macros to do it.
msg273702 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-08-26 11:16
GPS, I no longer want a general API for CPU cap vector. The ssl module and 3rd party packages are only interested in two properties of the current hardware: are AESNI and PCLMULQDQ instructions available and enabled.

I'm pretty sure ctypes won't work on Windows and OSX. The OpenSSL libraries are statically linked into the _ssl binary. The linker optimizer removes all functions that are not used. Somebody could give it a try, though.
msg301401 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-09-05 23:31
I'm retracting my patch. Instead of making the cipher suite selection more complicated, we should just rely on OpenSSL to provide sensible defaults.
msg301404 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-05 23:37
>  Instead of making the cipher suite selection more complicated, we should just rely on OpenSSL to provide sensible defaults.

+1 !
Date User Action Args
2017-09-05 23:37:52pitrousetmessages: + msg301404
2017-09-05 23:31:22christian.heimessetstatus: open -> closed
resolution: rejected
messages: + msg301401

stage: resolved
2016-09-24 21:26:17christian.heimesunlinkissue27766 dependencies
2016-09-15 08:33:23giampaolo.rodolasetnosy: - giampaolo.rodola
2016-09-15 07:58:07christian.heimessetassignee: christian.heimes
components: + SSL
2016-08-26 11:16:29christian.heimessetmessages: + msg273702
title: ssl: add public API for IA-32 processor capabilities vector -> ssl: get CPU cap flags for AESNI and PCLMULQDQ
2016-08-25 20:19:29gregory.p.smithsetmessages: + msg273676
2016-08-25 08:01:43pitrousetmessages: + msg273637
2016-08-25 07:54:05christian.heimessetmessages: + msg273636
2016-08-25 07:49:54pitrousetmessages: + msg273635
2016-08-25 07:36:43christian.heimessetmessages: + msg273633
2016-08-22 11:49:31pitrousetmessages: + msg273361
2016-08-22 11:41:52alexsetmessages: + msg273360
2016-08-22 11:40:21pitrousetnosy: + pitrou
messages: + msg273359
2016-08-16 07:14:37gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg272845
2016-08-15 13:55:33christian.heimeslinkissue27766 dependencies
2016-08-15 13:55:07christian.heimescreate