classification
Title: Add os.urandom_block()
Type: Stage:
Components: Versions: Python 3.6
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: Lukasa, alex, lemburg, ncoghlan, vstinner
Priority: normal Keywords:

Created on 2016-06-07 12:21 by vstinner, last changed 2016-07-28 17:06 by vstinner. This issue is now closed.

Messages (15)
msg267646 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-06-07 12:21
The change of the issue #26839 creates a long discussion around the behaviour of os.urandom() when the entropy pool is not seeded with enough entropy.

On Python 2.7, os.urandom() doesn't block in this case. I expect that bytes are random, but not "random enough" to generate a cryptographic key.

I propose to add a new os.urandom_block() function which is similar to os.urandom(), but with one single difference: don't fallback on reading /dev/urandom on Linux with the entropy pool is not initialized yet.

In short, os.urandom() can be used for everything except generating cryptographic keys. os.urandom_block() is suitable for cryptographic keys.

--

I expect that security experts would prefer the opposite: add a new os.pseudo_random() function and make os.urandom() blocking by default.

Let me elaborate my rationale. Python is used in various areas. In most cases, random is not used for security purpose, but for game, simulation, numerical computation, etc. "random" term is large in Python:

* random seed for the randomized hash function: Python SIPHASH requires 24 random bytes
* importing the random modules immediatly instanciate a random.Random which is Mersenne Twister RNG which requires not less than 2500 random bytes

These two users of random bytes blocked Python startup: see issue #26839.

I consider that random is used in more cases for game, simulation, etc. than for security.

That's why the secrets module was added to Python 3.6 rather than making the random module "more secure".
https://lwn.net/Articles/659643/
msg267649 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2016-06-07 12:24
This should be add ``os.urandom_noblock`` instead. Don't make every other application change just so the Python interpreter can continue to call ``os.urandom``.
msg267651 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-06-07 12:26
If os.urandom_block() is added, I don't think that it's worth to add os.urandom_info as I proposed in the issue #27249.
msg267652 - (view) Author: Cory Benfield (Lukasa) * Date: 2016-06-07 12:32
Let me make the security person argument even though you've dismissed it in your original post:

Security should be on by default and opted out of, not the other way around. If the obvious choice is insecure then users who aren't careful enough won't notice, because even bad RNGs produce numbers that *look* random if not carefully evaluated. This means they won't fix it and it will stay there, broken, until they are attacked (which they may never notice).

On the other hand, if the obvious choice is secure then users who aren't careful enough (that is, ones that don't care about security but do care about blocking) *will* notice, because their apps will hang on startup. They'll investigate the hang, see the docs, and fix their code. This is a *much better* failure mode. There is a reason that modern cars warn you about any upcoming problem with the car (e.g. worn brake pads, low oil etc.): users whose expectations are violated in a way they can easily detect are much more likely to act than users whose expectations are subtly violated.

This is doubly true here, because the only system that is complaining about the random numbers right now is CPython, which, being the same system providing these new functions, can always ensure that it's doing the right thing. With this patch as proposed here, external library developers need to see this discussion and realise that, for Python 3.5 or lower, they want os.urandom, and for Python 3.6 and higher, they want os.urandom_block(). And, again, because os.urandom() still appears to produce high quality random numbers (and most of the time *actually does so*), I guarantee at least one application will not notice the change and will become open to attack.

Those are the two arguments for making non-blocking be explicit, rather than making blocking be specific.
msg267653 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-06-07 12:35
See also issue #26970 "Replace OpenSSL's CPRNG with system entropy source".
msg267655 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2016-06-07 12:38
Should this ticket be named "Add os.random()" ? After all,
blocking in case of missing entropy is what /dev/random is
all about.
msg267657 - (view) Author: Cory Benfield (Lukasa) * Date: 2016-06-07 12:41
Marc-Andre: No. See the discussion in the related issue #27249 for more.
msg267658 - (view) Author: Cory Benfield (Lukasa) * Date: 2016-06-07 12:41
Uh, sorry, I meant #26839.
msg267662 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2016-06-07 13:03
On 07.06.2016 14:41, Cory Benfield wrote:
> 
> Uh, sorry, I meant #26839.

Ok, then perhaps "Add os.getrandom()" - also see the discussion
there :-)
msg267919 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-06-08 22:34
> Should this ticket be named "Add os.random()" ? After all, blocking in case of missing entropy is what /dev/random is all about.

In short, /dev/random must not be used :-) See:
https://haypo-notes.readthedocs.io/pep_random.html#rng-denial-of-service
msg267920 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-06-08 22:42
IMHO "Should os.urandom() block before the kernel collected enough entropy?" is the last major question in the large discussion around random:
https://haypo-notes.readthedocs.io/pep_random.html#os-urandom

It became clear that Python at startup should use a weak entropy if high-quality entropy is not available (read would block):
https://haypo-notes.readthedocs.io/pep_random.html#python-startup

So Python startup should no more be impacted if os.urandom() blocks or not. In this case, I'm in favor of making os.urandom() the most secure as possible: block until the kernel collected enough entropy. I'm in favor in the issue #27266 instead of this one.

If it is not possible to agree on a solution, I fear that a PEP will be required. I hope that once people understood that the Python startup issue is (no more) unrelated to the behaviour of os.urandom() (block or not), most people will be in favor of making os.urandom() as secure as possible.

See also https://haypo-notes.readthedocs.io/pep_random.html : my summary of the issue #26839.
msg267932 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-06-08 23:46
One request I'd make in these discussions is that we avoid using the term "block" - it makes people think of the /dev/random behaviour (i.e. blocking intermittently and unhelpfully), rather than the usually-desired "wait for sufficient entropy on system startup" behaviour.

I'd also request that we keep in mind that any Linux user always remains free to write the 3-line utility function:

    def read_urandom(num_bytes):
        with open('/dev/urandom', 'rb') as urandom:
            return urandom.read(num_bytes)

If they want to get precisely the Linux /dev/urandom semantics, and not a Python level abstraction that provides the same kinds of assurances offered by other *nix platforms.
msg267933 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-06-08 23:49
Victor, given the assumption that internal hashing and the random module will change to no longer depend on os.urandom, are you happy to close this one in favour of #27250?
msg267975 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2016-06-09 07:58
Please also see #27279 for an alternative plan.
msg271569 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-07-28 17:06
I don't support this idea anymore, so I close the issue.

I wrote the PEP 524 to propose to make os.urandom() blocking on Linux. The discussion moved to the security-sig mailing list.
History
Date User Action Args
2016-07-28 17:06:28vstinnersetstatus: open -> closed
resolution: rejected
messages: + msg271569
2016-06-16 13:26:26dstufftsetnosy: - dstufft
2016-06-12 11:34:53christian.heimessetnosy: - christian.heimes
2016-06-09 07:58:58lemburgsetmessages: + msg267975
2016-06-08 23:49:39ncoghlansetmessages: + msg267933
2016-06-08 23:46:49ncoghlansetmessages: + msg267932
2016-06-08 23:31:43ncoghlansetnosy: + ncoghlan
2016-06-08 22:42:03vstinnersetmessages: + msg267920
2016-06-08 22:34:33vstinnersetmessages: + msg267919
2016-06-07 13:03:02lemburgsetmessages: + msg267662
2016-06-07 12:41:51Lukasasetmessages: + msg267658
2016-06-07 12:41:23Lukasasetmessages: + msg267657
2016-06-07 12:38:39lemburgsetnosy: + lemburg
messages: + msg267655
2016-06-07 12:35:57vstinnersetmessages: + msg267653
2016-06-07 12:35:11vstinnersetnosy: + christian.heimes
2016-06-07 12:32:44Lukasasetnosy: + Lukasa
messages: + msg267652
2016-06-07 12:26:18vstinnersetmessages: + msg267651
2016-06-07 12:24:25dstufftsetnosy: + dstufft
messages: + msg267649
2016-06-07 12:22:54alexsetnosy: + alex
2016-06-07 12:21:07vstinnercreate