This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Warn users that os.urandom() prior to 3.6 can return insecure values
Type: security Stage: resolved
Components: Documentation Versions: Python 3.5, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Lukasa, christian.heimes, docs@python, georg.brandl, larry, martin.panter, ncoghlan, steven.daprano, vstinner
Priority: normal Keywords: 3.5regression, patch

Created on 2016-06-11 10:07 by christian.heimes, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
urandom-doc.patch martin.panter, 2016-06-16 11:01
urandom-doc.patch martin.panter, 2016-06-16 13:08 review
urandom-doc.v2.patch martin.panter, 2016-06-18 01:27 review
Messages (21)
msg268205 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-06-11 10:07
In #26839 os.urandom() was made non-blocking and non-exception-raising on Linux. As a result os.urandom() is no longer a CSPRNG under some conditions as it can and will return predictable random values without any sort of warning or error flag. These conditions are (including but not limited to):

* early boot state
* virtualization without host-passthrough, e.g. virtio-rng
* embedded devices without hardware RNG or RTC, e.g. raspberry pi

Please ensure that the documentation properly warns users about these edge cases.

We might also want to add that os.urandom() can block on other platforms, e.g. FreeBSD and OpenBSD.
msg268207 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2016-06-11 10:33
Relevant: issue #27293

(I've taken the liberty of subscribing those on this issues nosy list to the new issue, I hope that's okay)
msg268238 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2016-06-11 18:07
I don't think this is necessary, as the documentation for os.urandom() is already pretty good.  Here's the relevant bit:

    This function returns random bytes from an OS-specific randomness
    source. The returned data should be unpredictable enough for
    cryptographic applications, though its exact quality depends on
    the OS implementation. On a Unix-like system this will query
    /dev/urandom, and on Windows it will use CryptGenRandom().

ISTM that the Python documentation doesn't generally indulge in warning users about specific shortcomings of particular platforms; if it did it would be littered with such warnings.

Personally I'd approve of making the existing statements a little more forceful, like pulling it out into a red "warning" box and making it explicit that os.urandom() isn't any more sophisticated than the platform-specific technologies it uses.  But that's as far as I'd go.  I wouldn't add all the specifics you suggest.

Technically I think this actually is my call, as I'm the "platform expert" for the posix module:
  https://docs.python.org/devguide/experts.html
But really I think it's the call of the "Documentation Expert" for the relevant releases.  This is a stylistic concern--should the Python docs delve into these sorts of details?--and that's really the domain of the DE.

Georg Brandl is the DE for all currently-supported versions of Python.  (Well, 2.7 has no official DE, but I think Georg is de facto DE for that release too.)  I've nosied him here; hopefully he can tell us the standard Python doc aesthetic when it comes to these sorts of concerns.


By the way, the Raspberry PI does have hardware RNG:
  http://scruss.com/blog/2013/06/07/well-that-was-unexpected-the-raspberry-pis-hardware-random-number-generator/
It required loading an extra driver, at least as of 2014.  I concede I don't know what current crypto best-practices are on the PI.

That's one good reason why I think the Python documentation doesn't indulge in these laundry lists of platform failings--such information has a tendency to become out-of-date without anyone noticing.
msg268239 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2016-06-11 18:11
Oh, and, for 3.6 I would definitely support adding a mention of "Instead of using this function directly, we recommend you use the token_bytes() function in the secrets module", blah blah best practices etc.

That goes for os.getrandom() too, if we add it to 3.6 (which I rather suspect we will).
msg268242 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2016-06-11 18:34
I agree that we should add a warning to these, it's easy to see how someone might read the summary of the function "Return a string of n random bytes suitable for cryptographic use." and skip over the nuance in the rest of the body of the function. Adding a red box to ensure that they know that on popular platforms os.urandom is not going to always return bytes that are suitable for cryptographic use is pretty important in my opinion. The current wording makes it sound like it's something you only need to worry about on "weird" platforms, not on one of the (if not the) most popular platforms for running Python on.
msg268244 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2016-06-11 18:36
I would suggest weakening the one-line summary.  Currently the first line reads:

    Return a string of n random bytes suitable for cryptographic use.

I'd support adding some "weasel words" to this, e.g.:

    Return a string of n random bytes that should be suitable
    for cryptographic use.
msg268316 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2016-06-12 02:58
This is not a release blocker.
msg268649 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-16 08:11
As far as this bug goes, 3.5 is not very different from 2.7
msg268656 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-16 11:01
Here is a possible patch for 3.5+ based on my modest understanding of the concerns about insecure results and blocking. I hope that my wording is clear, couldn’t be confused with Linux’s /dev/random blocking and running out of fresh entropy, etc.

I also tried to make it clearer what APIs are used in what circumstances. It is not just Linux: we also call getrandom() on Solaris, because its getentropy() is not good enough.
msg268662 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-06-16 12:54
Strange, I don't see the [Review] button.

    .. versionchanged:: 3.5.2
-      On Linux, if ``getrandom()`` blocks (the urandom entropy pool is not
+      If ``getrandom()`` blocks (the urandom entropy pool is not
       initialized yet), fall back on reading ``/dev/urandom``.

Please keep "On Linux", getrandom() is also used on Solaris, and my change is really restricted to Linux.
msg268663 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-16 13:08
Rebased so Rietveld can work with it, earlier version was my fault.

As far as I can see (looking at Python/random.c and configure.ac), the Solaris version should also use GRND_NONBLOCK:

#ifdef MS_WINDOWS
#elif defined(HAVE_GETENTROPY) && !defined(sun)
#else

#if defined(HAVE_GETRANDOM) || defined(HAVE_GETRANDOM_SYSCALL)
    const int flags = GRND_NONBLOCK;
#ifdef HAVE_GETRANDOM
            n = getrandom(dest, n, flags);
#else
            n = syscall(SYS_getrandom, dest, n, flags);
#endif

Apart from using a C function call versus syscall(), I don’t see there is much difference between the Solaris and Linux cases. Correct me if I’m wrong though.
msg268666 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-06-16 13:14
> As far as I can see (looking at Python/random.c and configure.ac), the Solaris version should also use GRND_NONBLOCK:

Oh, you're right: I didn't notice that GRND_NONBLOCK was also used on Solaris. The change is not deliberate, but it is good to do that :-)

Solaris getrandom() is documented to fail with EAGAIN if "No entropy is available and GRND_NONBLOCK is set."
https://docs.oracle.com/cd/E53394_01/html/E54765/getrandom-2.html

The question is more if reading from /dev/urandom block in this case.

If we don't know, I would prefer to keep the "On Linux" prefix in the doc, and don't say anything about Solaris.

I'm able to check the behaviour of Solaris.

You should contact the developers who get access to Solaris, you can meet them in the previous random issues specific to Solaris: issue #25003 and issue #26735.
msg268752 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-18 01:27
Restored “On Linux” for the changed in 3.5.2 notice. I do think it is better to be general and future-proof, but that is a separate, less important issue to the main purpose of the patch. (I don’t know if Solaris’s version can block or not.)
msg274710 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-09-07 02:34
PEP 524 has been implemented for 3.6b1 in #27776, so os.urandom() itself will now do the right thing for cryptographic use cases on Linux. Accordingly, I've removed Python 3.6 from the affected versions for this issue.

I've also flagged this as purely a documentation issue, since no behavioural changes are currently proposed for older versions.
msg274723 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-09-07 03:01
I dislike urandom-doc.v2.patch. There is no need to worry all users. Just be explicit and explain that the issue is specific to Linux and explain when it occurs: before the entropy pool is filled *and* if the system was not able yet to write enough entropy on disk, this thing:
https://www.python.org/dev/peps/pep-0524/#load-entropy-from-disk-at-boot
msg275260 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-09-09 05:13
Do you want to do an alternative patch Victor? Or point out all the specific bits of my patch you don’t like?

I haven’t really been keeping up to date with the getrandom() changes. Though I imagine even Python 3.6’s os.urandom() will still fall back to /dev/urandom (with potential entropy problem) on older Linux versions. Is the consensus that we want to warn about insecure results on Linux >= 3.17, but don’t want to warn about older Linux versions?
msg275269 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-09-09 07:01
> Though I imagine even Python 3.6’s os.urandom() will still fall back to /dev/urandom (with potential entropy problem) on older Linux versions.

Right, but you should be more specific in the doc.

os.urandom() is unsecure if and only if:

* getrandom() is not available (ex: Linux < 3.17, or Python built without getrandom() for different reasons) or don't work (ex: blocked by a stupid SECCOMP policy, ex: issue #27955)
* and /dev/urandom is not initialized yet which means that:

  - the currently running Python runs very early during the system initialization,
  - the system has no good source of entropy and slow entropy sources, 
  - no entropy was stored on the disk on a previous boot.

What I don't want: a doc warning that "oh hey, os.urandom() is unsafe, don't use it for anything serious".

os.urandom() is always secure on all platforms except Linux. On Linux, it's secure is almost all cases, except on a very few very tiny corner cases.

I proposed something like:

"On Linux, os.urandom() can return weak entropy when /dev/urandom is used internally and the system urandom entropy pool is not initialized yet."

I'm not even sure about "weak entropy" because in most cases, /dev/urandom is already partially initialized with good entropy, but just not enough to consider that it's fully initialized. Linux uses many entropy sources but don't trust them, so don't consider that these input data counts for the entropy counter.
msg275600 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-09-10 08:27
On modern Intel chips, one of the entropy sources is the CPU itself, and so this problem is mostly theoretical on such systems unless you're worried about the quality of Intel's entropy generation (in which case you're well and truly into sovereign espionage and advanced persistent threat territory, and really shouldn't be blindly trusting operating systems and programming language runtimes to already be doing the right thing).

So the main ways to get bad entropy from /dev/urandom on Linux are:

- a misconfigured VM that has been cut off from all decent entropy sources, and doesn't have any persistent storage attached
- an embedded non-Intel chip that doesn't have any decent entropy sources or persistent storage attached

If you're not doing either of those things, you're probably fine. If you're worried that you might be, try running Python 3.6 and calling os.getrandom() explicitly to see what happens.
msg277073 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-09-20 21:28
> Please ensure that the documentation properly warns users about these edge cases.

I disagree. I don't think that the Python documentation is the right place to document the security level of system urandom.

It's just a mess, there are so many corner cases and it's very hard to provide a clear explanation for end users.

I suggest to keep the positive "suitable for cryptographic use". If you change this sentence, I only expect that users will use something WORSE. For example "os.urandom is not secure! we must use ssl.RAND_bytes!". No. Don't do that, ssl.RAND_bytes() has its own set of issues, like two processes with the same pid producing the same random sequence... (old known issue, very hard to fix)

Python cannot workaround OS limitations, we can only do our best to use the most secure source of entropy. That's why Python 3.5 now uses getrandom() on Linux. That's why Python 3.6 now calls getrandom() in blocking mode.
msg277115 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-09-21 09:15
With the 3.6 os.urandom() implementation doing the right thing consistently cross-platform, our guidance for folks that care about the quality of the CSPRNG they use should be that they either upgrade to that version, or else ensure that the kernel CSPRNG is properly seeded before they run Python.

That is, I think the tone we're aiming for in the older docs now should be "You're using an older Python version, so if this problem description worries you, you need to either upgrade or else take the necessary steps to satisfy yourself that your host system's CSPRNG is properly configured", rather than the more passive "os.urandom() isn't necessarily secure" (with minimal guidance on what to do about it) that we've previously adopted.
msg404487 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-10-20 16:15
2.7 and 3.5 are no longer supported
History
Date User Action Args
2022-04-11 14:58:32adminsetgithub: 71479
2021-10-20 16:15:24christian.heimessetstatus: open -> closed

nosy: + christian.heimes
messages: + msg404487

resolution: out of date
stage: patch review -> resolved
2016-09-23 22:03:33ned.deilysetnosy: - ned.deily
2016-09-21 09:15:14ncoghlansetmessages: + msg277115
2016-09-20 21:28:44vstinnersetmessages: + msg277073
2016-09-10 08:27:49ncoghlansetmessages: + msg275600
2016-09-09 07:01:00vstinnersetmessages: + msg275269
2016-09-09 05:13:01martin.pantersetmessages: + msg275260
2016-09-07 03:01:16vstinnersetmessages: + msg274723
2016-09-07 02:34:23ncoghlansetassignee: docs@python

components: + Documentation, - Library (Lib)
title: Warn users that os.urandom() can return insecure values -> Warn users that os.urandom() prior to 3.6 can return insecure values
nosy: + docs@python, ncoghlan
versions: - Python 3.6
messages: + msg274710
2016-06-18 01:27:55martin.pantersetfiles: + urandom-doc.v2.patch

messages: + msg268752
stage: needs patch -> patch review
2016-06-16 13:24:55dstufftsetnosy: - dstufft
2016-06-16 13:14:27vstinnersetmessages: + msg268666
2016-06-16 13:08:56martin.pantersetfiles: + urandom-doc.patch

messages: + msg268663
2016-06-16 12:54:08vstinnersetmessages: + msg268662
2016-06-16 11:01:24martin.pantersetfiles: + urandom-doc.patch
keywords: + patch
messages: + msg268656
2016-06-16 08:11:05martin.pantersetnosy: + martin.panter

messages: + msg268649
versions: + Python 2.7
2016-06-12 11:33:38christian.heimessetnosy: - christian.heimes
2016-06-12 02:58:53larrysetpriority: release blocker -> normal

messages: + msg268316
2016-06-11 18:36:48larrysetmessages: + msg268244
2016-06-11 18:34:28dstufftsetmessages: + msg268242
2016-06-11 18:11:41larrysetmessages: + msg268239
2016-06-11 18:07:17larrysetnosy: + georg.brandl
messages: + msg268238
2016-06-11 12:48:49Lukasasetnosy: + Lukasa
2016-06-11 10:33:29steven.dapranosetnosy: + steven.daprano
messages: + msg268207
2016-06-11 10:07:53christian.heimescreate