classification
Title: PEP 466: update os.urandom
Type: enhancement Stage: resolved
Components: Extension Modules, Interpreter Core Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: alex, benjamin.peterson, christian.heimes, dstufft, giampaolo.rodola, haypo, janssen, josh.r, ncoghlan, neologix, python-dev, tshepang
Priority: normal Keywords: needs review, patch

Created on 2014-04-19 00:51 by ncoghlan, last changed 2014-08-28 16:30 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
backport-urandom.diff alex, 2014-08-22 17:10
backport-urandom.diff alex, 2014-08-28 14:02
backport-urandom.diff alex, 2014-08-28 14:26 review
Messages (31)
msg216824 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-04-19 00:51
Tracker issue for the os.urandom persistent file descriptor backport to 2.7 described in PEP 466.
msg216852 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-19 09:10
Why exactly does it need to be backported? os.urandom under 2.7 works fine.
msg216858 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-04-19 13:59
It was in the list of security fixes Alex and Donald wanted to reduce
vulnerabilities in 2.x network services, and Guido was OK with backporting
it.
msg216859 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-19 14:05
> It was in the list of security fixes Alex and Donald wanted to reduce
> vulnerabilities in 2.x network services, and Guido was OK with backporting
> it.

What security issue is there exactly? os.urandom() does a similar thing
in 2.7 and 3.x (it reads from  /dev/urandom).
msg216860 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2014-04-19 14:34
It's not a security issue per-se, but if you're doing many small reads, there's such an enormous performance and scalability difference that if users run into an issue, they're likely to work around it by using a non-CS PRNG, and compromising their application security.
msg216861 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-19 14:39
> It's not a security issue per-se, but if you're doing many small
> reads, there's such an enormous performance and scalability difference
> that if users run into an issue, they're likely to work around it by
> using a non-CS PRNG, and compromising their application security.

Do you have any examples of this or is it just a gratuitous inference?
msg216862 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-19 14:41
Note that the 3.4 scheme is not fully debugged yet: issue21207. There is a reason we don't backport new features!

Regardless, I'm not interested in this, so I'll let you take the risk of regressions if you want to.
msg217281 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2014-04-27 16:31
Like Antoine, I'm really skeptical about the backport: honestly, this change doesn't bring much in a normal application. To run into the number of open file descriptors limit (so the "scalability" aspect), one would need to have *many* concurrent threads reading from /dev/urandom. For the "performance" aspect, I have a hard time believing that the overhead of the extra open() + close() syscalls is significant in a realistic workload. If reading from /dev/urandom becomes a bottleneck, this means that you're depleting your entropy pool anyway, so you're in for some potential trouble...

> There is a reason we don't backport new features!

Couldn't agree more. This whole "let's backport security enhancements" sounds scary to me.
msg217292 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-04-27 17:37
Yep, it's scary indeed, but such a long lived feature release is a novel
situation that may require some adjustments to our risk management.

However, we can still decide to defer some of the changes until 2.7.8, even
though the notion of backporting them has been approved in principle.

For 2.7.7, we should probably focus on the more essential SSL enhancements.
msg217364 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2014-04-28 11:21
"Depleting" /dev/urandom isn't actually a thing. /dev/urandom on all modern *nix OSs uses a fast PRNG which is secure as long as it has received enough bytes of initial entropy.
msg217366 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2014-04-28 11:51
> "Depleting" /dev/urandom isn't actually a thing. /dev/urandom on all modern *nix OSs uses a fast PRNG which is secure as long as it has received enough bytes of initial entropy.

I didn't say "deplete /dev/urandom", I said that when reading from
/dev/urandom "you're depleting your entropy pool". So reading from
/dev/urandom won't block, but it can starve processes that read from
/dev/random, and that's a problem.

See https://groups.google.com/forum/#!msg/fa.linux.kernel/Ocl01d8TzT0/KDCon2ZUm1AJ

I think since 2.6 Linux uses two different entropy pools for
/dev/random and /dev/urandom, but that might not be true for every OS.
msg217369 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2014-04-28 12:52
I don't think what you're worrying about here is something that has a high chance of happening, if it even occurs in the wild at all. To be clear in order for that to matter at all in the context of this ticket, some software would need to be reading from /dev/random (which almost zero software should be doing) on a system where you have a high number of threads or async handlers all reading from /dev/urandom at the same time and reading enough data off of it to drop the entropy estimation in the primary pool down below whatever amount of data that the other process is attempting to read from /dev/random. In that case no "trouble" will occur and the process reading from /dev/random will just block waiting on additional entropy to be collected so that the entropy estimation is high enough to fulfill the request.

AFAIK there are zero practical concerns from reading as much as you want off of /dev/urandom as often as you want.

What this change does is make it "safe" to just simply use os.urandom in order to generate random bytes in a Python application. The current situation is such that any use of os.urandom is potentially a place where your application will crash in highly concurrent environments. This will drive people to either:

A) Poorly reimplement the persistent FD option, especially troublesome on Windows because the simple approach to doing so will flat out not work on Windows
B) Use a userspace CSPRNG, this is considered ill advised by most reputable cryptographer's as most of them have had issues at one point in time or another, and they all generally depend on the security of /dev/urandom anyways so if /dev/urandom fall they fall as well.

Using os.urandom is the *right* thing to do for getting random in an application, but the current implementation effectively punishes people who use it if their application is highly concurrent.
msg217372 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2014-04-28 13:34
> Using os.urandom is the *right* thing to do for getting random in an application, but the current implementation effectively punishes people who use it if their application is highly concurrent.

And I argue that this scenario is almost as likely as the one you
depict above: we never had a bug report before, and if you have a look
at the the bug report which led to the change in question, it's not
clear at all that all threads were indeed reading from /dev/urandom
when EMFILE was raised. Since reading from /dev/urandom shouldn't
block, it's not clear at all how a realistic workload would actually
hit the file descriptor limit because RLIMIT_NOFILE threads are
reading from /dev/urandom.

But don't get me wrong, I'm not saying this change is useless, it
actually makes sense to use a persistent FD. But backporting always
has a risk, which has to be balanced.
msg217374 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2014-04-28 13:38
> But backporting always has a risk, which has to be balanced.

Sure, which is why a PEP was written, discussed and accepted to find that balance.
msg217425 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-04-28 21:51
Please don't backport this "feature". We had to wait 20 years before someone requested the feature, but only a few months before the first user reported an issue ("regression"?).

IMO it would be much better to use explicitly a random.SystemRandom instance which would keep the fd open, and only use it if you hit the bug (*many* threads calling os.urandom in parallel.
msg217429 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2014-04-28 22:22
Well except random.SystemRandom doesn't keep the file open (At least in 2.7) and actually it just calls os.urandom under the covers, also it doesn't make it very nice to get a glob of random bytes.
msg217432 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2014-04-28 22:26
Just verified that 3.x also does not exhibit this behavior with random.SystemRandom (except implicitly through os.urandom doing it).
msg217470 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-04-29 05:44
Le 29 avr. 2014 00:22, "Donald Stufft" <report@bugs.python.org> a écrit :
> Well except random.SystemRandom doesn't keep the file open (At least in
2.7) and actually it just calls os.urandom under the covers, also it
doesn't make it very nice to get a glob of random bytes.

Yes, I'm proposing to enhance this class.
msg217472 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2014-04-29 05:56
> Yes, I'm proposing to enhance this class.

The problem is AFAICT there's currently no way to get a file
descriptor to the underlying /dev/urandom (and I don't know how it
works on Windows).

Also, this would duplicate the work which has already been done.
msg217476 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2014-04-29 06:29
One of the reasons the PEP was done the way it was done was it allowed you to write 2/3 compatible code without version checks. Enhancing that class won't land until 3.5 which is 18+ months away. Further more the os.urandom persistent FD's already exists and is a complete superset of functionality that a random.SystemRandom random would have.

This conversation would have been a lot more useful when the PEP was being discussed on python-dev.
msg217478 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-04-29 07:06
Note that the discussion of this PEP *did* suffer from the "language summit
effect" where folks that couldn't make it to the summit are missing some of
the context. I believe I included all of the key motivating points in the
PEP itself, but it's still not the same as being there from a "why take the
risk?" perspective.

Of the approved feature backports, this one was the most borderline, and as
noted above, I think it could comfortably wait until 2.7.8. The SSL changes
are most significant, followed by the constant time comparison function,
and only then the other updates.
msg217494 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-04-29 08:37
> The problem is AFAICT there's currently no way to get a file
> descriptor to the underlying /dev/urandom (and I don't know how it
> works on Windows).

We can reimplement os.urandom in SystemRandom on UNIX to keep the file (fd)
open. The code is very simple, basically it's just a call to file.read(n).

Adding a randbytes() method in Python 3.5 would be nice.

The io module can handle boring things for you, like calling read in a loop
until you get enough bytes and handle InterruptError.

Except if you would prefer to use os.read or FileIO.read to avoid readahead.
msg217496 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-04-29 08:41
> (and I don't know how it
> works on Windows).

On Windows, the OS CryptoAPI is used and a handle is kept open between
calls to os.urandom. On Windows, I don't think that it's a an issue to keep
a handle open. Handle are not sequential numbers and users don't manipulate
them directly.
msg225172 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-08-10 23:33
See also the issue #22181: "os.urandom() should use Linux 3.17 getrandom() syscall".
msg225684 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2014-08-22 17:10
Attached patch backports the persistent FD for urandom.
msg226025 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2014-08-28 13:56
You should probably backport _PyRandom_Fini and cleanup the FD like a good citizen.
msg226026 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2014-08-28 14:02
This patch adds the finalizer to the backport -- not sure how I missed this the first time.
msg226027 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-08-28 14:04
@alex: please disable git format in your hgrc, so the bug tracker can create a "review" link to Rietveld for your patches.
msg226028 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2014-08-28 14:26
Victor -- new patch is in `hg` format.
msg226029 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-08-28 14:41
The third  backport-urandom.diff  (the one with the review link) looks good to me.
msg226031 - (view) Author: Roundup Robot (python-dev) Date: 2014-08-28 16:30
New changeset 3e7f88550788 by Benjamin Peterson in branch '2.7':
PEP 466: backport persistent urandom fd (closes #21305)
http://hg.python.org/cpython/rev/3e7f88550788
History
Date User Action Args
2014-08-28 16:30:19python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg226031

resolution: fixed
stage: needs patch -> resolved
2014-08-28 14:41:51hayposetmessages: + msg226029
2014-08-28 14:26:54alexsetfiles: + backport-urandom.diff

messages: + msg226028
2014-08-28 14:04:19hayposetmessages: + msg226027
2014-08-28 14:02:53alexsetfiles: + backport-urandom.diff

messages: + msg226026
2014-08-28 13:56:26benjamin.petersonsetmessages: + msg226025
2014-08-22 17:10:47alexsetkeywords: + patch, needs review
files: + backport-urandom.diff
messages: + msg225684

components: + Extension Modules, Interpreter Core
2014-08-10 23:33:19hayposetmessages: + msg225172
2014-04-29 08:41:11hayposetmessages: + msg217496
2014-04-29 08:37:46hayposetmessages: + msg217494
2014-04-29 07:06:25ncoghlansetmessages: + msg217478
2014-04-29 06:29:47dstufftsetmessages: + msg217476
2014-04-29 05:56:47neologixsetmessages: + msg217472
2014-04-29 05:44:26hayposetmessages: + msg217470
2014-04-28 22:26:40dstufftsetmessages: + msg217432
2014-04-28 22:22:37dstufftsetmessages: + msg217429
2014-04-28 21:51:59hayposetnosy: + haypo
messages: + msg217425
2014-04-28 13:38:36dstufftsetmessages: + msg217374
2014-04-28 13:34:01neologixsetmessages: + msg217372
2014-04-28 12:52:33dstufftsetmessages: + msg217369
2014-04-28 11:51:11neologixsetmessages: + msg217366
2014-04-28 11:21:17dstufftsetmessages: + msg217364
2014-04-27 17:37:48ncoghlansetmessages: + msg217292
2014-04-27 16:31:43neologixsetnosy: + neologix
messages: + msg217281
2014-04-25 16:56:56tshepangsetnosy: + tshepang
2014-04-19 14:41:39pitrousetnosy: - pitrou
2014-04-19 14:41:29pitrousetnosy: ncoghlan, janssen, pitrou, giampaolo.rodola, christian.heimes, benjamin.peterson, alex, dstufft, josh.r
messages: + msg216862
2014-04-19 14:39:22pitrousetmessages: + msg216861
2014-04-19 14:34:59alexsetmessages: + msg216860
2014-04-19 14:05:38pitrousetmessages: + msg216859
2014-04-19 13:59:13ncoghlansetmessages: + msg216858
2014-04-19 09:10:49pitrousetmessages: + msg216852
2014-04-19 02:21:32josh.rsetnosy: + josh.r
2014-04-19 00:51:14ncoghlancreate