classification
Title: os.urandom() should use Linux 3.17 getrandom() syscall
Type: security Stage: resolved
Components: Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: 700eb415, Arfrever, alex, anand.jeyahar, christian.heimes, haypo, josh.r, jwilk, neologix, pitrou, python-dev
Priority: normal Keywords: patch

Created on 2014-08-10 23:32 by haypo, last changed 2015-03-30 11:42 by haypo. This issue is now closed.

Files
File name Uploaded Description Edit
random.patch haypo, 2015-03-03 02:04 review
random-2.patch haypo, 2015-03-04 16:38
Messages (23)
msg225171 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-08-10 23:32
The future Linux kernel 3.17 will have a new getrandom() syscall which avoids the need of a file descriptor:
http://lwn.net/Articles/606141/

The file descriptor of os.urandom() causes perfomance issues and surprising bugs: #18756, #21207.

I don't know when the function will land in the libc.

OpenBSD 5.6 (not released yet) will also have a new getentropy() syscall.

For Python 2.7, see also the PEP 466 and the issue #21305.
msg225363 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-08-15 20:13
Manual page of the OpenBSD getentropy() function:
http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/getentropy.2

LibreSSL didn't wait for the libc, search for getentropy_getrandom():
http://openbsd.cs.toronto.edu/cgi-bin/cvsweb/src/lib/libcrypto/crypto/getentropy_linux.c?rev=1.32&content-type=text/x-cvsweb-markup

The code is currently disabled with "#if 0". The syscall is directly used, the function doesn't handle the ENOSYS error.

See also this issue of the cryptography project, "Use getentropy(2) and getrandom(2) syscalls when available 1299":
https://github.com/pyca/cryptography/issues/1299
msg228250 - (view) Author: (700eb415) Date: 2014-10-02 18:03
It's worth noting that LibreSSL has now enabled the blocked code. If anyone is interested, I would be willing to help port it.
msg228639 - (view) Author: anand jeyahar (anand.jeyahar) Date: 2014-10-06 05:02
Hi, 
   This will need latest kernel to develop, fix and test. I (on Debian 7) couldn't find the latest kernel, but picked up ubuntu kernel from here http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.17-rc7-utopic/. I picked up the latest i.e: linux-image-3.17.0-031700rc7-generic_3.17.0-031700rc7.201409281835_amd64.deb and installed manually, but couldn't find the getrandom() function call either in stdlib.h or linux/random.h. 

Can anyone confirm it's availability in a kernel image (from some other distribution?).
msg228645 - (view) Author: Charles-Fran├žois Natali (neologix) * (Python committer) Date: 2014-10-06 07:22
Note that I'm not fussed about it: far from simplifying the code, it
will make it more complex, thus more error-prone.
msg228655 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-10-06 10:24
The Linux kernel 3.17 has been released with the new getrandom() syscall.

glibc request to implement the function in the C library:
https://sourceware.org/bugzilla/show_bug.cgi?id=17252
"Bug 17252 - getrandom and getentropy syscall"

It looks like nobody asks for it on the libc-alpha mailing list yet.
msg228785 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2014-10-08 10:51
Let's not be early adopters here. I suggest we wait until glibc has a proper interface.
msg228790 - (view) Author: (700eb415) Date: 2014-10-08 13:59
OpenBSD already provides high quality pseudorandom numbers from arc4random(). I don't think this would make us "early adopters" since it has been around for some time on this platform.

It's also worth mentioning that getentropy() is not recommended in use for normal code as per stated in the man page. arc4random() is recommended, but there may be a reason the first poster has recommended getentropy()
msg228791 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-08 14:06
This issue is about Linux support. Does the glibc have arc4random? I can't find it on my Ubuntu 13.10 system.
msg228792 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2014-10-08 14:20
As I said on the other ticket, using arc4random() indiscriminately would be a very poor idea, on some platforms (such as OS X) arc4random() really does use ARC4, which means there are serious security concerns with it.
msg228794 - (view) Author: (700eb415) Date: 2014-10-08 14:29
While I agree it may not be wise to use arc4random() globally, OpenBSD is unlikely to create a duplicate interface since it's already available.

Python is currently unusable in chroots on that platform without reducing the security of the host partition by removing the nodev mount flag.

I feel like there must be a good solution to this.
msg228795 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-08 14:30
Since this is a Linux-specific issue (see the title), you should create a separate issue for OpenBSD support. Bonus points if you want to submit a patch as well :-)
msg228847 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-10-09 09:27
This issue is specific to Linux: it depends on the Linux kernel version and we are waiting until the new syscall is available in the C library (especially the glibc). For these reasons, I prefer to open a new specific issue for OpenBSD, since they release the kernel and C library at the same time (different release process): issue #22585. OpenBSD 5.6 scheduled in one month will get the new getentropy() syscall and a new getentropy() function at the same time.
msg228850 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-10-09 09:45
> Since this is a Linux-specific issue (see the title), you should create a separate issue for OpenBSD support.

700eb415 opened the issue #22542 for arc4random().
msg237102 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-03 02:04
Commit in the Linux kernel:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c6e9d6f38894798696f23c8084ca7edbf16ee895

--

Here is a patch to use the new getrandom() syscall of Linux 3.17 in the Python function os.urandom().

The function falls back to reading /dev/urandom if getrandom() is not supported (returns ENOSYS at runtime).

On my Linux 3.18, the EINTR path is never taken. But I was able to test it manually by setting flags to GRND_RANDOM (2) and injecting many signals using signal.setitimer(): see my http://bugs.python.org/issue23285#msg237100
msg237190 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-04 16:38
random-2.patch: updated patch (I don't understand why random.patch doesn't apply cleanly).
msg238440 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-18 13:42
New changeset 1fc32bf069ff by Victor Stinner in branch 'default':
Issue #22181: On Linux, os.urandom() now uses the new getrandom() syscall if
https://hg.python.org/cpython/rev/1fc32bf069ff
msg238504 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2015-03-19 12:43
Oh, test_os now fails on Linux because os.urandom() doesn't use a file descriptor anymore. The test should be skipped when getrandom() is used. The test is already skipped when getentropy() is used.
msg238559 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-19 21:23
New changeset 4491bdb6527b by Victor Stinner in branch 'default':
Issue #22181: The availability of the getrandom() is now checked in configure,
https://hg.python.org/cpython/rev/4491bdb6527b
msg238568 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-19 22:30
New changeset 8c73af0b3cd9 by Victor Stinner in branch 'default':
Issue #22181: Fix dev_urandom_noraise(), try calling py_getrandom() before
https://hg.python.org/cpython/rev/8c73af0b3cd9
msg238676 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2015-03-20 14:07
> New changeset 4491bdb6527b by Victor Stinner in branch 'default':
> Issue #22181: The availability of the getrandom() is now checked in configure,
> https://hg.python.org/cpython/rev/4491bdb6527b

You forgot to run aclocal, which resulted in PKG_PROG_PKG_CONFIG not being expanded in configure.
msg238688 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-20 15:03
New changeset b8ceb071159f by Victor Stinner in branch 'default':
Issue #22181: Run "aclocal; autoconf; autoheader" to regenerate configure
https://hg.python.org/cpython/rev/b8ceb071159f
msg239585 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-30 09:20
New changeset 28b465d8c519 by Victor Stinner in branch 'default':
Issue #22181: os.urandom() now releases the GIL when the getrandom()
https://hg.python.org/cpython/rev/28b465d8c519
History
Date User Action Args
2015-03-30 11:42:45hayposetstatus: open -> closed
resolution: fixed
2015-03-30 09:20:16python-devsetmessages: + msg239585
2015-03-20 15:03:32python-devsetmessages: + msg238688
2015-03-20 14:07:42Arfreversetmessages: + msg238676
2015-03-19 22:30:20python-devsetmessages: + msg238568
2015-03-19 21:23:01python-devsetmessages: + msg238559
2015-03-19 12:43:17hayposetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg238504
2015-03-19 10:56:32Arfreversetstage: needs patch -> resolved
2015-03-18 13:42:55hayposetstatus: open -> closed
resolution: fixed
2015-03-18 13:42:37python-devsetnosy: + python-dev
messages: + msg238440
2015-03-04 16:38:32hayposetfiles: + random-2.patch

messages: + msg237190
2015-03-03 02:04:17hayposetfiles: + random.patch
keywords: + patch
messages: + msg237102
2014-10-09 09:45:14hayposetmessages: + msg228850
2014-10-09 09:27:57hayposetmessages: + msg228847
2014-10-09 09:23:14hayposettype: enhancement -> security
2014-10-08 14:30:54pitrousetmessages: + msg228795
2014-10-08 14:29:56700eb415setmessages: + msg228794
2014-10-08 14:20:22alexsetmessages: + msg228792
2014-10-08 14:06:54pitrousetnosy: + pitrou
messages: + msg228791
2014-10-08 14:04:05Arfreversetnosy: + Arfrever
2014-10-08 13:59:57700eb415setmessages: + msg228790
2014-10-08 10:51:20christian.heimessetmessages: + msg228785
2014-10-06 11:46:04jwilksetnosy: + jwilk
2014-10-06 10:24:01hayposetmessages: + msg228655
2014-10-06 07:22:32neologixsetmessages: + msg228645
2014-10-06 05:02:37anand.jeyaharsetnosy: + anand.jeyahar
messages: + msg228639
2014-10-02 18:03:18700eb415setnosy: + 700eb415
messages: + msg228250
2014-08-18 00:22:24josh.rsetnosy: + josh.r
2014-08-15 20:13:55hayposetmessages: + msg225363
2014-08-11 09:12:26christian.heimessetnosy: + christian.heimes
2014-08-10 23:42:57pitrousettype: enhancement
stage: needs patch
2014-08-10 23:42:25pitrousetnosy: + neologix
2014-08-10 23:34:42alexsetnosy: + alex
2014-08-10 23:32:43haypocreate