classification
Title: PEP 524: Add os.getrandom()
Type: enhancement Stage: resolved
Components: Documentation, Library (Lib) Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: christian.heimes, docs@python, martin.panter, ncoghlan, python-dev, vstinner
Priority: low Keywords: patch

Created on 2016-08-16 17:25 by vstinner, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Files
File name Uploaded Description Edit
getrandom.patch vstinner, 2016-08-16 17:25 review
getrandom_errno.patch vstinner, 2016-09-20 21:07 review
Pull Requests
URL Status Linked Edit
PR 552 closed dstufft, 2017-03-31 16:36
Messages (17)
msg272867 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-08-16 17:25
Attached patch adds os.getrandom(): thin wrapper on the Linux getrandom() syscall.

os.getrandom() can return less bytes than requested.

The patch is incomplete: it doesn't include documentation.

I chose to not implement a loop to not loose entropy if a following call fails (ex: fail with EINTR). Rationale:
https://mail.python.org/pipermail/security-sig/2016-July/000072.html

We should also add Solaris support later.

See also #27776: "PEP 524: Make os.urandom() blocking on Linux".
msg273024 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-08-18 12:27
Given docs (with the Linux-only platform support disclaimer), +1 for this as an initial implementation.

Providing it on Solaris as well can be a separate patch, but it's less important there (since /dev/urandom and os.urandom() are already blocking APIs)
msg274662 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-06 23:20
New changeset 27267d2fb091 by Victor Stinner in branch 'default':
Add os.getrandom()
https://hg.python.org/cpython/rev/27267d2fb091
msg274698 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-09-07 01:44
HAVE_GETRANDOM_SYSCALL seems to be a compile-time library check, not a runtime check. I compiled and run on Linux 3.15.5, and os.getrandom() exists but raises ENOSYS:

======================================================================
ERROR: test_getrandom0 (test.test_os.GetRandomTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/media/disk/home/proj/python/cpython/Lib/test/test_os.py", line 1280, in test_getrandom0
    empty = os.getrandom(0)
OSError: [Errno 38] Function not implemented
msg274717 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-09-07 02:49
Huh, I thought I'd already filed an issue for that, but it looks like it was only a security-sig thread: https://mail.python.org/pipermail/security-sig/2016-June/000060.html

I've now remedied that omission and filed http://bugs.python.org/issue27990 to cover it explicitly.

Since that was a pre-existing problem that also happens to affect this API, rather than something new introduced by Victor's patch, closing this again.
msg274722 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-07 02:59
New changeset 7a243a40b421 by Victor Stinner in branch 'default':
Fix test_os.GetRandomTests()
https://hg.python.org/cpython/rev/7a243a40b421
msg274724 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-09-07 03:03
> HAVE_GETRANDOM_SYSCALL seems to be a compile-time library check, not a runtime check. I compiled and run on Linux 3.15.5, and os.getrandom() exists but raises ENOSYS:

Oh, I'm surprised the configure sees getrandom() as available. But well ok, the error can occur if you compile Python on a more recent kernel than the running kernel.

I fixed the unit test: skip getrandom() tests if getrandom() fails with ENOSYS.

Do you think that it's worth to document this case?
msg274730 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-09-07 03:12
Ah, I'd missed that Martin was talking about the other way around from #27990.

Yes, I think it's worth documenting that os.getrandom() may raise OSError if the running kernel doesn't provide the syscall - that's going to be pretty easy to trigger by running a container with Python 3.6 on a container host running an older Linux kernel.
msg274733 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-09-07 03:12
I run Arch Linux, but only update packages when I have to. As a result, I am running Linux 3.15 installed and running, but the linux-api-headers has more recently been updated to 4.7 (i.e. matching Linux 4.7).
msg276323 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-09-13 18:53
3.6 is in beta phase. Are you interested to add the feature to 3.7?
msg276356 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-09-13 22:20
I understand it’s already implemented, and Victor just reopened it for more documentation.
msg276386 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-09-14 06:40
Right, the only missing piece now is documentation of the ENOSYS case, which end users may encounter if a Python 3.6 binary that supports os.getrandom() is run against an older kernel.

That's pretty easy to trigger via containers, as getrandom() was added in Linux 3.17 and hasn't generally been backported to LTS distribution kernels.

Debian 8: based on 3.16
Ubuntu 14.04: 3.13 default, 4.4 (from 16.04) available as of 14.04.5
RHEL/CentOS 7: based on 3.10
RHEL/CentOS 6: based on 2.6

So of those potential LTS container hosts, a recent Ubuntu or Fedora container running Python 3.6 will currently get ENOSYS for everything except a fresh Ubuntu 14.04 install that uses the Ubuntu 16.04 kernel.
msg276389 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-09-14 07:41
Oh sorry, I looked in the wrong location and missed it.

* if (PyErr_CheckSignals() < 0) {return NULL;} does not free buffer with PyMem_Free(buffer);

* The function allocates memory once with PyMem_Malloc() and later a second time with PyBytes_FromStringAndSize(buffer, n). You can avoid the first allocation and a memcpy() with PyBytes_FromStringAndSize(NULL, n) and PyBytes_AS_STRING().

* The syscall can also raise EPERM as reported by a user on QNAP. IIRC a seccomp policy caused EPERM.
msg277068 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-20 21:01
New changeset d31b4de433b7 by Victor Stinner in branch '3.6':
Fix memleak in os.getrandom()
https://hg.python.org/cpython/rev/d31b4de433b7
msg277070 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-09-20 21:07
I pushed the fix for the issue #27955, os.urandom() now handles getrandom() failing with EPERM.

@Christian: Thanks for your review, I pushed a change fixing the two issues that you reported (memory leak and inefficient temporarily buffer).

I attached getrandom_errno.patch: a change proposing to document ENOSYS and EPERM. What do you think?
msg277335 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-09-24 21:30
I think the documentation is too specific. We typically don't document all possible error numbers. Something along the lines "fails with OSError when getrandom is not supported" is sufficient.
msg278815 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-10-17 16:21
Because of the lack of interest for getrandom_errno.patch, and Christian saying that it's not good to document specific errors, I now close the bug.

Thank you all for your help on this nice security enhancement in Python 3.6!
History
Date User Action Args
2017-03-31 16:36:09dstufftsetpull_requests: + pull_request848
2016-10-17 16:21:08vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg278815
2016-09-24 21:30:03christian.heimessetpriority: normal -> low

type: security -> enhancement
assignee: docs@python
components: + Documentation

nosy: + docs@python
messages: + msg277335
2016-09-20 21:07:21vstinnersetfiles: + getrandom_errno.patch

messages: + msg277070
2016-09-20 21:01:52python-devsetmessages: + msg277068
2016-09-14 07:41:05christian.heimessetmessages: + msg276389
versions: + Python 3.6
2016-09-14 06:40:51ncoghlansetmessages: + msg276386
2016-09-13 22:20:49martin.pantersetmessages: + msg276356
2016-09-13 18:53:20christian.heimessetnosy: + christian.heimes

messages: + msg276323
versions: + Python 3.7, - Python 3.6
2016-09-07 03:12:59martin.pantersetmessages: + msg274733
2016-09-07 03:12:34ncoghlansetmessages: + msg274730
2016-09-07 03:03:43vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg274724
2016-09-07 02:59:53python-devsetmessages: + msg274722
2016-09-07 02:49:13ncoghlansetstatus: open -> closed

messages: + msg274717
stage: resolved
2016-09-07 01:44:03martin.pantersetstatus: closed -> open
nosy: + martin.panter
messages: + msg274698

2016-09-06 23:44:27vstinnersetstatus: open -> closed
resolution: fixed
2016-09-06 23:20:07python-devsetnosy: + python-dev
messages: + msg274662
2016-08-18 12:27:32ncoghlansetnosy: + ncoghlan
messages: + msg273024
2016-08-16 17:25:41vstinnercreate