classification
Title: OSError with "import random" when /dev/urandom doesn't exist (regression from 2.6)
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.1, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, barry, benjamin.peterson, bugs-python@vendor.thewrittenword.com, georg.brandl, iwienand, loewis, pitrou, python-dev, richardw
Priority: high Keywords: patch

Created on 2012-07-12 22:30 by iwienand, last changed 2013-12-22 17:50 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
hash-randomization-not-implemented.patch bugs-python@vendor.thewrittenword.com, 2012-08-31 10:39 hash-randomization-not-implemented.patch
hash-randomization-not-implemented.patch bugs-python@vendor.thewrittenword.com, 2012-09-04 04:53 Revised PyExc_NotImplemented hash randomization patch
Messages (19)
msg165340 - (view) Author: Ian Wienand (iwienand) Date: 2012-07-12 22:30
Hi,

Lib/random.py has a fallback if os.urandom() returns NotImplementedError

---
from os import urandom as _urandom
...
   def seed(self, a=None):
        if a is None:
            try:
                a = long(_hexlify(_urandom(16)), 16)
            except NotImplementedError:
                import time
                a = long(time.time() * 256) # use fractional seconds
---

In 2.6, this is indeed what happens in Lib/os.py where "import urandom from os" gets [2]:

---
if not _exists("urandom"):
    def urandom(n):
...
      try:
            _urandomfd = open("/dev/urandom", O_RDONLY)
        except (OSError, IOError):
            raise NotImplementedError("/dev/urandom (or equivalent) not found")
---

however, in 2.7, things have shuffled around as a result of issue Issue #13703 and now _PyOS_URandom will return an OSError if it can't find /dev/urandom [3].

This means if you "import random" without "/dev/urandom" available it crashes trying to seed

I'm not sure if this is intentional?  One easy solution would be to catch OSError in random.py and fall back then too

[1] http://hg.python.org/cpython/file/70274d53c1dd/Python/random.c#l227
[2] http://hg.python.org/cpython/file/9f8771e09052/Lib/os.py#l746
[3] http://hg.python.org/cpython/file/70274d53c1dd/Lib/random.py#l111
msg169507 - (view) Author: The Written Word (bugs-python@vendor.thewrittenword.com) Date: 2012-08-31 07:33
Actually, this regression appeared after the Hash Randomization patches prior to 2.6.8, 2.7.3, 3.1.4 and 3.2.3.

Also, it not only breaks `from os import urandom`, but also prevents installation of many third-party packages that use setuptools or distribute, where the interpreter bails out with: "OSError: No such file or directory /dev/urandom" inside setup.py on all Tru64 machines, and HPUX 11.00 and 11.11 (at least).

As best I can tell it's failing either because dev_urandom_noraise aborts the interpreter if /dev/urandom is missing, or later an uncaught PyExc_OSError in dev_urandom_python triggers for the same reason.  In either case there's no NotImplemented exception raised for the fallback code be used :(
msg169520 - (view) Author: The Written Word (bugs-python@vendor.thewrittenword.com) Date: 2012-08-31 10:39
The root of the problem preventing me from running some 3rd party setup.py scripts correctly is the mismatch between (recently) raising an OSError in Python/random.c, but catching only NotImplementedError in Lib/random.py.

For backwards compatibility (previous releases all raised and caught NotImplementedError), here is a patch that stops Python bailing out with "OSError: No such file or directory /dev/urandom" for me.  Tested with Python-2.6.8, Python-2.7.3 and Python-3.2.3, on HPUX 11.00, HPUX 11.11 and Tru 64 5.1.

Arguably, as long as 3rd party code doesn't rely on the NotImplementedError exception, changing random.py to catch OSErrors would work equally well.

This patch does not stop dev_urandom_noraise() from halting the interpreter on machines with no /dev/urandom, but that seems intentional to me so I didn't try to fix it.
msg169521 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-08-31 10:51
Looks like a critical regression to me.
msg169798 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-09-03 19:46
As for the patch: looks good on the principle. However, PyExc_NotImplementedError has no "errno" or "filename", attribute, so you shouldn't use PyErr_SetFromErrnoWithFilename. Instead, simply call PyErr_SetString.
msg169815 - (view) Author: The Written Word (bugs-python@vendor.thewrittenword.com) Date: 2012-09-04 04:53
Hi Antoine,

Thanks for the heads up.

I've attached a revised patch that doesn't misuse PyErr_SetFromErrnoWithFilename.
msg170000 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-09-07 18:10
I disagree that the regression is critical. IIUC, it fails on systems without urandom, such as Tru64 and HPUX. However, failure to support such systems is *not* critical, IMO; I think that OS-specific failures should be considered critical only if they occur on Linux, Windows, or OSX.

So I suggest that the priority of this issue is reduced.

More relevant than breaking HPUX is, IMO, that urandom is actually documented to raise NotImplementedError, so the patch looks good. For best compatibility, the actual spelling of the error message from 2.6 should be restored.

I'm puzzled by 

https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=KRNG11I

which claims that HPUX 11.11 (at least) *does* have /dev/urandom. Maybe your installation doesn't have KRNG11i in /etc/loadmods?

Also, the claim that it breaks Tru64 contradicts with

http://en.wikipedia.org/wiki//dev/random

which claims that Tru64 5.1B (at least) does have /dev/urandom.
msg170001 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-09-07 18:19
An interesting question is whether the patch should be applied to 2.6 and 3.1. It is not a security fix in itself, which suggests that it shouldn't apply. OTOH, it's a follow-up to an earlier security fix.
msg170002 - (view) Author: Ian Wienand (iwienand) Date: 2012-09-07 18:28
I'm not sure what systems are defined as critical or not.

Although python is not really installable/configurable by end-users on ESXi, I noticed during development because we use python very early in the boot, before /dev/urandom appears for us (it comes from a kernel module loaded later).
msg170004 - (view) Author: The Written Word (bugs-python@vendor.thewrittenword.com) Date: 2012-09-07 19:04
We're running Tru64 UNIX 5.1A, not 5.1B which definitely doesn't have /dev/urandom.
msg170005 - (view) Author: The Written Word (bugs-python@vendor.thewrittenword.com) Date: 2012-09-07 19:06
We do not have KRNG11i installed. It did not ship with the original installation of HP-UX 11.11. It needs to be loaded after-the-fact and we cannot be ensured that our customers will have this module installed nor do we wish to make it a requirement.
msg170014 - (view) Author: Roundup Robot (python-dev) Date: 2012-09-07 21:56
New changeset 2e8b01583839 by Antoine Pitrou in branch '3.2':
Issue #15340: Fix importing the random module when /dev/urandom cannot be opened.
http://hg.python.org/cpython/rev/2e8b01583839

New changeset a53fc9982e2a by Antoine Pitrou in branch 'default':
Issue #15340: Fix importing the random module when /dev/urandom cannot be opened.
http://hg.python.org/cpython/rev/a53fc9982e2a
msg170015 - (view) Author: Roundup Robot (python-dev) Date: 2012-09-07 21:58
New changeset edbf37ace03c by Antoine Pitrou in branch '2.7':
Issue #15340: Fix importing the random module when /dev/urandom cannot be opened.
http://hg.python.org/cpython/rev/edbf37ace03c
msg170016 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-09-07 22:00
Ok, this is now fixed in 3.2/3.3/2.7. I'll leave it to Martin and Benjamin whether this should be backported to 2.6 and 3.1.
(Georg, this changeset should probably be ported to 3.3.0 too)
msg170027 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-09-08 05:48
Now picked into the 3.3.0 release clone as 6a782496f90a.
msg170042 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-09-08 10:29
It's actually up to Barry (and Benjamin) to decide whether to apply this to 2.6 (and 3.1).
msg170086 - (view) Author: Roundup Robot (python-dev) Date: 2012-09-09 09:18
New changeset 6a782496f90a by Antoine Pitrou in branch 'default':
Issue #15340: Fix importing the random module when /dev/urandom cannot be opened.
http://hg.python.org/cpython/rev/6a782496f90a
msg180523 - (view) Author: Richard Wall (richardw) Date: 2013-01-24 14:00
This bug also causes problems when you try to install Python in a Linux  chroot environment or systemd-nspawn - before mounting devtmpfs.

For example, this Redhat bug "yum traceback with python-2.6.6-29.el6_2.2 and higher + missing /dev/urandom"
 * https://bugzilla.redhat.com/show_bug.cgi?id=893034
msg206823 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-12-22 17:50
2.6 and 3.1 don't receive bug fixes anymore, closing.
History
Date User Action Args
2013-12-22 17:50:18pitrousetstatus: open -> closed
resolution: fixed
messages: + msg206823

stage: resolved
2013-01-24 14:00:28richardwsetnosy: + richardw
messages: + msg180523
2012-09-09 09:18:53python-devsetmessages: + msg170086
2012-09-08 10:29:26loewissetmessages: + msg170042
2012-09-08 05:48:25georg.brandlsetmessages: + msg170027
2012-09-07 22:00:16pitrousetpriority: release blocker -> high

messages: + msg170016
versions: - Python 2.7, Python 3.2, Python 3.3
2012-09-07 21:58:47python-devsetmessages: + msg170015
2012-09-07 21:56:05python-devsetnosy: + python-dev
messages: + msg170014
2012-09-07 19:06:02bugs-python@vendor.thewrittenword.comsetmessages: + msg170005
2012-09-07 19:04:42bugs-python@vendor.thewrittenword.comsetmessages: + msg170004
2012-09-07 18:28:15iwienandsetmessages: + msg170002
2012-09-07 18:19:31loewissetmessages: + msg170001
2012-09-07 18:10:05loewissetnosy: + loewis
messages: + msg170000
2012-09-04 07:22:54Arfreversetnosy: + Arfrever
2012-09-04 04:53:44bugs-python@vendor.thewrittenword.comsetfiles: + hash-randomization-not-implemented.patch

messages: + msg169815
2012-09-03 19:46:39pitrousetmessages: + msg169798
2012-08-31 10:51:05pitrousetpriority: normal -> release blocker
versions: + Python 3.3
nosy: + pitrou, barry, benjamin.peterson, georg.brandl

messages: + msg169521
2012-08-31 10:39:32bugs-python@vendor.thewrittenword.comsetfiles: + hash-randomization-not-implemented.patch
keywords: + patch
messages: + msg169520

components: + Interpreter Core, - Library (Lib)
2012-08-31 07:33:45bugs-python@vendor.thewrittenword.comsetnosy: + bugs-python@vendor.thewrittenword.com

messages: + msg169507
versions: + Python 2.6, Python 3.1, Python 3.2
2012-07-12 22:30:28iwienandcreate