classification
Title: Virtualenv not portable from Python 2.7.2 to 2.7.3 (os.urandom missing)
Type: behavior Stage:
Components: Versions: Python 3.1, Python 3.2, Python 2.7, Python 2.6
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: barry, benjamin.peterson, carljm, dmalcolm, eric.araujo, eric.snow, georg.brandl, jason.coombs, loewis, r.david.murray
Priority: release blocker Keywords:

Created on 2012-03-29 16:41 by jason.coombs, last changed 2012-04-12 21:34 by jason.coombs. This issue is now closed.

Messages (16)
msg157066 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2012-03-29 16:41
Summary of this thread: http://mail.python.org/pipermail/python-dev/2012-March/118233.html

When upgrading a host from Python 2.6.7 to 2.6.8 or from 2.7.2 to 2.7.3, virtualenvs will no longer work if they attempt to use os.urandom. I have not yet investigated whether this affects Python 3.1 or 3.2.

On a related note, it appears that downgrading (taking a virtualenv created on Python 2.7.3 and deploying it to a host with 2.7.2) results in a similar error where _socket cannot be imported.

Based on the discussion, I've drafted some release notes (and Carl has edited them) to be included with the appropriate releases to help inform those this issue might impact.

http://piratepad.net/PAZ3CEq9CZ

I humbly request that these notes be included in the releases.
msg157067 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2012-03-29 16:42
3.1 and 3.2 are affected as well.
msg157079 - (view) Author: Carl Meyer (carljm) * Date: 2012-03-29 18:25
Alternatively, the conditional definition of urandom in os.py (removed in http://hg.python.org/cpython/rev/a0f43f4481e0#l7.1) could be reintroduced, allowing the new stdlib to be used with older interpreters. (Thanks to Dave Malcolm for pointing this out.) This seems like perhaps a reasonable concession to backwards compatibility for a bugfix release.
msg157081 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2012-03-29 18:42
On Mar 29, 2012, at 06:25 PM, Carl Meyer wrote:

>
>Carl Meyer <carl@dirtcircle.com> added the comment:
>
>Alternatively, the conditional definition of urandom in os.py (removed in
>http://hg.python.org/cpython/rev/a0f43f4481e0#l7.1) could be reintroduced,
>allowing the new stdlib to be used with older interpreters. (Thanks to Dave
>Malcolm for pointing this out.) This seems like perhaps a reasonable
>concession to backwards compatibility for a bugfix release.

Seems reasonable to me too.  I support making this change in 2.6.8.  I won't
personally have time to do that for the next week or so, so if someone else is
applying this to the other stable versions, I hereby give permission to apply
this to the 2.6 branch.
msg157087 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-03-29 19:12
I'm not so sure that it is desirable to make it compatible. It is ultimately virtualenv's "fault" to use the 2.7.3 library with a 2.7.2 binary. If we get this to "work", people will still not gain the hash randomization. IOW, they get the library update, but not the interpreter update fixing the security issue. They will then find that if they create a new virtualenv, their code may suddenly break because of the hash randomization.

Ideally, virtualenv would have arranged to incorporate an upgrade to the executable automatically. Given that it doesn't, the failure mode sounds ok to me.
msg157090 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2012-03-29 19:42
Martin makes a good point, but I see it somewhat differently.

virtualenv and its users have always accepted the risk of running an old interpreter against a different standard library (of the same minor version). So the risk of not receiving the security patch in the interpreter is well-known.

The risk they have not (previously) accepted (afaik) is that an interpreter of one patch version will not be compatible with the standard library of another patch version.

I could very well be wrong about the latter.

While I think we all agree that this is not a bug in Python, per se, the more practical matter is that this issue is likely to cause substantial trouble in practice, perhaps an unprecedented experience. I would hate for all the hard work that was put into this security fix to be tainted by cries of trouble caused by the fix (however unjustified). Providing backward-compatibility for virtualenv would avoid that risk and would not expose the users of virtualenv to any more risk than they've previously accepted.

For that reason, I'm +1 on the compatibility patch(es).
msg157097 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-03-29 20:48
Running the python interpreter against a different version of the stdlib is completely unsupported, and I'm surprised it hasn't broken more. I'm rejecting any shims for it for 2.7 and 3.1.
msg157100 - (view) Author: Carl Meyer (carljm) * Date: 2012-03-29 21:10
There's no question that this is a case of virtualenv allowing users to do something that's not supported.

Nonetheless, virtualenv is very widely used, and in practice it does not break "more often". This, however, will break for lots of users, and those users will (wrongly) perceive the breakage to be caused by a Python security bugfix, not by virtualenv.

I think the purity argument is completely correct, but this certainly seems like a situation where practicality might nonetheless beat purity.
msg157101 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-03-29 21:12
This "bug" will only happen in virtualenvs, though, so I expect virtualenv will be blamed.
msg157102 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2012-03-29 21:14
The consensus from the release managers seems to be "don't patch", so I propose we move forward with including the release notes as drafted.
msg157105 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2012-03-29 21:23
My mistake - Georg has not yet weighed in. Perhaps it's best to wait for his opinion.
msg157106 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-03-29 21:27
I am with Benjamin.  First we put a lot of effort into fixing this issue, and then we're letting quite a lot of deployments (and I assume that quite a lot of deployments *are* in virtualenvs) "escape" the fix?  That doesn't sound right.
msg157107 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2012-03-29 21:42
On Mar 29, 2012, at 09:24 PM, Jason R. Coombs wrote:

>My mistake - Georg has not yet weighed in. Perhaps it's best to wait for his
>opinion.

I'm happy to go along with RM consensus.  It should work the same in all
stable releases.
msg157108 - (view) Author: Carl Meyer (carljm) * Date: 2012-03-29 21:53
I'd been thinking the "escape the security fix" argument didn't apply, because the security fix requires opt-in anyway and the -R flag would fail immediately on a non-updated virtualenv.

But there is also the environment variable. It is quite possible that someone could update their system Python, set PYTHONHASHSEED and think they are protected from the hash collision vulnerability, but not be because they are running in a virtualenv. That is a strong argument for letting this break and forcing the update.
msg158073 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-04-11 20:45
Closing, now that we've released finals.
msg158172 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2012-04-12 21:34
For posterity, here's the release notes that we had drafted on the pirate pad:

Note: This patch release of Python may have compatibility implications for environments utilizing the third-party virtualenv. For more detail see XXX.
[the note above is intended to be included as a line-item in the release notes, referencing the file below]
Upgrade issues with virtualenv
======================
In order to enact a security fix in Python 2.6.8, 2.7.3, 3.1.5, and 3.2.3, the implementation of os.urandom was updated. The implementation of urandom was moved from the os module (Python stdlib) to the posix module (built in to the Python executable binary). As a result, when upgrading a host to one of these new Python versions, virtual environments created with earlier versions of Python will now be missing os.urandom. This issue only exists for Unix hosts.
This happens because the virtualenv bundles the older Python executable (which does not have an implementation of urandom), but then references the newer standard library (in which the os module also does not have an implementation of urandom).
In environments where this error occurs, the error is most likely exhibited as an AttributeError:
    $ $ENV/bin/python -c "import os; os.urandom"
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    AttributeError: 'module' object has no attribute 'urandom'
Workaround
=========
The workaround is to remove the 'python' binary from $ENV/bin and re-run virtualenv for the required Python version or versions on that environment. For example:
    $ rm $ENV/bin/python
    $ virtualenv --python python2.7 $ENV
It's necessary to remove the main python binary because virtualenv will not replace it if it already exists. If the virtualenv contained multiple Python versions, run virtualenv on it again with each Python. If the --distribute flag was used in the initial creation of the virtualenv, use it again here.
History
Date User Action Args
2012-04-12 21:34:38jason.coombssetmessages: + msg158172
2012-04-11 20:45:16benjamin.petersonsetstatus: open -> closed
resolution: not a bug
messages: + msg158073
2012-03-29 23:02:56eric.araujosetnosy: + eric.araujo
2012-03-29 22:10:56r.david.murraysetnosy: + r.david.murray
2012-03-29 21:53:26carljmsetmessages: + msg157108
2012-03-29 21:42:11barrysetmessages: + msg157107
2012-03-29 21:27:53georg.brandlsetmessages: + msg157106
2012-03-29 21:23:59jason.coombssetmessages: + msg157105
2012-03-29 21:18:22eric.snowsetnosy: + eric.snow
2012-03-29 21:14:41jason.coombssetmessages: + msg157102
2012-03-29 21:12:23benjamin.petersonsetmessages: + msg157101
2012-03-29 21:11:55dmalcolmsetnosy: + dmalcolm
2012-03-29 21:10:25carljmsetmessages: + msg157100
2012-03-29 20:48:39benjamin.petersonsetmessages: + msg157097
2012-03-29 19:42:10jason.coombssetmessages: + msg157090
2012-03-29 19:12:02loewissetnosy: + loewis
messages: + msg157087
2012-03-29 18:42:52barrysetmessages: + msg157081
2012-03-29 18:25:48carljmsetmessages: + msg157079
2012-03-29 16:42:15jason.coombssetmessages: + msg157067
2012-03-29 16:41:50jason.coombssetversions: + Python 3.1, - Python 3.3
2012-03-29 16:41:30jason.coombscreate