classification
Title: Have imp.cache_from_source() raise NotImplementedError when cache tag not available
Type: Stage: test needed
Components: Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: 15166 Superseder:
Assigned To: Nosy List: Arfrever, Pranav.Ravichandran, brett.cannon, eric.araujo, eric.snow, georg.brandl, python-dev
Priority: release blocker Keywords: easy, patch

Created on 2012-06-13 17:10 by brett.cannon, last changed 2012-07-09 17:59 by brett.cannon. This issue is now closed.

Files
File name Uploaded Description Edit
cachetag.patch Pranav.Ravichandran, 2012-06-20 17:12 Possible patch review
Messages (10)
msg162712 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-06-13 17:10
Both imp.cache_from_source() and source_from_cache() should throw NotImplementedError when sys.implementation.cache_tag is None. See the thread starting at http://mail.python.org/pipermail/python-dev/2012-June/120145.html for discussion of this decision.
msg162713 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-06-13 17:26
It seems to me there is overlap between sys.implementation.cache_tag and sys.dont_write_bytecode.  I was expecting sys.impl.cache_tag to be purely informational, and not actually controlling some behavior. “If cache_tag is set to None, it indicates that module caching should be disabled”: isn’t that the job of sys.dont_write_bytecode?

(The difference that I see is that sys.impl.cache_tag is defined by the VM and dont_write_bytecode by each interpreter invocation; I was expecting VMs that never want to write bytecode to just always set dont_write_bytecode to False, i.e. make -B redundant.)
msg162801 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-06-14 15:32
So sys.dont_write_bytecode is there to prevent bytecode writing but not loading. This is an issue for some systems (e.g. clusters) where there are so many processes running Python that they start to trample each others bytecode files and it leads to malformed data. If you set sys.implementation.cache_tag to None you stop all bytecode usage (reading and writing), and thus is "disabled" entirely,  not just for writing.
msg163289 - (view) Author: Pranav Ravichandran (Pranav.Ravichandran) Date: 2012-06-20 17:12
First-time contributor here, so just to make sure, let me just clarify this - the trunk builds fine, but one of the tests, test_urllib2_localnet, fails, I'm not sure why. Also, the patch is kind of messed up because of changes to Python/importlib.h. I looked this up on google, but wasn't able to find much help. Awaiting feedback :)
msg163727 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-06-24 06:33
> So sys.dont_write_bytecode is there to prevent bytecode writing but not loading.
Ah right.  Providing a way to avoid reading them sounds good.

> If you set sys.implementation.cache_tag to None you stop all bytecode usage (reading and writing)
This I feel uncomfortable with.  First, I would separate writing and reading; second, I think of sys.implementation as a read-only information holder, and would think it immutable (note that I haven’t read the latest version of the PEP); just like sys.flags.dont_write_bytecode does not change but sys.dont_write_bytecode can change.  So even though I think sys has a lot of attributes, the cleanest thing here would be a new sys.dont_read_bytecode attribute (as was proposed in the thread), maybe also with a new command-line option (or using -BB) and environment variable.
msg163730 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-06-24 06:37
Brett meant the implementation would set it.  You'd still expect it to not change values during the execution of the interpreter.  I suppose anyone could change that value, but I'd see that more as a consenting adults situation.
msg163799 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-06-24 15:32
Thanks for the patch, Pranav. I'm probably going to be too busy for a couple of weeks, so I won't look at this immediately, but I wanted to let you know I will eventually get to it.
msg164100 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-06-26 20:51
Moving back to blocker for beta2.
msg165100 - (view) Author: Roundup Robot (python-dev) Date: 2012-07-09 17:58
New changeset e86330669bb5 by Brett Cannon in branch 'default':
Issue #15056: imp.cache_from_source() and source_from_cache() raise
http://hg.python.org/cpython/rev/e86330669bb5
msg165101 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-07-09 17:59
I released Pranav's patch and fleshed it out with docs, tests, and changes to importlib's use of cache_from_source().
History
Date User Action Args
2012-07-09 17:59:21brett.cannonsetstatus: open -> closed
resolution: fixed
messages: + msg165101
2012-07-09 17:58:16python-devsetnosy: + python-dev
messages: + msg165100
2012-07-02 18:58:22brett.cannonsetdependencies: + Implement imp.get_tag() using sys.implementation
2012-06-26 20:51:11georg.brandlsetpriority: deferred blocker -> release blocker
nosy: + georg.brandl
messages: + msg164100

2012-06-24 15:32:39brett.cannonsetmessages: + msg163799
2012-06-24 06:37:41eric.snowsetmessages: + msg163730
2012-06-24 06:33:49eric.araujosetmessages: + msg163727
2012-06-20 17:12:42Pranav.Ravichandransetfiles: + cachetag.patch

nosy: + Pranav.Ravichandran
messages: + msg163289

keywords: + patch
2012-06-14 15:38:50Arfreversetnosy: + Arfrever
2012-06-14 15:32:15brett.cannonsetmessages: + msg162801
2012-06-13 18:20:34eric.snowsetnosy: + eric.snow
2012-06-13 17:26:23eric.araujosetnosy: + eric.araujo
messages: + msg162713
2012-06-13 17:10:57brett.cannonsetstage: test needed
2012-06-13 17:10:51brett.cannoncreate