classification
Title: imp.load_module() leads to the improper caching of the 'file' argument
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: brett.cannon, kushal.das, python-dev, rpetrov
Priority: normal Keywords: easy

Created on 2013-03-05 19:33 by brett.cannon, last changed 2013-04-28 16:00 by brett.cannon. This issue is now closed.

Messages (7)
msg183549 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-03-05 19:33
As of right now, if you use imp.load_module(), it will store any 'file' argument you give it in a hacked loader. That's a problem when you call imp.reload() on such a module, though, as subsequent calls to __loader__.load_module() will attempt to use the cached file object which was closed on the initial load. What the code should do is use the hacked loader to load the module, but then set an unhacked loader to __loader__ for reloads to work properly.

Traceback below:

======================================================================
ERROR: test_source (test.test_imp.ReloadTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/work/cf49a3092a96daebb23bf3c6e0d86ac9/google3/tmp/lib/python3.3/test/test_imp.py", line 248, in test_source
    imp.reload(os)
  File "/build/work/cf49a3092a96daebb23bf3c6e0d86ac9/google3/tmp/lib/python3.3/imp.py", line 252, in reload
    return module.__loader__.load_module(name)
  File "<frozen importlib._bootstrap>", line 586, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1023, in load_module
  File "<frozen importlib._bootstrap>", line 1004, in load_module
  File "<frozen importlib._bootstrap>", line 562, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 854, in _load_module
  File "<frozen importlib._bootstrap>", line 978, in get_code
  File "/build/work/cf49a3092a96daebb23bf3c6e0d86ac9/google3/tmp/lib/python3.3/imp.py", line 88, in get_data
    with self.file:
ValueError: I/O operation on closed file.
msg183553 - (view) Author: Roumen Petrov (rpetrov) * Date: 2013-03-05 21:10
How to reproduce issue with normal in source tree build ?

I'm asking because to avoid issue, on read only file system , I use patch posted in scope of issue3754 and issue15833 (0016-CROSS-reload-may-fail-with-operation-on-closed-file-.patch) .
msg183554 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-03-05 21:24
I don't have a Python 3.3 install happy (someone at work reported the failure), but simply calling imp.load_module() and then __loader__.load_module() on the returned module immediately after should trigger it.
msg184491 - (view) Author: Kushal Das (kushal.das) * (Python committer) Date: 2013-03-18 18:53
Working on a patch for this.
msg184497 - (view) Author: Kushal Das (kushal.das) * (Python committer) Date: 2013-03-18 19:28
Can not reproduce it :(
msg187989 - (view) Author: Roundup Robot (python-dev) Date: 2013-04-28 15:58
New changeset 3dcc81c2eef5 by Brett Cannon in branch '3.3':
Issue #17358: imp.load_source() and load_compiled() should now return
http://hg.python.org/cpython/rev/3dcc81c2eef5

New changeset be6bbc9f0561 by Brett Cannon in branch 'default':
merge for issue #17358
http://hg.python.org/cpython/rev/be6bbc9f0561
msg187990 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-04-28 16:00
I figured out why testing was difficult; the file is only read from if it's necessary. That means if you are trying to load source which has bytecode available which is legitimate it will simply skip over the source and read from the bytecode.

So in the end I just fixed it without a test since it's for a marginal case for a deprecated module. Plus the fix is rather simple and still passed the test suite.
History
Date User Action Args
2013-04-28 16:00:24brett.cannonsetstatus: open -> closed
resolution: fixed
messages: + msg187990

stage: test needed -> resolved
2013-04-28 15:58:44python-devsetnosy: + python-dev
messages: + msg187989
2013-04-10 01:24:20brett.cannonsetassignee: brett.cannon
2013-03-18 19:28:24kushal.dassetmessages: + msg184497
2013-03-18 18:53:47kushal.dassetnosy: + kushal.das
messages: + msg184491
2013-03-05 21:24:08brett.cannonsetmessages: + msg183554
2013-03-05 21:10:44rpetrovsetnosy: + rpetrov
messages: + msg183553
2013-03-05 19:33:05brett.cannoncreate