Title: Intermittent import failure
Type: behavior Stage: test needed
Components: Interpreter Core Versions: Python 3.3
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, eric.snow, exarkun
Priority: normal Keywords:

Created on 2012-09-10 19:48 by exarkun, last changed 2012-09-11 14:42 by eric.snow. This issue is now closed.

File name Uploaded Description Edit exarkun, 2012-09-10 19:48
Messages (5)
msg170225 - (view) Author: Jean-Paul Calderone (exarkun) * (Python committer) Date: 2012-09-10 19:48
The attached unit test fails with an ImportError... sometimes.  Here's a little blob of shell that seems to make the failure come up more quickly:

while ~/Projects/cpython/3.3/python -m unittest -v test_broken_import; do
    rm -rf test_broken_import;
    rm -rf __pycache__/;

An example of the output when I run it:

exarkun@top:/tmp$ while ~/Projects/cpython/3.3/python -m unittest -v test_broken_import_minimal; do rm -rf test_broken_import; rm -rf __pycache__/; done
test_renamedSource (test_broken_import_minimal.BrokenTests) ... ok

Ran 1 test in 0.004s

test_renamedSource (test_broken_import_minimal.BrokenTests) ... ok

Ran 1 test in 0.005s

test_renamedSource (test_broken_import_minimal.BrokenTests) ... > /tmp/
-> print(e)
(Pdb) c
No module named 'twisted_renamed_helper'

ERROR: test_renamedSource (test_broken_import_minimal.BrokenTests)
Traceback (most recent call last):
  File "./", line 50, in test_renamedSource
    from twisted_renamed_helper import module
ImportError: No module named 'twisted_renamed_helper'

Ran 1 test in 0.759s

FAILED (errors=1)
msg170231 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-09-10 20:28
The FileFinder class (a "path entry finder") uses a cache to efficiently track changes to files.  You can manually clear this cache by calling importlib.invalidate_caches().  The Python test suite has several examples of clearing the FileFinder cache in this way.

I've verified the failure as described.  Using importlib.invalidate_caches() at the spot where the script cleans up sys.modules, the failure goes away.

Depending on OS filesystem timestamp resolution, you could see the behavior just as you've described it.  Since normally you wouldn't remove and re-add modules like this, it isn't normally an issue.
msg170254 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-09-11 00:39
Thinking about it, it may be worth adding a note to the docs for sys.modules making it clear about importlib.invalidate_caches().  I'll see about writing up a patch when I get a chance.
msg170300 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-09-11 12:50
I don't think sys.modules is the right place about a warning regarding importlib.invalidate_caches(); this has nothing to do with sys.modules but instead import itself. Probably something more along those lines would be more fitting.

This is also mentioned the 3.3 What's New on how to port your code, so at least that's covered.
msg170310 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-09-11 14:42
The relationship between invalidate_caches() and sys.modules is definitely tenuous.  However, my rationale was that people would look for an explanation on why modifying sys.modules was not working as expected.  The sys.modules doc entry was the one that seemed to make the most sense.  

However, this is a pretty uncommon case and you're right that the What's New entry should be enough.
Date User Action Args
2012-09-11 14:42:38eric.snowsetstatus: open -> closed
resolution: rejected
messages: + msg170310
2012-09-11 12:50:28brett.cannonsetnosy: + brett.cannon
messages: + msg170300
2012-09-11 00:39:05eric.snowsetstatus: pending -> open

messages: + msg170254
2012-09-10 21:02:08eric.snowsetstatus: open -> pending
2012-09-10 20:28:15eric.snowsetnosy: + eric.snow
messages: + msg170231

type: behavior
stage: test needed
2012-09-10 19:48:07exarkuncreate