classification
Title: sys.modules changes size during iteration in regrtest module
Type: behavior Stage: resolved
Components: Tests Versions: Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: eric.araujo, ezio.melotti, flox, r.david.murray, xdegaye
Priority: low Keywords: patch

Created on 2010-07-13 16:36 by xdegaye, last changed 2010-09-07 14:38 by eric.araujo. This issue is now closed.

Files
File name Uploaded Description Edit
sys.modules_regrtest.patch xdegaye, 2010-07-13 16:36 patch forcing email modules imports
Messages (6)
msg110211 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2010-07-13 16:36
Python 2.7 - svn revision 82852

Bug description
===============

Test script foo.py
------------------

    #!/usr/bin/env python
    import distutils.core
    import test.regrtest


Exception when running foo.py
-----------------------------

Traceback (most recent call last):
  File "foo.py", line 3, in <module>
    import test.regrtest
  File "/usr/local/lib/python2.7/test/regrtest.py", line 184, in <module>
    for module in sys.modules.itervalues():
RuntimeError: dictionary changed size during iteration


Workaround
==========

Revese the order of the imports in foo.py.


Proposed patch
==============

Root cause: email modules are being imported inside the loop.

The following patch (svn revision 82852) forces the import of the
email modules before the main iteration. These modules have been
lazily imported in email/__init__.py and need to be effectively
imported before updating the __file__ and __path__ attributes.

The patch is also attached to this report.

--- release27-maint/Lib/test/regrtest.py       2010-07-13 18:11:32.000000000 +0200
+++ /usr/local/lib/python2.7/test/regrtest.py   2010-07-13 18:00:52.000000000 +0200
@@ -181,6 +181,12 @@
 # (site.py absolutize them), the __file__ and __path__ will be absolute too.
 # Therefore it is necessary to absolutize manually the __file__ and __path__ of
 # the packages to prevent later imports to fail when the CWD is different.
+
+# Email modules are imported lazily, force their import first.
+if 'email' in sys.modules:
+    [hasattr(sys.modules[name], '__foo__') for name in list(sys.modules)
+                                                if name.startswith('email.')]
+
 for module in sys.modules.itervalues():
     if hasattr(module, '__path__'):
         module.__path__ = [os.path.abspath(path) for path in module.__path__]
msg110214 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-07-13 17:10
Why are you importing regrtest?  It isn't designed to be imported, really.  Or at least, not any more.
msg110220 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2010-07-13 19:15
http://pyclewn.sourceforge.net uses the regrtest module to run its
testsuite.

The test package is documented in the Standard Library documentation,
quote:
"The test package contains all regression tests for Python as well
as the modules test.test_support and test.regrtest. test.test_support
is used to enhance your tests while test.regrtest drives the testing
                   ^^^
suite."

It is not clear why regrtest "isn't designed to be imported, really.
Or at least, not any more." when test/autotest.py in the python
source code does import regrtest.
msg110227 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-07-13 21:04
The wording is poor. I'm going to fix them so it's obvious that the test package is meant for core developers only.
msg110229 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-07-13 21:26
Autotest runs the Python regression suite, it just does it a different way.  And it still works on trunk. Since it isn't documented (I wasn't even aware it existed), maybe we should just delete it. :)

Note that I'm not saying there's no bug here.  But importing regrtest in the general case is not supported by the core team, so we may well break that usage for any number of good reasons.  In this case, it was to support changing the CWD when running tests to facilitate test cleanup.  The loop expects a limited number of modules to be in sys.modules when it runs (it is fixing a test-only startup issue with __file__ in early-imported modules when run from Lib).

Note that the change that triggered the error you are seeing was made relatively recently (it worked before I updated and failed afterward, and my checkout was fairly up to date).  There were no regrtest changes, but there were distutils changes (utils and sysconfig were the modules I noticed getting updated).

I'm adding ezio and flox to nosy since I believe they added the regrtest loop in question, but I think we will probably close this won't fix.
msg110235 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2010-07-13 21:53
Thanks for your comments, that was fast!
I will skip using the test package then.
History
Date User Action Args
2010-09-07 14:38:17eric.araujosetstatus: pending -> closed
stage: resolved
2010-07-13 21:54:57xdegayesetstatus: open -> pending
2010-07-13 21:53:43xdegayesetstatus: pending -> open

messages: + msg110235
2010-07-13 21:26:03r.david.murraysetstatus: open -> pending
priority: normal -> low

nosy: + ezio.melotti, flox, - brett.cannon
messages: + msg110229

resolution: wont fix
2010-07-13 21:04:44brett.cannonsetnosy: + brett.cannon
messages: + msg110227
2010-07-13 19:49:39eric.araujosetnosy: + eric.araujo
2010-07-13 19:15:10xdegayesetmessages: + msg110220
2010-07-13 17:10:01r.david.murraysettype: crash -> behavior

messages: + msg110214
nosy: + r.david.murray
2010-07-13 16:36:10xdegayecreate