An interesting bug failure on one of the buildbots. It seems importlib has been enabled as the default import mechanism (when? where?).

test_multiprocessing
Process Process-24:
Traceback (most recent call last):
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/multiprocessing/process.py", line 233, in _bootstrap
    self.run()
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/test/test_multiprocessing.py", line 1210, in _putter
    manager.connect()
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/multiprocessing/managers.py", line 477, in connect
    conn = Client(self._address, authkey=self._authkey)
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/multiprocessing/connection.py", line 427, in XmlClient
    import xmlrpc.client as xmlrpclib
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/importlib/_bootstrap.py", line 151, in decorated
    return fxn(self, module)
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/importlib/_bootstrap.py", line 399, in load_module
    return self._load_module(module)
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/importlib/_bootstrap.py", line 324, in _load_module
    code_object = self.get_code(module.__name__)
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/importlib/_bootstrap.py", line 411, in get_code
    pyc_timestamp = marshal._r_long(data[4:8])
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/importlib/__init__.py", line 65, in _r_long
    x = int_bytes[0]
IndexError: index out of range
test test_multiprocessing failed -- Traceback (most recent call last):
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/test/test_multiprocessing.py", line 1223, in test_rapid_restart
    queue = manager.get_queue()
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/multiprocessing/managers.py", line 644, in temp
    token, exp = self._create(typeid, *args, **kwds)
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/multiprocessing/managers.py", line 542, in _create
    conn = self._Client(self._address, authkey=self._authkey)
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/multiprocessing/connection.py", line 427, in XmlClient
    import xmlrpc.client as xmlrpclib
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/importlib/_bootstrap.py", line 151, in decorated
    return fxn(self, module)
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/importlib/_bootstrap.py", line 399, in load_module
    return self._load_module(module)
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/importlib/_bootstrap.py", line 324, in _load_module
    code_object = self.get_code(module.__name__)
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/importlib/_bootstrap.py", line 411, in get_code
    pyc_timestamp = marshal._r_long(data[4:8])
  File "/home/buildbot/slave/py-build/3.1.norwitz-amd64/build/Lib/importlib/__init__.py", line 65, in _r_long
    x = int_bytes[0]
IndexError: index out of range

I don't know who is doing this, but from what I can tell it isn't test_imoprtlib; running with -i shows __import__ to still be the built-in one along with no new sys.path_hooks entries or stale entries in sys.path_importer_cache. Same goes for when I run test_multiprocessing after test_importlib.

I will probably have to modify regrtest to check for new values in some key places to see who the culprit is.

OK, I figured this one out. Someone is using importlib.import_module() which uses importlib itself to do the import and not builtins.__import__. That has the side-effect of importlib replacing all None entries in sys.path_importer_cache with the finder that importlib creates (an optimization so that importlib doesn't have to re-construct the finder constantly and to slowly drop the use of None).

There are three ways to fix this. One is to not insert the new finders into sys.path_importer_cache. Two is to have importlib.import_module use __import__. Three is to leave it as-is (and to obviously fix the problem importlib is triggering).

From a selfish perspective I like option 3 since it helps weed out bugs in importlib, but that is unfair to __import__ for testing plus it slows down the test suite overall from importlib being slower. And I won't do option 1 since that would make importlib perform even worse. So I guess that leaves option 2 unless people are actually okay with option 3.

That would be test_support.import_module, which at your recommendation I changed to use importlib.import_module at the last pycon sprint :)

I'm not clear on why option 3 would interfere with __import__ getting tested?

I meant test.support for py3k, of course.

> That would be test_support.import_module, which at your recommendation 
> I changed to use importlib.import_module at the last pycon sprint :)

I don't think it's the culprit. According to the traceback, importlib is triggered by a simple "import xmlrpc.client as xmlrpclib".

So option 3 would marginalize __import__ testing as importlib's finders and loaders as pulled from sys.path_importer_cache would be used over __import__'s own implicit importers that it contains.

As for test.support.import_module being the culprit, it is. The reason the traceback shows an import statement as the trigger is that __import__ is using a loader as defined by importlib.

Option 3 is ok with me.

There could also be option 4: add a special flag (or environment var) to regrtest so as to run all tests with __import__ pulled from importlib. Then we can setup a buildbot with that flag / env var.

Importlib actually already can do this: importlib.test.regrtest. It runs regrtest after importlib has been set for __import__. It also skips tests known to fail for stupid reasons (typically they don't expect __loader__ to be defined). You can also run the importlib tests using __import__; importlib.test --builtin.

Anyway, seems like people are fine with option 3, and since that's the easiest that's what I will go with.

With issue #7361 fixed, this should now no longer be an issue.