This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: importlib not checking bytecode length
Type: behavior Stage: test needed
Components: Library (Lib) Versions: Python 3.1
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: brett.cannon, jnoller, r.david.murray
Priority: low Keywords:

Created on 2009-11-19 19:18 by r.david.murray, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Messages (10)
msg95511 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2009-11-19 19:18
One of the buildbots suffered a test_multiprocessing failure, which
itself is not such an unusual thing.  But the error that was
subsequently thrown looks like it might point to a place where
importlib's error handling could be improved.  The error that was thrown
certainly gives no clear indication of what really caused the problem.

The full buildbot report was:
http://www.python.org/dev/buildbot/all/builders/ia64%20Ubuntu%203.1/builds/93/steps/test/logs/stdio

Here are the tracebacks (the second one being the interesting one):

    Traceback (most recent call last):
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/process.py",
line 233, in _bootstrap
        self.run()
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/process.py",
line 88, in run
        self._target(*self._args, **self._kwargs)
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/test/test_multiprocessing.py",
line 1211, in _putter
        queue = manager.get_queue()
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/managers.py",
line 644, in temp
        token, exp = self._create(typeid, *args, **kwds)
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/managers.py",
line 544, in _create
        id, exposed = dispatch(conn, None, 'create', (typeid,)+args, kwds)
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/managers.py",
line 79, in dispatch
        kind, result = c.recv()
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/connection.py",
line 408, in recv
        s = self._conn.recv_bytes()
    IOError: [Errno 104] Connection reset by peer
    test test_multiprocessing failed -- Traceback (most recent call last):
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/test/test_multiprocessing.py",
line 1222, in test_rapid_restart
        queue = manager.get_queue()
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/managers.py",
line 644, in temp
        token, exp = self._create(typeid, *args, **kwds)
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/managers.py",
line 542, in _create
        conn = self._Client(self._address, authkey=self._authkey)
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/multiprocessing/connection.py",
line 427, in XmlClient
        import xmlrpc.client as xmlrpclib
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/importlib/_bootstrap.py",
line 151, in decorated
        return fxn(self, module)
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/importlib/_bootstrap.py",
line 399, in load_module
        return self._load_module(module)
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/importlib/_bootstrap.py",
line 324, in _load_module
        code_object = self.get_code(module.__name__)
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/importlib/_bootstrap.py",
line 411, in get_code
        pyc_timestamp = marshal._r_long(data[4:8])
      File
"/home/pybot/buildarea/3.1.klose-debian-ia64/build/Lib/importlib/__init__.py",
line 65, in _r_long
        x = int_bytes[0]
    IndexError: index out of range
msg95525 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2009-11-20 02:49
I will obviously have a look, but why the heck is test_multiprocessing 
calling importlib? Is it meant to do that? Or is this part of the -j flag 
for regrtest?
msg95526 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2009-11-20 02:51
And it looks like the error was triggered from a malformed .pyc file that 
was truncated. Guess I need to be more careful with that possibility. =)
msg95528 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2009-11-20 03:07
This was on a buildbot, and regrtest -j isn't used there, but in any
case it wouldn't affect importlib as far as I know.  From the traceback
it looks like multiprocessing is just doing an import.
msg95560 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2009-11-20 19:22
Huh. I wonder how importlib is ending up being the implementation of 
import then? Guess I also need to double-check everything is still 
cleaning itself up properly.
msg98778 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-02-03 08:02
I can't reproduce the failure as running test_multiprocessing and then test_importlib does not show any left over stuff in the interpreter what would lead to importlib being used for __import__. But as the IndexError part is a valid issue I am renaming the issue.
msg99023 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-02-07 21:52
issue #7875 brought up the original failure that caused the creation of this bug again and led to me finally diagnosing the problem.
msg99025 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-02-07 22:01
A quick check of things shows that a totally empty file, a missing timestamp, or no marshalled code triggers the use of source. I still need to check what happens when the magic number is truncated by a byte or so along with the timestamp.

Any screw-up with the marshalled code should still be an error.
msg99575 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-02-19 16:01
Committed to py3k as r78242. Still need to backport.
msg99576 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-02-19 16:05
Backported to 3.1 in r78244.
History
Date User Action Args
2022-04-11 14:56:54adminsetgithub: 51610
2010-02-19 16:05:50brett.cannonsetstatus: pending -> closed

messages: + msg99576
2010-02-19 16:01:54brett.cannonsetstatus: open -> pending
resolution: fixed
2010-02-19 16:01:41brett.cannonsetmessages: + msg99575
2010-02-07 22:01:35brett.cannonsetmessages: + msg99025
2010-02-07 21:52:04brett.cannonsetmessages: + msg99023
2010-02-07 21:48:51brett.cannonlinkissue7875 dependencies
2010-02-03 08:02:16brett.cannonsetmessages: + msg98778
title: Strange importlib error during test_multiprocessing failure -> importlib not checking bytecode length
2009-11-20 19:22:40brett.cannonsetmessages: + msg95560
2009-11-20 03:08:00r.david.murraysetmessages: + msg95528
2009-11-20 02:51:49brett.cannonsetmessages: + msg95526
2009-11-20 02:49:52brett.cannonsetassignee: brett.cannon
messages: + msg95525
2009-11-19 19:18:06r.david.murraycreate