classification
Title: try/except import fails --without-threads
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pitrou Nosy List: Arfrever, brett.cannon, eric.snow, nadeem.vawda, pitrou, python-dev, skrah
Priority: release blocker Keywords: patch

Created on 2012-04-14 20:45 by skrah, last changed 2012-05-07 19:48 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
impstuff.patch pitrou, 2012-05-06 16:12 review
Messages (13)
msg158281 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-04-14 20:45
In the build --without-threads, catching an ImportError in support.py
fails. Fedora buildbot:

http://www.python.org/dev/buildbot/all/builders/AMD64%20Fedora%20without%20threads%203.x/builds/1986/steps/test/logs/stdio

./python  ./Tools/scripts/run_tests.py -j 1 -u all -W --timeout=3600 
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/test/__main__.py", line 1, in <module>
    from test import regrtest, support
  File "<frozen importlib._bootstrap>", line 1038, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 977, in _find_and_load
  File "<frozen importlib._bootstrap>", line 561, in load_module
  File "<frozen importlib._bootstrap>", line 218, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 446, in _load_module
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/test/regrtest.py", line 243, in <module>
    from test import support
  File "<frozen importlib._bootstrap>", line 1038, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 977, in _find_and_load
  File "<frozen importlib._bootstrap>", line 561, in load_module
  File "<frozen importlib._bootstrap>", line 218, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 446, in _load_module
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/test/support.py", line 34, in <module>
    import multiprocessing.process
SystemError: NULL result without error in PyObject_Call
[110893 refs]
make: *** [buildbottest] Error 255
msg158401 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-04-16 06:41
The SystemError has changed to a KeyError:


http://www.python.org/dev/buildbot/all/builders/AMD64%20Fedora%20without%20threads%203.x/builds/2004/steps/test/logs/stdio


At least that makes it easy to spot the location in import.c. The
shortest way to reproduce the error is to compile --without-threads,
then:

import multiprocessing
import multiprocessing.process
msg158472 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-16 15:44
Just to clarify the failure for the bug history, somehow multiprocessing is not ending up in sys.modules as expected. It changed from a SystemError to a KeyError because I started to properly check a PyDict_GetItem() return value instead of blindly assuming something was going to be in sys.modules.
msg159500 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-04-27 21:56
This issue is now apparently causing a segfault:

(gdb) r ./Tools/scripts/run_tests.py -j 1 -u all -W --timeout=3600
Starting program: /home/stefan/pydev/cpython/python ./Tools/scripts/run_tests.py -j 1 -u all -W --timeout=3600

[...]

Program received signal SIGSEGV, Segmentation fault.
0x000000000041b038 in PyObject_Repr (v=<unknown at remote 0x7ffff43b7660>) at Objects/object.c:377
377         if (Py_TYPE(v)->tp_repr == NULL)
(gdb) l

[...]



#0  0x000000000041b038 in PyObject_Repr (v=<unknown at remote 0x7ffff43b7660>) at Objects/object.c:377
#1  0x0000000000460db0 in PyUnicode_FromFormatV (format=0x63e3e8 "%R not in sys.modules as expected", vargs=
    0x7fffffff1ae0) at Objects/unicodeobject.c:2609
#2  0x00000000004e1be2 in PyErr_Format (exception=<type at remote 0x8e68c0>, format=
    0x63e3e8 "%R not in sys.modules as expected") at Python/errors.c:697
#3  0x00000000004ed53f in PyImport_ImportModuleLevelObject (name='multiprocessing.process
msg160086 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-06 16:09
Here is a patch. The __import__ function's crazy API never ceases to amaze me.
msg160087 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-06 16:12
Oops, there was a duplicate test.
msg160090 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-05-06 16:23
Only two comments, otherwise LGTM (and I can't believe the solution was to go back through the import system just to pull out the cached module; the things we would change if we were doing this from scratch).

One, you have some "XXX False" markers in the tests. Should those get deleted or replaced with something?

Two, in your first test (at least) you only test what is in sys.modules once instead of after each attempted import. I would repeat the test after each import.
msg160092 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-06 16:31
> One, you have some "XXX False" markers in the tests. Should those get
> deleted or replaced with something?

Well, I don't know what to replace them with. I would have expected
pkg.module to end up in sys.modules, but as mentioned in the comments
the relative import line actually fails with an ImportError. There's
probably some logic behind that, but that's beyond me.

> Two, in your first test (at least) you only test what is in
> sys.modules once instead of after each attempted import. I would
> repeat the test after each import.

Ok, will do.
msg160094 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-06 16:35
> > One, you have some "XXX False" markers in the tests. Should those get
> > deleted or replaced with something?
> 
> Well, I don't know what to replace them with. I would have expected
> pkg.module to end up in sys.modules, but as mentioned in the comments
> the relative import line actually fails with an ImportError. There's
> probably some logic behind that, but that's beyond me.

I should mention that the same happens with 2.7 and 3.2, so it's not a
regression. It's just that I don't understand the logic :-)
msg160105 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-05-06 19:24
So I was going to try to figure out the logic, so I manually created the test files to start debugging, but I didn't get the ImportError but instead the 1/0 error for the relative import. Maybe it's specific to lack of threads or the change you made? I mean if that's how it has always worked then I'm not arguing that it's wrong, just that it's a weird side-effect:


>>> import pkg
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 977, in _find_and_load
  File "<frozen importlib._bootstrap>", line 596, in load_module
  File "<frozen importlib._bootstrap>", line 262, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 484, in _load_module
  File "./pkg/__init__.py", line 3, in <module>
    1/0
ZeroDivisionError: division by zero
[70552 refs]
>>> import sys
[70554 refs]
>>> sys.modules['pkg']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'pkg'
[70462 refs]
>>> sys.modules['pkg.module']
<module 'pkg.module' from './pkg/module.py'>
[70465 refs]
>>> with open('pkg/__init__.py') as file: print(file.read())
... 
from . import module
#import pkg.module
1/0
msg160106 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-06 19:42
> So I was going to try to figure out the logic, so I manually created
> the test files to start debugging, but I didn't get the ImportError
> but instead the 1/0 error for the relative import. Maybe it's specific
> to lack of threads or the change you made? I mean if that's how it has
> always worked then I'm not arguing that it's wrong, just that it's a
> weird side-effect:

The first "import pkg" gets you a ImportError but the second one should
get you a ZeroDivisionError (yes, weird).
msg160164 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-07 19:44
New changeset d6324941b739 by Antoine Pitrou in branch 'default':
Issue #14583: Fix importlib bug when a package's __init__.py would first import one of its modules then raise an error.
http://hg.python.org/cpython/rev/d6324941b739
msg160167 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-07 19:48
Should be fixed now. The buildbot has been able to launch the test suite.
History
Date User Action Args
2012-05-07 19:48:52pitrousetstatus: open -> closed
resolution: fixed
messages: + msg160167

stage: patch review -> resolved
2012-05-07 19:44:04python-devsetnosy: + python-dev
messages: + msg160164
2012-05-06 19:42:43pitrousetmessages: + msg160106
2012-05-06 19:24:32brett.cannonsetmessages: + msg160105
2012-05-06 16:35:44pitrousetmessages: + msg160094
2012-05-06 16:32:00pitrousetmessages: + msg160092
2012-05-06 16:24:00brett.cannonsetassignee: pitrou
messages: + msg160090
2012-05-06 16:12:06pitrousetfiles: + impstuff.patch

messages: + msg160087
stage: needs patch -> patch review
2012-05-06 16:11:42pitrousetfiles: - impstuff.patch
2012-05-06 16:10:01pitrousetfiles: + impstuff.patch

nosy: + pitrou
messages: + msg160086

keywords: + patch
2012-05-06 13:33:54nadeem.vawdasetnosy: + nadeem.vawda
2012-04-27 21:57:45skrahlinkissue14685 superseder
2012-04-27 21:56:20skrahsetpriority: high -> release blocker

messages: + msg159500
2012-04-17 00:42:26Arfreversetnosy: + Arfrever
2012-04-16 15:44:40brett.cannonsetmessages: + msg158472
2012-04-16 06:41:37skrahsetpriority: normal -> high

messages: + msg158401
2012-04-14 21:23:35eric.snowsetnosy: + eric.snow
2012-04-14 20:45:30skrahcreate