classification
Title: runpy swallows ImportError information with relative imports
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: brett.cannon Nosy List: amaury.forgeotdarc, brett.cannon, chris.jerdonek, eric.snow, georg.brandl, jeffknupp, ncoghlan, python-dev, r.david.murray
Priority: critical Keywords: 3.2regression, patch

Created on 2012-07-10 05:39 by chris.jerdonek, last changed 2012-08-24 22:27 by brett.cannon. This issue is now closed.

Files
File name Uploaded Description Edit
issue-15316-failing-test-1.patch chris.jerdonek, 2012-08-22 19:08 review
Messages (14)
msg165166 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-07-10 05:39
With the following package directory structure--

foo/
    __init__.py
    __main__.py
        from foo import bar
    bar.py
        print('***')
        raise ImportError('test...')

Running--

$ ./python.exe -m foo

Yields--

***
Traceback (most recent call last):
  File ".../python/cpython/cpython/Lib/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File ".../python/cpython/cpython/Lib/runpy.py", line 75, in _run_code
    exec(code, run_globals)
  File "./foo/__main__.py", line 1, in <module>
    from foo import bar
ImportError: cannot import name bar
$

The exception text gets swallowed.

Changing the relative import "from foo import bar" to "import foo.bar" does show the exception text.
msg165175 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-07-10 07:54
Looks very similar to issue15111.
msg168830 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-22 00:15
I randomly ran into this issue again.  I'm not sure this was ever resolved (i.e. I think it may always have been different from issue 15111).

I still get the above behavior in the default branch.

And here is what I get in the 3.2 branch (the error information is not swallowed):
 
$ ./python.exe -m foo
***
Traceback (most recent call last):
  File ".../Lib/runpy.py", line 161, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File ".../Lib/runpy.py", line 74, in _run_code
    exec(code, run_globals)
  File ".../foo/__main__.py", line 1, in <module>
    from foo import bar
  File "foo/bar.py", line 2, in <module>
    raise ImportError('test...')
ImportError: test...
msg168831 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-22 00:25
I ran into this again because an error while running `./python.exe -m test` was getting masked.  The use of __main__.py in the package may be the distinguishing characteristic.
msg168846 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-22 07:21
Should this issue be fixed before the release?  If it is not fixed, certain problems found after the release may become harder to report and diagnose (because the true source of error will be masked).

Two months ago issue 15111 which was thought to be the same was given a priority of "high".
msg168896 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-08-22 16:33
It has nothing to do with runpy and __main__.py and everything to do with rev 78619:0d52f125dd32 (which fixed issue #15715) which was done to ignore bogus names in fromlist. If you simply change the import line to "import foo.bar" then you get the expected result.

The options I see to solve this without invalidating issue #15715 is:

1) Parse the ImportError message to notice that it was a lookup failure and not some other ImportError (ewww)

2) Importlib has to tag every exception is raises because of a finder failure to know when an ImportError was triggered because the module wasn't found (less eww, but still brittle as it means only importlib or people aware of the flag will have the expected semantics)

3) Create a ModuleNotFoundError exception that subclasses ImportError and catch that (breaks doctests and introduces a new exception that people will need to be aware of, plus the question of whether it should just exist in importlib or be a builtin)

4) change importlib._bootstrap._handle_fromlist (and various other bits of importlib code) to call _find_module() and silently ignore when no finder is found as desired (probably need to add a flag to _find_and_load() to signal whether finder failure just returns None or raises ImportError)

While I prefer 3, I think it's a bit late in the release to try to introduce a new exception to begin separating the meaning of 16 different ``raise ImportError`` cases in importlib._bootstrap based on inheritance. My gut says 4 is the best solution given the timeframe. I should be able to get a patch in on Friday if that works for people.
msg168903 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-22 19:08
Here is a formal unit test case that passes in 3.2 but not in 3.3 (a "simpler" case not using __main__.py).

(script_helper.create_empty_file() doesn't seem to be available in 3.2.)
msg168904 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-08-22 19:12
Thanks for the test, Chris. It can probably be simplified using the utilities in test_importlib (e.g. managing the cleanup of sys.path, using mocked loaders to raise the exception instead of having to write to the file system, etc.), but otherwise the idea of the test is accurate.
msg168905 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-08-22 19:18
You're welcome, Brett.  I'll let you or someone else recast the test using the latest preferred techniques.  I was just using the style of the immediately surrounding tests.
msg168907 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-08-22 19:46
> While I prefer 3, I think it's a bit late in the release to try to
> introduce a new exception to begin separating the meaning of 16
> different ``raise ImportError`` cases in importlib._bootstrap based on
> inheritance. My gut says 4 is the best solution given the timeframe.

Agreed.  I still think option 3 would be suitable and have created issue 15767 to track it.
msg169031 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-08-24 15:57
I don't agree that this is a blocker; would be nice to fix it, of course.
msg169036 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-08-24 16:07
It will get fixed today.
msg169093 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-08-24 21:44
I am running the test suite now using the "secret" attribute on ImportError. I tried to pass a flag, but locking became a bit messy/complicated. And I also realized that if I didn't do this then using different implementation of import_ in importlib wouldn't work because I would be hard-coding in what import implementation was used which would bypass the accelerated code in import.c. Plus with the compatibility issues I have had in the passed, I didn't want to skip a step of import that someone was probably relying on.

Once Python 3.4 comes out I will create a new exception that is raised when no module is found and catch that class specifically to avoid this hack.
msg169096 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-24 22:26
New changeset ca4bf8e10bc0 by Brett Cannon in branch 'default':
Issue #15316: Let exceptions raised during imports triggered by the
http://hg.python.org/cpython/rev/ca4bf8e10bc0
History
Date User Action Args
2012-08-24 22:27:06brett.cannonsetstatus: open -> closed
resolution: fixed
stage: test needed -> resolved
2012-08-24 22:26:06python-devsetnosy: + python-dev
messages: + msg169096
2012-08-24 21:44:56brett.cannonsetmessages: + msg169093
2012-08-24 16:07:31brett.cannonsetmessages: + msg169036
2012-08-24 15:57:55georg.brandlsetpriority: release blocker -> critical

messages: + msg169031
2012-08-22 19:46:29eric.snowsetnosy: + eric.snow
messages: + msg168907
2012-08-22 19:18:12chris.jerdoneksetmessages: + msg168905
2012-08-22 19:12:39brett.cannonsetmessages: + msg168904
2012-08-22 19:08:01chris.jerdoneksetfiles: + issue-15316-failing-test-1.patch
keywords: + patch
messages: + msg168903
2012-08-22 16:33:33brett.cannonsetpriority: normal -> release blocker
assignee: brett.cannon
messages: + msg168896

stage: test needed
2012-08-22 15:02:54jeffknuppsetnosy: + jeffknupp
2012-08-22 07:21:23chris.jerdoneksetnosy: + georg.brandl, r.david.murray
messages: + msg168846
2012-08-22 02:25:20chris.jerdoneksetkeywords: + 3.2regression
2012-08-22 00:25:26chris.jerdoneksetmessages: + msg168831
2012-08-22 00:15:24chris.jerdoneksetstatus: closed -> open

superseder: Wrong ImportError message with importlib ->
nosy: + brett.cannon, ncoghlan

messages: + msg168830
type: behavior
resolution: duplicate -> (no value)
2012-07-10 07:54:48amaury.forgeotdarcsetstatus: open -> closed

nosy: + amaury.forgeotdarc
messages: + msg165175

superseder: Wrong ImportError message with importlib
resolution: duplicate
2012-07-10 05:39:17chris.jerdonekcreate