classification
Title: Improve the robustness of "pydoc -k" in the face of broken modules
Type: behavior Stage: resolved
Components: Demos and Tools Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: Christian.S..Perone, ajaksu2, dmalcolm, eric.araujo, ned.deily, pitrou, python-dev, r.david.murray
Priority: normal Keywords: patch

Created on 2009-12-02 20:49 by dmalcolm, last changed 2011-10-06 22:11 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
make-pydoc-more-robust-001.patch dmalcolm, 2009-12-02 20:49 Patch to make "pydoc -k" silently ignore all exceptions when importing modules, not just ImportError review
Messages (10)
msg95918 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2009-12-02 20:49
I see occasional failures where a broken module prevents "pydoc -k" ("apropos") from working.

Examples of failures (from our downstream bug tracker) are:
https://bugzilla.redhat.com/show_bug.cgi?id=461419 : "pydoc -k" yields "NameError: name 'wglUseFontBitmapsW' is not 
defined"
(a broken OpenGL/WGL/__init__.py module)

https://bugzilla.redhat.com/show_bug.cgi?id=447779 : "pydoc turbogears" doesn't work
(broken TurboGears module)

https://bugzilla.redhat.com/show_bug.cgi?id=246212 : "pydoc -k searchterm" MemoryError. permission denied.
(module only intended to be importable as "root" user)

In each case one or more of the many modules on the system are broken, and importing them leads to an exception being 
raised that isn't ImportError.

In each case, the exception bubbles up through the pydoc call ands leads to it exiting.

Obviously the broken modules should be fixed, but it seems to me that pydoc could be more robust against this situation.

I'm attaching a simple patch which makes "pydoc -k" more robust against this situation, by silently discarding all 
exceptions raised by imported modules.

How does this look?

One downstream bug report requested taking it further:
> Running "pydoc -k" should catch all exceptions while importing modules, and
> display failed modules _after_ all positive keyword matches (not in between).
> It also should redirect stdout/stderr while importing so error message e.a.
> don't clutter up the list of hits.  
to deal with broken modules that spew error messages (this was in https://bugzilla.redhat.com/show_bug.cgi?id=461419#c3 
); I've seen some do it to stdout and some to stderr.

How does this sound?  I can try to update the patch for this too, if this sounds sane to you.
msg102668 - (view) Author: Daniel Diniz (ajaksu2) (Python triager) Date: 2010-04-09 02:00
Nice improvement. This would also solve the "help(), modules" brokenness, right?
msg102745 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-09 20:08
Is it a dup of issue1785?
msg102760 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-04-10 01:02
It doesn't look like it is the same bug to me.  This one is more general.
msg118731 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-10-14 23:24
I notice this issue is in stage "unit test needed".

It's not clear to me how to add a unit test for this: one idea I had is to create a broken module in some subdir somewhere, and invoke "pydoc -k" as a subprocess with PYTHONPATH containing the extra subdir (and assert no traceback seen in stderr of the child).

Does that sound sane?
msg118737 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-10-15 02:00
Yes, and I can't think of any other way to approach it.
msg143542 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-09-05 16:12
Does this patch solves only pydoc -k, not all “invalid spam module makes pydoc crash”?  (I wanted to consolidate the handful of reports we have into one but did not get the time.)
msg144811 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-10-03 10:20
It turns out that the proposed fix here for pydoc was independently added in the early days of Python 3 but was not backported.  That fix for 2.7 plus a fix-in-progress for Issue7367 (for both 2.7 and 3.x) and additional test cases (also in progress) should address most if not all of the pydoc crash issues.  I'll also review the other open issues.
msg145042 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-10-06 21:43
New changeset 45862f4ab1c5 by Ned Deily in branch '2.7':
Issue #7425: Refactor test_pydoc test case for '-k' behavior and add
http://hg.python.org/cpython/rev/45862f4ab1c5

New changeset 3acf90f71178 by Ned Deily in branch '2.7':
Issue #7425: Prevent pydoc -k failures due to module import errors.
http://hg.python.org/cpython/rev/3acf90f71178

New changeset 6a45f917f167 by Ned Deily in branch '3.2':
Issue #7425: Refactor test_pydoc test case for '-k' behavior and add
http://hg.python.org/cpython/rev/6a45f917f167

New changeset add444274c3d by Ned Deily in branch '2.7':
Issue #7425 and Issue #7367: add NEWS items.
http://hg.python.org/cpython/rev/add444274c3d
msg145044 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-10-06 22:10
The applied changesets backport the "ignore exceptions" fix for pydoc -k from 3.x to 2.7 and also refactor test_pydoc to remove unneeded complexity and add test cases for importing bad packages and unreadable package directories (a problem addressed in Issue7367).  Applied to 2.7 (for 2.7.3), 3.2 (for 3.2.3), and default (for 3.3.0).
History
Date User Action Args
2012-02-04 16:09:55ned.deilylinkissue13880 superseder
2011-10-07 07:10:20ned.deilylinkissue7015 superseder
2011-10-06 22:11:02ned.deilysetstage: patch review -> resolved
2011-10-06 22:10:41ned.deilysetstatus: open -> closed
resolution: fixed
messages: + msg145044
2011-10-06 21:43:08python-devsetnosy: + python-dev
messages: + msg145042
2011-10-03 10:20:10ned.deilysetassignee: ned.deily
messages: + msg144811
2011-09-05 16:12:20eric.araujosetnosy: + eric.araujo

messages: + msg143542
title: [PATCH] Improve the robustness of "pydoc -k" in the face of broken modules -> Improve the robustness of "pydoc -k" in the face of broken modules
2011-09-05 01:33:12ned.deilysetnosy: + ned.deily, Christian.S..Perone

versions: + Python 3.3, - Python 3.1
2011-09-05 01:31:25ned.deilylinkissue12894 superseder
2011-01-03 20:19:17pitrousetnosy: pitrou, ajaksu2, r.david.murray, dmalcolm
stage: test needed -> patch review
versions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6
2010-10-15 02:00:55r.david.murraysetmessages: + msg118737
2010-10-14 23:24:32dmalcolmsetmessages: + msg118731
2010-04-10 01:02:07r.david.murraysetnosy: + r.david.murray
messages: + msg102760
2010-04-09 20:08:58pitrousetnosy: + pitrou
messages: + msg102745
2010-04-09 02:00:03ajaksu2setpriority: normal

nosy: + ajaksu2
messages: + msg102668

type: behavior
stage: test needed
2009-12-02 20:49:29dmalcolmcreate