classification
Title: pydoc modules/help('modules') crash in dirs with unreadable subdirs
Type: crash Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: pkgutil.walk_packages fails on write-only directory in sys.path
View: 7367
Assigned To: ned.deily Nosy List: eric.araujo, ned.deily, okopnik
Priority: normal Keywords:

Created on 2011-04-24 03:34 by okopnik, last changed 2011-10-07 06:08 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
pydoc_crash_test okopnik, 2011-04-27 15:43 pydoc crash test
pydoc_crash_test okopnik, 2011-04-27 17:27
Messages (6)
msg134323 - (view) Author: Ben Okopnik (okopnik) Date: 2011-04-24 03:34
Long-standing problem (happens in every Python version I've tested). The usual situation is when invoking Python (and then "help('modules')") or "pydoc modules" in /tmp, but also happens when located anywhere with unreadable subdirs:

ben@Jotunheim:~$ mkdir /tmp/foo; cd /tmp/foo
ben@Jotunheim:/tmp/foo$ mkdir bar; sudo chmod 000 bar
[sudo] password for ben: 
ben@Jotunheim:/tmp/foo$ pydoc modules

Please wait a moment while I gather a list of all available modules...

Traceback (most recent call last):
  File "/usr/bin/pydoc2.6", line 5, in <module>
    pydoc.cli()
  File "/usr/lib/python2.6/pydoc.py", line 2309, in cli
    help.help(arg)
  File "/usr/lib/python2.6/pydoc.py", line 1765, in help
    elif request == 'modules': self.listmodules()
  File "/usr/lib/python2.6/pydoc.py", line 1886, in listmodules
    ModuleScanner().run(callback, onerror=onerror)
  File "/usr/lib/python2.6/pydoc.py", line 1937, in run
    for importer, modname, ispkg in pkgutil.walk_packages(onerror=onerror):
  File "/usr/lib/python2.6/pkgutil.py", line 105, in walk_packages
    for importer, name, ispkg in iter_modules(path, prefix):
  File "/usr/lib/python2.6/pkgutil.py", line 147, in iter_modules
    for name, ispkg in iter_importer_modules(i, prefix):
  File "/usr/lib/python2.6/pkgutil.py", line 211, in iter_modules
    for fn in os.listdir(path):
OSError: [Errno 13] Permission denied: './bar'

Proposed patch:

Seems like an easy fix. In Python 3.1.2, change line 206 in /usr/lib/python3.1/pkgutil.py from

if not modname and os.path.isdir(path) and '.' not in fn:

to

if not modname and os.path.isdir(path) and '.' not in fn and os.access(path, os.R_OK):

Other versions much the same (although the specified line number will probably be different.)


Best regards,
Ben Okopnik
msg134586 - (view) Author: Ben Okopnik (okopnik) Date: 2011-04-27 15:43
Here's a test that should exercise every version of "pydoc" installed on the system:

mkdir -p /tmp/foo/bar; cd /tmp/foo; chmod 0 bar
for n in `whereis -b pydoc`; do echo "**** $n ****"; $n modules; done

Tested under Ubuntu with bash and sh; should work fine with any Bourne-derived shell that supports 'whereis'. Please see attached file.
msg134588 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-04-27 15:49
The script is bugged, since whereis prefixes its output with its argument (i.e. here “pydoc: ”).  It’s not a concern anyway: branches open for bugfixes are 2.7, 3.1, 3.2 and 3.3, so when we have a test (preferably as a patch to test_pydoc.py, see http://docs.python.org/devguide for more guidelines on testing and contributing) we check versions and fix where needed.
msg134600 - (view) Author: Ben Okopnik (okopnik) Date: 2011-04-27 17:27
Trivial fix: please see attached. As to test_pydoc.py, I don't know the system well enough to fiddle with it, but something like this should work (untested):

    def test_unreadable_dir(self):
        ''' pydoc should handle unreadable subdirs gracefully '''

    @contextmanager
    def mk_unreadable_dir():
        top_level_dir = tempfile.mkdtemp()
        bad_dir = tempfile.mkdtemp(dir=top_level_dir)
        os.chmod(bad_dir, 0)
        os.chdir(top_level_dir)
        yield
        os.removedirs(top_level_dir)

    with mk_unreadable_dir():
        doc = pydoc.render_doc('modules')
        self.assertTrue("modules" in doc)
msg134617 - (view) Author: Ben Okopnik (okopnik) Date: 2011-04-27 21:01
Whoops... with all of that, I just realized that this bug should be filed against pkgutil, not pydoc (pydoc, of course, calls pkgutil to do the path resolution, which is where this crash occurs.) My bad.

>>> import pkgutil
>>> inst = pkgutil.ImpImporter(path='/tmp')
>>> list(inst.iter_modules())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.1/pkgutil.py", line 209, in iter_modules
    for fn in os.listdir(path):
OSError: [Errno 13] Permission denied: '/tmp/orbit-gdm'
msg145056 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-10-07 06:08
Thanks for your suggested code and test. Somewhat different fixes to pkgutil have been added by the changes for Issue7367 and a necessary backport of a fix for pydoc from 3.x to 2.7 was made for Issue7425.  With these fixes in place, pydoc for 3.3.0, 3.2.3, and 2.7.3 should not exhibit these problems anymore.
History
Date User Action Args
2011-10-07 06:08:33ned.deilysetstatus: open -> closed
resolution: duplicate
messages: + msg145056

superseder: pkgutil.walk_packages fails on write-only directory in sys.path
stage: test needed -> resolved
2011-08-06 02:59:32ned.deilysetassignee: ned.deily

nosy: + ned.deily
versions: - Python 3.1
2011-04-27 21:01:39okopniksetmessages: + msg134617
2011-04-27 17:27:44okopniksetfiles: + pydoc_crash_test

messages: + msg134600
2011-04-27 15:49:30eric.araujosetmessages: + msg134588
2011-04-27 15:43:34okopniksetfiles: + pydoc_crash_test

messages: + msg134586
2011-04-27 08:50:26eric.araujosetnosy: + eric.araujo
stage: test needed

components: + Library (Lib), - Demos and Tools
versions: - Python 2.6, Python 2.5, Python 3.4
2011-04-24 03:34:52okopnikcreate