Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\Python30\lib\site.py", line 427, in __call__
    return pydoc.help(*args, **kwds)
  File "c:\Python30\lib\pydoc.py", line 1675, in __call__
    self.interact()
  File "c:\Python30\lib\pydoc.py", line 1693, in interact
    self.help(request)
  File "c:\Python30\lib\pydoc.py", line 1711, in help
    self.listmodules(request.split()[1])
  File "c:\Python30\lib\pydoc.py", line 1799, in listmodules
    apropos(key)
  File "c:\Python30\lib\pydoc.py", line 1913, in apropos
    ModuleScanner().run(callback, key, onerror=onerror)
  File "c:\Python30\lib\pydoc.py", line 1875, in run
    source = loader.get_source(modname)
  File "c:\Python30\lib\pkgutil.py", line 293, in get_source
    self.source = self.file.read()
  File "c:\Python30\lib\io.py", line 1720, in read
    decoder = self._decoder or self._get_decoder()
  File "c:\Python30\lib\io.py", line 1506, in _get_decoder
    make_decoder = codecs.getincrementaldecoder(self._encoding)
  File "c:\Python30\lib\codecs.py", line 960, in getincrementaldecoder
    decoder = lookup(encoding).incrementaldecoder
LookupError: unknown encoding: uft-8

When enumerating all possible modules, the help system tries to import 
test.bad_coding.py :-(
I tried to catch this LookupError, but then it fail on test.badsyntax_pep3120.py, 
with a SyntaxError
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python30\lib\site.py", line 427, in __call__
    return pydoc.help(*args, **kwds)
  File "c:\python30\lib\pydoc.py", line 1675, in __call__
    self.interact()
  File "c:\python30\lib\pydoc.py", line 1693, in interact
    self.help(request)
  File "c:\python30\lib\pydoc.py", line 1711, in help
    self.listmodules(request.split()[1])
  File "c:\python30\lib\pydoc.py", line 1799, in listmodules
    apropos(key)
  File "c:\python30\lib\pydoc.py", line 1913, in apropos
    ModuleScanner().run(callback, key, onerror=onerror)
  File "c:\python30\lib\pydoc.py", line 1872, in run
    loader = importer.find_module(modname)
  File "C:\Python30\lib\pkgutil.py", line 186, in find_module
    file, filename, etc = imp.find_module(subname, path)
SyntaxError: Non-UTF-8 code starting with '\xf6' in file (null) on line 1, but n
o encoding declared; see http://python.org/dev/peps/pep-0263/ for details


[Note: I don't like the (null) as a filename...]

Amaury, I don't see where imp.find_module() tries to do a compile()
(which would trigger the SyntaxError). Is that really the full traceback ?

Regarding the strategy of pydoc to try to list all modules: I don't
think that's such a good idea. It's likely to take a long time to
compile such a list on a moderately used Python system and can have lots
of unwanted side effects via import hooks.

Instead, pydoc should just say that a module was not specified and be
done with it.

Yes, it is the full traceback.

The modules are not compiled nor imported. 
But to load the source code, importer.find_module() needs to open the
file in text mode, so it calls PyTokenizer_FindEncoding() (this is the
second failure, with test.badsyntax_pep3120.py), then the encoding is
used to decode text (this is the first failure, with test.bad_coding.py)

> Instead, pydoc should just say that a module was not specified 
> and be done with it.
I'm not sure to understand.

On 2008-12-05 11:24, Amaury Forgeot d'Arc wrote:
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
> 
> Yes, it is the full traceback.
> 
> The modules are not compiled nor imported. 
> But to load the source code, importer.find_module() needs to open the
> file in text mode, so it calls PyTokenizer_FindEncoding() (this is the
> second failure, with test.badsyntax_pep3120.py), then the encoding is
> used to decode text (this is the first failure, with test.bad_coding.py)

I see.

>> Instead, pydoc should just say that a module was not specified 
>> and be done with it.
>
> I'm not sure to understand.

Sorry, I didn't see the attached PNG screenshot until now. Forget that
comment.

What I don't understand is that this works:

help> modules

Please wait a moment while I gather a list of all available modules...
...

while this doesn't:

help> modules web

Here is a list of matching modules.  Enter any module name to get more help.

Traceback (most recent call last):
...

Shouldn't the first method also cause a traceback ?

Explanation is simple: 
help("modules") just lists all files but do not open them. 
help("modules web") has to search text inside.

On 2008-12-05 11:49, Amaury Forgeot d'Arc wrote:
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
> 
> Explanation is simple: 
> help("modules") just lists all files but do not open them. 
> help("modules web") has to search text inside.

Ok, thanks.

What if we skip any errors that occur during this scan ? After all,
it's just a help function, not a vital system operation.

Attached patch ignores any Exception during module import.
Note that KeyboardInterrupt is not caught: it is not an Exception.

I could provide a similar patch for 2.7, if there is some interest; but
no crash was reported.

On 2008-12-05 13:28, Amaury Forgeot d'Arc wrote:
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
> 
> Attached patch ignores any Exception during module import.
> Note that KeyboardInterrupt is not caught: it is not an Exception.

Is there a reason why "formatter" is imported inside the method
rather than at module scope ?

Other than that the patch looks good.

> I could provide a similar patch for 2.7, if there is some interest; but
> no crash was reported.

Python 2.6 and 2.7 have the same problem:

help> modules web

Here is a list of matching modules.  Enter any module name to get more help.

...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/python-2.6-ucs2/lib/python2.6/site.py", line 429, in __call__
    return pydoc.help(*args, **kwds)
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1687, in __call__
    self.interact()
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1705, in interact
    self.help(request)
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1723, in help
    self.listmodules(split(request)[1])
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1812, in
listmodules
    apropos(key)
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1912, in apropos
    ModuleScanner().run(callback, key)
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1877, in run
    for importer, modname, ispkg in pkgutil.walk_packages(onerror=onerror):
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pkgutil.py", line 110, in
walk_packages
    __import__(name)
  File "/home/lemburg/lib/DNS/__init__.py", line 13, in <module>
    import Type,Opcode,Status,Class
  File "/home/lemburg/lib/DNS/Type.py", line 67
SyntaxError: Non-ASCII character '\xf6' in file /home/lemburg/lib/DNS/Type.py on
line 67, but no encoding declared; see http://www.python.org/peps/pep-0263.html
for details

(pydoc will look in all modules on sys.path, so chances are high
that users will run into similar problems)

I can still reproduce on py3

>>> help("modules anything")

Traceback (most recent call last):
...

This patch works (on Py3.1a1). 


Amaury, are you still o.k with catching Exception rather
than just (SyntaxError, UnicodeDecodeError, ImportError)?

The code-refactoring does clean the structure up well.

Cheers,

Jervis

This appears to be the same issue as described in 5453 in which I
supplied a patch for release30-maint and py3k branches.

It's probably useful to address both issue 4540 & 5453 at the same time.

I would have added to this issue, but I didn't see it when I raised 5453.

On a side note, the line importer.find_module(modname) in pydoc.py line
1925 appears that it may return None.  If this is the case then the None
return value should be handled.

I faced similar issue. The solution is either to add tests as an
exception in pydoc or to move tests to /usr/share or /usr/share/doc.

I favor the second solution

This was fixed together with issue5453