classification
Title: typo in a module describes utf-8 as uft-8
Type: crash Stage: patch review
Components: Versions: Python 3.0
process
Status: closed Resolution: out of date
Dependencies: Superseder: pydoc -k fails (release30-maint patch)
View: 5453
Assigned To: amaury.forgeotdarc Nosy List: amaury.forgeotdarc, dariusp, djinn, jdwhitley, john.weldon, lemburg
Priority: normal Keywords: needs review, patch

Created on 2008-12-04 23:38 by john.weldon, last changed 2009-07-15 21:43 by amaury.forgeotdarc. This issue is now closed.

Files
File name Uploaded Description Edit
python3bug.png john.weldon, 2008-12-04 23:38 screenshot
help_badencoding.patch amaury.forgeotdarc, 2008-12-05 12:28
Messages (13)
msg76945 - (view) Author: John Weldon (john.weldon) Date: 2008-12-04 23:38
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\Python30\lib\site.py", line 427, in __call__
    return pydoc.help(*args, **kwds)
  File "c:\Python30\lib\pydoc.py", line 1675, in __call__
    self.interact()
  File "c:\Python30\lib\pydoc.py", line 1693, in interact
    self.help(request)
  File "c:\Python30\lib\pydoc.py", line 1711, in help
    self.listmodules(request.split()[1])
  File "c:\Python30\lib\pydoc.py", line 1799, in listmodules
    apropos(key)
  File "c:\Python30\lib\pydoc.py", line 1913, in apropos
    ModuleScanner().run(callback, key, onerror=onerror)
  File "c:\Python30\lib\pydoc.py", line 1875, in run
    source = loader.get_source(modname)
  File "c:\Python30\lib\pkgutil.py", line 293, in get_source
    self.source = self.file.read()
  File "c:\Python30\lib\io.py", line 1720, in read
    decoder = self._decoder or self._get_decoder()
  File "c:\Python30\lib\io.py", line 1506, in _get_decoder
    make_decoder = codecs.getincrementaldecoder(self._encoding)
  File "c:\Python30\lib\codecs.py", line 960, in getincrementaldecoder
    decoder = lookup(encoding).incrementaldecoder
LookupError: unknown encoding: uft-8
msg76955 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-12-05 00:47
When enumerating all possible modules, the help system tries to import 
test.bad_coding.py :-(
I tried to catch this LookupError, but then it fail on test.badsyntax_pep3120.py, 
with a SyntaxError
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python30\lib\site.py", line 427, in __call__
    return pydoc.help(*args, **kwds)
  File "c:\python30\lib\pydoc.py", line 1675, in __call__
    self.interact()
  File "c:\python30\lib\pydoc.py", line 1693, in interact
    self.help(request)
  File "c:\python30\lib\pydoc.py", line 1711, in help
    self.listmodules(request.split()[1])
  File "c:\python30\lib\pydoc.py", line 1799, in listmodules
    apropos(key)
  File "c:\python30\lib\pydoc.py", line 1913, in apropos
    ModuleScanner().run(callback, key, onerror=onerror)
  File "c:\python30\lib\pydoc.py", line 1872, in run
    loader = importer.find_module(modname)
  File "C:\Python30\lib\pkgutil.py", line 186, in find_module
    file, filename, etc = imp.find_module(subname, path)
SyntaxError: Non-UTF-8 code starting with '\xf6' in file (null) on line 1, but n
o encoding declared; see http://python.org/dev/peps/pep-0263/ for details


[Note: I don't like the (null) as a filename...]
msg76996 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2008-12-05 09:48
Amaury, I don't see where imp.find_module() tries to do a compile()
(which would trigger the SyntaxError). Is that really the full traceback ?

Regarding the strategy of pydoc to try to list all modules: I don't
think that's such a good idea. It's likely to take a long time to
compile such a list on a moderately used Python system and can have lots
of unwanted side effects via import hooks.

Instead, pydoc should just say that a module was not specified and be
done with it.
msg76999 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-12-05 10:24
Yes, it is the full traceback.

The modules are not compiled nor imported. 
But to load the source code, importer.find_module() needs to open the
file in text mode, so it calls PyTokenizer_FindEncoding() (this is the
second failure, with test.badsyntax_pep3120.py), then the encoding is
used to decode text (this is the first failure, with test.bad_coding.py)

> Instead, pydoc should just say that a module was not specified 
> and be done with it.
I'm not sure to understand.
msg77000 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2008-12-05 10:39
On 2008-12-05 11:24, Amaury Forgeot d'Arc wrote:
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
> 
> Yes, it is the full traceback.
> 
> The modules are not compiled nor imported. 
> But to load the source code, importer.find_module() needs to open the
> file in text mode, so it calls PyTokenizer_FindEncoding() (this is the
> second failure, with test.badsyntax_pep3120.py), then the encoding is
> used to decode text (this is the first failure, with test.bad_coding.py)

I see.

>> Instead, pydoc should just say that a module was not specified 
>> and be done with it.
>
> I'm not sure to understand.

Sorry, I didn't see the attached PNG screenshot until now. Forget that
comment.

What I don't understand is that this works:

help> modules

Please wait a moment while I gather a list of all available modules...
...

while this doesn't:

help> modules web

Here is a list of matching modules.  Enter any module name to get more help.

Traceback (most recent call last):
...

Shouldn't the first method also cause a traceback ?
msg77001 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-12-05 10:49
Explanation is simple: 
help("modules") just lists all files but do not open them. 
help("modules web") has to search text inside.
msg77002 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2008-12-05 10:52
On 2008-12-05 11:49, Amaury Forgeot d'Arc wrote:
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
> 
> Explanation is simple: 
> help("modules") just lists all files but do not open them. 
> help("modules web") has to search text inside.

Ok, thanks.

What if we skip any errors that occur during this scan ? After all,
it's just a help function, not a vital system operation.
msg77009 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-12-05 12:28
Attached patch ignores any Exception during module import.
Note that KeyboardInterrupt is not caught: it is not an Exception.

I could provide a similar patch for 2.7, if there is some interest; but
no crash was reported.
msg77065 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2008-12-05 20:51
On 2008-12-05 13:28, Amaury Forgeot d'Arc wrote:
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
> 
> Attached patch ignores any Exception during module import.
> Note that KeyboardInterrupt is not caught: it is not an Exception.

Is there a reason why "formatter" is imported inside the method
rather than at module scope ?

Other than that the patch looks good.

> I could provide a similar patch for 2.7, if there is some interest; but
> no crash was reported.

Python 2.6 and 2.7 have the same problem:

help> modules web

Here is a list of matching modules.  Enter any module name to get more help.

...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/python-2.6-ucs2/lib/python2.6/site.py", line 429, in __call__
    return pydoc.help(*args, **kwds)
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1687, in __call__
    self.interact()
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1705, in interact
    self.help(request)
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1723, in help
    self.listmodules(split(request)[1])
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1812, in
listmodules
    apropos(key)
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1912, in apropos
    ModuleScanner().run(callback, key)
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pydoc.py", line 1877, in run
    for importer, modname, ispkg in pkgutil.walk_packages(onerror=onerror):
  File "/usr/local/python-2.6-ucs2/lib/python2.6/pkgutil.py", line 110, in
walk_packages
    __import__(name)
  File "/home/lemburg/lib/DNS/__init__.py", line 13, in <module>
    import Type,Opcode,Status,Class
  File "/home/lemburg/lib/DNS/Type.py", line 67
SyntaxError: Non-ASCII character '\xf6' in file /home/lemburg/lib/DNS/Type.py on
line 67, but no encoding declared; see http://www.python.org/peps/pep-0263.html
for details

(pydoc will look in all modules on sys.path, so chances are high
that users will run into similar problems)
msg83421 - (view) Author: Jervis Whitley (jdwhitley) Date: 2009-03-10 09:03
I can still reproduce on py3

>>> help("modules anything")

Traceback (most recent call last):
...

This patch works (on Py3.1a1). 


Amaury, are you still o.k with catching Exception rather
than just (SyntaxError, UnicodeDecodeError, ImportError)?

The code-refactoring does clean the structure up well.

Cheers,

Jervis
msg83431 - (view) Author: (dariusp) Date: 2009-03-10 12:53
This appears to be the same issue as described in 5453 in which I
supplied a patch for release30-maint and py3k branches.

It's probably useful to address both issue 4540 & 5453 at the same time.

I would have added to this issue, but I didn't see it when I raised 5453.

On a side note, the line importer.find_module(modname) in pydoc.py line
1925 appears that it may return None.  If this is the case then the None
return value should be handled.
msg87608 - (view) Author: Supreet (djinn) Date: 2009-05-12 06:46
I faced similar issue. The solution is either to add tests as an
exception in pydoc or to move tests to /usr/share or /usr/share/doc.

I favor the second solution
msg90545 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-07-15 21:43
This was fixed together with issue5453
History
Date User Action Args
2009-07-15 21:43:10amaury.forgeotdarcsetstatus: open -> closed
resolution: out of date
superseder: pydoc -k fails (release30-maint patch)
messages: + msg90545
2009-05-12 14:30:46ajaksu2linkissue4135 superseder
2009-05-12 06:46:59djinnsetnosy: + djinn
messages: + msg87608
2009-03-10 12:53:37dariuspsetnosy: + dariusp
messages: + msg83431
2009-03-10 09:03:52jdwhitleysetnosy: + jdwhitley
messages: + msg83421
2008-12-05 22:22:30amaury.forgeotdarcsetassignee: amaury.forgeotdarc
2008-12-05 20:51:27lemburgsetmessages: + msg77065
2008-12-05 12:28:07amaury.forgeotdarcsetkeywords: + patch, needs review
files: + help_badencoding.patch
messages: + msg77009
stage: patch review
2008-12-05 10:52:41lemburgsetmessages: + msg77002
2008-12-05 10:49:20amaury.forgeotdarcsetmessages: + msg77001
2008-12-05 10:39:38lemburgsetmessages: + msg77000
2008-12-05 10:24:55amaury.forgeotdarcsetmessages: + msg76999
2008-12-05 09:48:18lemburgsetnosy: + lemburg
messages: + msg76996
2008-12-05 00:47:49amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg76955
2008-12-04 23:38:42john.weldoncreate