classification
Title: pydoc.synopsis fails to load binary modules
Type: behavior Stage: patch review
Components: Versions: Python 3.3
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, brett.cannon, eric.snow, gregory.p.smith, koobs, larry, ncoghlan, python-dev, r.david.murray, vstinner
Priority: normal Keywords: patch

Created on 2014-01-04 22:27 by eric.snow, last changed 2014-11-21 09:34 by gregory.p.smith.

Files
File name Uploaded Description Edit
issue20123-fix-pydoc-synopsis.diff eric.snow, 2014-01-05 03:40 review
koobs-freebsd9-3.4-build57.log koobs, 2014-04-13 06:45
Messages (20)
msg207319 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-04 22:27
I'm guessing this is a very seldom (never?) used code path.  I've included a patch to test and fix the problem.  The patch includes several related tests for pydoc.

$ py3 -c 'import pydoc; pydoc.synopsis("/opt/python3.4/lib/python3.4/lib-dynload/time.cpython-34m.so")'
Traceback (most recent call last):
  File "/opt/python3.4/lib/python3.4/tokenize.py", line 368, in find_cookie
    line_string = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf4 in position 96: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/python3.4/lib/python3.4/pydoc.py", line 229, in synopsis
    file = tokenize.open(filename)
  File "/opt/python3.4/lib/python3.4/tokenize.py", line 429, in open
    encoding, lines = detect_encoding(buffer.readline)
  File "/opt/python3.4/lib/python3.4/tokenize.py", line 409, in detect_encoding
    encoding = find_cookie(first)
  File "/opt/python3.4/lib/python3.4/tokenize.py", line 373, in find_cookie
    raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration for '/opt/python3.4/lib/python3.4/lib-dynload/time.cpython-34m.so'
msg207321 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-04 22:37
FYI, I found this while working on issue #19703.
msg207328 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2014-01-05 02:20
Well please get a second opinion.  I don't know why you added me, I'm not qualified.
msg207330 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-05 02:22
Not sure why I nosy'ed you, either. :)
msg207334 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-01-05 02:47
In 2.7 the code just does an open.  Victor changed it to call tokenize.open in 3.2, but tokenize.open obviously only works on python source files.  So the logic of that method is now completely wrong.

I'm not sure the logic made a lot of sense even before...if the extension is binary it just closes it again.  So the initial open is for its side effect of returning None if the file can't be opened.

Anyway, the logic of that method clearly needs to be rewritten.  And obviously there are some missing tests....

I don't see any patch attached to the issue, by the way.
msg207335 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-05 02:52
Thanks for taking a look.  Here's the patch that I totally forget to attach.
msg207337 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-01-05 03:15
Is it the case that given a filename, it might be possible to load a module even if open(filename) fails?

I think the logic is clearer in the form where it is not pulled out into a separate helper function.  You can avoid the double check on the extension by doing:

    if filename.endswith(importlib.machinery.BYTECODE_SUFFIXES):
        loader = importlib.machinery.SourcelessFileLoader('__temp__',
                                                          filename)
    elif filename.endswith(importlib.machinery.EXTENSION_SUFFIXES):
        loader = importlib.machinery.ExtensionFileLoader('__temp__',
                                                         filename)
    else:
        loader = None

    if loader:
        xxxxx
    else:
        xxxxx
msg207344 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-05 03:40
Sounds good to me.  Here's an updated patch.
msg207345 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-01-05 03:47
Looks good, except you can replace those any calls with simple calls to endswith using a tuple of strings (I forgot the call to tuple in my example).
msg207346 - (view) Author: Roundup Robot (python-dev) Date: 2014-01-05 03:58
New changeset d6c3fb8d5f84 by Eric Snow in branch 'default':
Issue 20123: Fix pydoc.synopsis() for "binary" modules.
http://hg.python.org/cpython/rev/d6c3fb8d5f84
msg207347 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-05 04:00
Ah, I missed that in your earlier suggestion.  I followed your recommendation.  Thanks for that. :)
msg207348 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-05 04:44
This broke one of the FreeBSD buildbots:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%203.x/builds/6102
msg207349 - (view) Author: Roundup Robot (python-dev) Date: 2014-01-05 05:05
New changeset ff3be21338d5 by Eric Snow in branch 'default':
Issue 20123: try using a different builtin module in a pydoc test.
http://hg.python.org/cpython/rev/ff3be21338d5
msg207350 - (view) Author: Roundup Robot (python-dev) Date: 2014-01-05 06:06
New changeset efcf163d04f5 by Eric Snow in branch 'default':
Issue 20123: Disable a problematic test.
http://hg.python.org/cpython/rev/efcf163d04f5
msg207351 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-05 06:08
I've run out of time to trouble-shoot the failure (specific to 1 buildbot).  Until I can get back to it, I've disabled the problematic test (even though it's only a problem on 1 buildbot).
msg207354 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-05 07:11
The buildbot is happy again.  I'll address fixing that test in issue #20128.
msg207356 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2014-01-05 07:14
Thanks for seeing this through, Eric.
msg207673 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2014-01-08 05:59
I realized today that this should probably be fixed in 3.3 as well, as RDM implied by marking it as also a 3.3 bug. :)
msg215997 - (view) Author: Kubilay Kocak (koobs) Date: 2014-04-13 06:45
koobs-freebsd9 (3.4) buildbot has also been failing for a while on what seems to be this changeset:

======================================================================
ERROR: test_synopsis_sourceless (test.test_pydoc.PydocDocTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.4.koobs-freebsd9/build/Lib/test/test_pydoc.py", line 504, in test_synopsis_sourceless
    synopsis = pydoc.synopsis(filename)
  File "/usr/home/buildbot/python/3.4.koobs-freebsd9/build/Lib/pydoc.py", line 238, in synopsis
    mtime = os.stat(filename).st_mtime
FileNotFoundError: [Errno 2] No such file or directory: '/usr/home/buildbot/python/3.4.koobs-freebsd9/build/Lib/__pycache__/os.cpython-34.pyc'

Full buildlog is attached.
msg231470 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2014-11-21 09:34
fyi - tracking the new issue koobs reported in http://bugs.python.org/issue22910
History
Date User Action Args
2014-11-21 09:34:48gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg231470
2014-04-13 06:45:24koobssetfiles: + koobs-freebsd9-3.4-build57.log

messages: + msg215997
2014-04-12 15:31:33koobssetnosy: + koobs
2014-02-23 00:00:23Arfreversetnosy: + Arfrever
2014-01-08 05:59:47eric.snowsetstatus: closed -> open
versions: - Python 3.4
messages: + msg207673

resolution: fixed ->
stage: resolved -> patch review
2014-01-05 07:14:32larrysetmessages: + msg207356
2014-01-05 07:11:50eric.snowsetstatus: open -> closed

messages: + msg207354
2014-01-05 06:08:58eric.snowsetmessages: + msg207351
2014-01-05 06:06:10python-devsetmessages: + msg207350
2014-01-05 05:05:37python-devsetmessages: + msg207349
2014-01-05 04:44:01eric.snowsetstatus: closed -> open

messages: + msg207348
2014-01-05 04:00:08eric.snowsetstatus: open -> closed
resolution: fixed
messages: + msg207347

stage: patch review -> resolved
2014-01-05 03:58:39python-devsetnosy: + python-dev
messages: + msg207346
2014-01-05 03:47:41r.david.murraysetmessages: + msg207345
2014-01-05 03:40:41eric.snowsetfiles: - issue20123-fix-pydoc-synopsis.diff
2014-01-05 03:40:34eric.snowsetfiles: + issue20123-fix-pydoc-synopsis.diff

messages: + msg207344
2014-01-05 03:15:09r.david.murraysetmessages: + msg207337
2014-01-05 02:52:25eric.snowsetfiles: + issue20123-fix-pydoc-synopsis.diff
keywords: + patch
messages: + msg207335
2014-01-05 02:47:42r.david.murraysetnosy: + r.david.murray, vstinner

messages: + msg207334
versions: + Python 3.3, Python 3.4
2014-01-05 02:22:51eric.snowsetmessages: + msg207330
2014-01-05 02:20:54larrysetnosy: + brett.cannon, ncoghlan
messages: + msg207328
2014-01-04 22:37:43eric.snowsetmessages: + msg207321
2014-01-04 22:36:27eric.snowlinkissue19703 dependencies
2014-01-04 22:27:05eric.snowcreate