classification
Title: pydoc generates invalid docs.python.org link for xml.etree.ElementTree and other modules
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, eric.araujo, eric.snow, georg.brandl, kau8hik, martin.panter, mgedmin, python-dev, r.david.murray, sean.rodman, terry.reedy
Priority: high Keywords: patch

Created on 2012-11-16 10:46 by mgedmin, last changed 2016-06-12 08:00 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
issue16484_python2.7.patch sean.rodman, 2014-02-26 22:33 fixed patch for python 2.7 review
issue16484_python3.2.patch sean.rodman, 2014-02-26 22:56 patch for pydoc module docs link (python 3.2 version) review
issue16484_python3.3.patch sean.rodman, 2014-02-26 22:56 patch for pydoc module docs link (python 3.3 version) review
Issue16484_rev_python3.6.patch kau8hik, 2016-06-03 19:02 patch for pydoc module docs link with tests(python 3.6 version) review
issue16484_test_python3.6.patch kau8hik, 2016-06-12 02:58 patch for pydoc module tests(python 3.6 version) review
Messages (33)
msg175675 - (view) Author: Marius Gedminas (mgedmin) * Date: 2012-11-16 10:46
Do this:

  pydoc2.7 xml.etree.ElementTree

then click on the MODULE DOCS link, which is http://docs.python.org/library/xml.etree.ElementTree

You're redirected to http://docs.python.org/2/library/xml.etree.ElementTree which is a 404 page.
msg175706 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-11-16 19:15
The redirect is not the problem.  The page never existed.
msg175713 - (view) Author: Marius Gedminas (mgedmin) * Date: 2012-11-16 20:25
A working link is http://docs.python.org/2/library/xml.etree.elementtree.html or http://docs.python.org/2/library/xml.etree.elementtree (with no .html at the end).

Looks like capitalization is causing a problem.

Did these pydoc links ever work?

Would it be possible to make them work?

There's a limited number of standard library modules with names that contain capital letters.  find /usr/lib/python2.7 -name dist-packages -prune -o -name '*[A-Z]*' | wc -l gives me 211 names, but this is an imprecise upper bound because it includes files like 'distutils/README' that aren't modules or packages.

Can I do anything to help?
msg175784 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-11-17 17:30
A patch and test making sure that pydoc generates the right URIs for upper-case modules would help.
msg177029 - (view) Author: Marius Gedminas (mgedmin) * Date: 2012-12-06 06:55
Considering many existing Python installations out there would it be possible to fix this on the server side?  I.e. lowercase the URL while redirecting?
msg177035 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-12-06 10:26
Adding a redirect should be easy, yes.
msg212155 - (view) Author: Sean Rodman (sean.rodman) * Date: 2014-02-25 01:17
I could try to create the patch for pydoc if you would like for me to.
msg212156 - (view) Author: Sean Rodman (sean.rodman) * Date: 2014-02-25 02:20
To display my ignorance, I have run the pydoc command listed in the original message but I can't actually see where it lists the url. Or even where it has a link.
msg212220 - (view) Author: Marius Gedminas (mgedmin) * Date: 2014-02-25 21:53
Near the top:

Help on module xml.etree.ElementTree in xml.etree:

NAME
    xml.etree.ElementTree

FILE
    /usr/lib/python2.7/xml/etree/ElementTree.py

MODULE DOCS
    http://docs.python.org/library/xml.etree.ElementTree

DESCRIPTION
    ...
msg212297 - (view) Author: Sean Rodman (sean.rodman) * Date: 2014-02-26 20:16
Here is a working patch for python 2.7. all it does is lowercase the module name, but once I did that and clicked the link it worked correctly.
msg212298 - (view) Author: Sean Rodman (sean.rodman) * Date: 2014-02-26 20:17
Note: It doesn't change the actual module name. Just how it is represented in the link.
msg212309 - (view) Author: Sean Rodman (sean.rodman) * Date: 2014-02-26 21:49
Here is the patch for python 3.2. It implements the same fix that the 2.7 patch does.
msg212317 - (view) Author: Sean Rodman (sean.rodman) * Date: 2014-02-26 22:33
Sorry guys, I missed a place I needed to add the lower() fuction to the module.__name__. Here is a fixed patch for python2.7.
msg212321 - (view) Author: Sean Rodman (sean.rodman) * Date: 2014-02-26 22:56
Here are the python3.2 and python3.3 patches. Please let me know if there is anything I need to change on these.
msg213601 - (view) Author: Sean Rodman (sean.rodman) * Date: 2014-03-14 23:30
Is there anything else I can do for this?
msg214819 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-03-25 12:48
It would be nice to have unit tests for this change.

Did you get an email from the review system?  If not, follow the “review” link on the right of the list of files on this page.
msg216663 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-04-17 06:49
I don’t think these ones could be so easily fixed, but on my computer “pydoc” references:

* library/importlib.machinery.html (ideally should be library/importlib.html#module-importlib.machinery)
* library/tkinter.font.html (not in Python documentation at all that I am aware, except for brief mention in §25.1.1)
msg216706 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-04-17 16:05
Brett, any opposition to moving the doc about importlib submodules to separate files?
msg216709 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-04-17 16:43
No, I have no objections.
msg266970 - (view) Author: Kaushik Nadikuditi (kau8hik) * Date: 2016-06-02 22:05
Applied patch from sean.rodman.
Created a test.
msg267137 - (view) Author: Kaushik Nadikuditi (kau8hik) * Date: 2016-06-03 19:02
Applied patch from sean.rodman.
Created a tests.
Incorporated comments from rdmurray
msg267174 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-06-03 22:10
This patch looks good to me.  Note that the reason for the basedir change is that the test would fail if run from a build directory which hadn't been installed.  An implication of this is that 'doc_loc' is None most of the time when the other tests are run, meaning that this new test is the only one that consistently tests the body of getdocloc.  I think that method could use some more tests, but that is a separate issue.
msg267186 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-06-03 23:29
New changeset 64e7c8531131 by R David Murray in branch '3.5':
#16484: Fix pydoc doc links to modules whose names are mixed case.
https://hg.python.org/cpython/rev/64e7c8531131

New changeset 912b9aad0b45 by R David Murray in branch 'default':
Merge: #16484: Fix pydoc doc links to modules whose names are mixed case.
https://hg.python.org/cpython/rev/912b9aad0b45
msg267188 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-06-03 23:34
Thanks Sean and Kaushik.

The final patch didn't apply to 2.7.  If someone wants to port the patch I'll apply it, but I'm going to close the issue.
msg267242 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-04 08:00
It seems the test is broken on Windows:

http://buildbot.python.org/all/builders/x86%20Windows7%203.5/builds/915/steps/test/logs/stdio
======================================================================
ERROR: test_mixed_case_module_names_are_lower_cased (test.test_pydoc.PydocDocTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.5.bolen-windows7\build\lib\test\test_pydoc.py", line 458, in test_mixed_case_module_names_are_lower_cased
    self.assertIn('xml.etree.elementtree', doc_link)
  File "D:\cygwin\home\db3l\buildarea\3.5.bolen-windows7\build\lib\unittest\case.py", line 1076, in assertIn
    if member not in container:
TypeError: argument of type 'NoneType' is not iterable
msg268090 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-10 05:44
Kaushik or David (or anyone else), does you have an idea of how to fix the Windows tests? It would be good to make the buildbots healthy again, then it is easier to see new failures in Windows.
msg268166 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-06-10 23:10
Those are the only failures on 2 of the 4 buildbots running gui tests.
msg268188 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-11 06:03
My guess is the file.startswith(basedir) in the getdocloc() method is getting tricked by non-canonical paths (e.g. lowercase vs uppercase C: drive?). Would it help to use something like os.path.normcase(), or maybe os.path.commonpath(), pathlib.PurePath.parts or PurePath.relative_to()?

file = inspect.getabsfile(object)
...
basedir = os.path.normcase(basedir)
if (...
     (file.startswith(basedir) and ...

Also, in the new get_pydoc_link() function, is that os.path.join() call meant to be there?

basedir = os.path.join(dirname(dirname(__file__)))
msg268315 - (view) Author: Kaushik Nadikuditi (kau8hik) * Date: 2016-06-12 02:58
- fixed broken test - test_mixed_case_module_names_are_lower_cased
- tested on Windows, Mac, Ubuntu
- thank you Martin Panter for suggestions.
msg268321 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-06-12 04:38
New changeset d84268ed62ac by Martin Panter in branch '3.5':
Issue #16484: Fix pydoc link and test on Windows, by Kaushik Nadikuditi
https://hg.python.org/cpython/rev/d84268ed62ac

New changeset 84d1e5a4b1c5 by Martin Panter in branch 'default':
Issue #16484: Merge pydoc Windows fixes from 3.5
https://hg.python.org/cpython/rev/84d1e5a4b1c5
msg268323 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-06-12 04:49
New changeset c60d2e4f4f96 by Martin Panter in branch '2.7':
Issue #16484: PYTHONDOCS still uses “http:” in Python 2
https://hg.python.org/cpython/rev/c60d2e4f4f96
msg268331 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-12 06:00
Thanks Kaushik. I also changed the https:// detection to also detect http://, to maintain backwards compatibility.

I am porting these changes to 2.7.
msg268333 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-06-12 06:28
New changeset ede10138327e by Martin Panter in branch '2.7':
#16484: Change PYTHONDOCS to "https:", and fix links to use lowercase
https://hg.python.org/cpython/rev/ede10138327e
History
Date User Action Args
2016-06-12 08:00:55martin.pantersetstatus: open -> closed
2016-06-12 06:28:46python-devsetmessages: + msg268333
2016-06-12 06:00:43martin.pantersetmessages: + msg268331
versions: + Python 2.7
2016-06-12 05:11:04berker.peksagsettype: compile error -> behavior
2016-06-12 04:49:23python-devsetmessages: + msg268323
2016-06-12 04:38:21python-devsetmessages: + msg268321
2016-06-12 02:58:01kau8hiksetfiles: + issue16484_test_python3.6.patch
type: compile error
messages: + msg268315

components: + Tests, - Documentation
2016-06-11 06:03:21martin.pantersetmessages: + msg268188
2016-06-10 23:10:03terry.reedysetnosy: + terry.reedy
messages: + msg268166
2016-06-10 05:44:44martin.pantersetpriority: normal -> high

messages: + msg268090
2016-06-04 08:00:31martin.pantersetstatus: closed -> open

messages: + msg267242
versions: + Python 3.6, - Python 2.7, Python 3.4
2016-06-03 23:34:18r.david.murraysetstatus: open -> closed
type: behavior -> (no value)
messages: + msg267188

resolution: fixed
stage: test needed -> resolved
2016-06-03 23:29:58python-devsetnosy: + python-dev
messages: + msg267186
2016-06-03 22:10:35r.david.murraysetnosy: + r.david.murray
messages: + msg267174
2016-06-03 19:02:22kau8hiksetfiles: + Issue16484_rev_python3.6.patch

messages: + msg267137
2016-06-03 19:00:17kau8hiksetfiles: - Issue16484_python3.6.patch
2016-06-02 22:05:34kau8hiksetfiles: + Issue16484_python3.6.patch
nosy: + kau8hik
messages: + msg266970

2014-04-17 19:25:21eric.snowsetnosy: + eric.snow
2014-04-17 16:43:03brett.cannonsetmessages: + msg216709
2014-04-17 16:05:23eric.araujosetnosy: + brett.cannon

messages: + msg216706
versions: + Python 3.5, - Python 3.3
2014-04-17 06:49:56martin.pantersetnosy: + martin.panter
messages: + msg216663
2014-03-25 12:48:05eric.araujosetmessages: + msg214819
2014-03-14 23:30:01sean.rodmansetmessages: + msg213601
2014-03-07 10:39:26eric.araujosetstage: needs patch -> test needed
2014-02-27 14:39:44berker.peksagsetversions: + Python 3.4, - Python 3.2
2014-02-26 22:56:38sean.rodmansetfiles: + issue16484_python3.3.patch
2014-02-26 22:56:15sean.rodmansetfiles: + issue16484_python3.2.patch

messages: + msg212321
2014-02-26 22:33:46sean.rodmansetfiles: + issue16484_python2.7.patch

messages: + msg212317
2014-02-26 22:32:40sean.rodmansetfiles: - issue16484.patch
2014-02-26 22:31:33sean.rodmansetfiles: - issue16484_python3.2.patch
2014-02-26 21:49:38sean.rodmansetfiles: + issue16484_python3.2.patch

messages: + msg212309
2014-02-26 20:17:36sean.rodmansetmessages: + msg212298
2014-02-26 20:16:44sean.rodmansetfiles: + issue16484.patch
keywords: + patch
messages: + msg212297
2014-02-25 21:53:37mgedminsetmessages: + msg212220
2014-02-25 02:20:41sean.rodmansetmessages: + msg212156
2014-02-25 01:17:42sean.rodmansetnosy: + sean.rodman
messages: + msg212155
2014-02-25 00:41:37ned.deilylinkissue20146 superseder
2012-12-06 10:26:24georg.brandlsetmessages: + msg177035
2012-12-06 06:55:27mgedminsetmessages: + msg177029
2012-11-17 17:30:03eric.araujosetassignee: docs@python ->
type: behavior
title: Missing/broken documentation redirect for http://docs.python.org/library/xml.etree.ElementTree -> pydoc generates invalid docs.python.org link for xml.etree.ElementTree and other modules
nosy: + eric.araujo, - docs@python
versions: + Python 3.2, Python 3.3
messages: + msg175784
stage: needs patch
2012-11-16 20:25:47mgedminsetmessages: + msg175713
2012-11-16 19:15:13georg.brandlsetnosy: + georg.brandl
messages: + msg175706
2012-11-16 10:46:29mgedmincreate