Title: Missing doc links to source code for Python-coded modules.
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.6, Python 3.5
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: Friedrich.Spee.von.Langenfeld, SilentGhost, Yoni Lavi, belopolsky, berker.peksag, docs@python, ezio.melotti, nharold, python-dev, r.david.murray, rhettinger, terry.reedy
Priority: normal Keywords: easy, patch

Created on 2014-10-05 13:05 by Friedrich.Spee.von.Langenfeld, last changed 2016-06-13 04:16 by Yoni Lavi. This issue is now closed.

File name Uploaded Description Edit Friedrich.Spee.von.Langenfeld, 2014-10-05 13:05 Friedrich.Spee.von.Langenfeld, 2014-10-05 15:09
mywork.patch Yoni Lavi, 2016-04-20 02:28 review
mywork2.patch Yoni Lavi, 2016-05-22 00:34 review
mywork3.patch Yoni Lavi, 2016-06-06 00:40 review
Messages (34)
msg228563 - (view) Author: Friedrich Spee von Langenfeld (Friedrich.Spee.von.Langenfeld) Date: 2014-10-05 13:05
As mentioned in Issue22528, in some modules´ documentation, the link to their source code is missing. To check the stdlib, I´ve written the script
It seems that in the following module entries, there aren´t a link to their source code (the last part of output of

Is it possible to add the specific source code link to all of this entries?
msg228564 - (view) Author: SilentGhost (SilentGhost) * (Python triager) Date: 2014-10-05 13:14
Your code produces many false positives, would you care to reduce this list to only those modules that actually do not have a Source code link?
msg228571 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-10-05 14:38
And that have source code.

That said, some modules may have been consciously omitted.
msg228576 - (view) Author: Friedrich Spee von Langenfeld (Friedrich.Spee.von.Langenfeld) Date: 2014-10-05 15:09
I decremented the shown number of modules by 25. Here is the new list:   # false positive

@R. David Murray: I will try to look at the source code for each module, but I have updated the list to make it possible to work on this subject in the meantime.

PS: Is it possible for me to edit my own contributions (comments, files) after uploading?
msg229015 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-10-10 16:43
Messages and patches can be unlinked from an issue (and that action recorded in the history list) but not edited.  Proofread before clicking.

Originally, only a few module docs had a source code link where the source code was thought to be especially useful.  When these links were determined to be useful, more have gradually been added as some developer thought of it or as some user requested, as you did in #22528.  I don't know if any requests have been denied.  I also don't know if there is a written policy anywhere.  To me, it might be easier to link all python sources and be done with the issue.
msg230495 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2014-11-02 15:55
The source links have been added where the code proved to be readable, easy to understand, and self documenting, and have been omitted when the code is complicated and not self documenting.  This has been done under the assumption that reading the code might help understanding the documentation, but there are other situations where users might want to see the code regardless of how readable it is, so I wouldn't object if you decide to add links to all the modules.
msg231033 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2014-11-11 16:56
+1 "to add links to all the [Python] modules"

If the code is not readable, hard to understand or not self-documenting that's the reason to improve the code not to make it harder to find and see.
msg257574 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-01-06 01:19
Actually uploading a patch for this should be easy, right?
msg257583 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-01-06 03:18
Guido, should we take your comment as approving linking every source, as opposed to only some?  This scope issue is the main sticking point for this issue.  Otherwise, the standard patch is 3 easy new lines after the :synopsis: (and any 'New in Version x.y' note for the module -- see argparse).  The following is Benjamin's patch for #22528.

--- a/Doc/library/pdb.rst
+++ b/Doc/library/pdb.rst
@@ -6,6 +6,9 @@
 .. module:: pdb
    :synopsis: The Python debugger for interactive interpreters.

+**Source code:** :source:`Lib/`

 .. index:: single: debugging
Friedrich, were you planning to write a patch, once the list of modules is decided on?  Or should we invite submissions on the core-mentorship list?
msg257586 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-01-06 05:11
It's fine to add a source link to any module for which there is Python source code.  I suppose this adds a slight maintenance burden when a module moves (e.g. when a module is turned into a package, or when the subdirectory structure of the Lib directory changes).

I'm a little confused about the "New in x.y" note -- why is that connected to the source code link?

Of course, the source tells a different story from the docs -- e.g. undocumented implementation details may change, and sometimes the source is hard to understand (on occasion I've been confused myself :-).  But Python is open source, so people can always read the source -- I don't see why we should try to make reading the source harder for people who don't yet have the chops to just read it on their own computer!
msg257602 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-01-06 12:05
If there is a New-in note (which follows Synopsis), then (to be consistent) the Source note should follow it.  If not (the normal situation), then Source follows Synopsis directly.

A point in favor of linking all Python sources from the docs is that default single users installs are much harder to access on Windows than they used to be: /Users/Name/[Appdata - hidden by default]/Roaming?/Python/Pythonxy (I believe) versus the old /Pythonxy.
msg263790 - (view) Author: Yoni Lavi (Yoni Lavi) * Date: 2016-04-20 02:28
I went over all the library rst files and added a link to the source for each of the modules.

Also, I standardized the structure of the headers in all of the files to be in the following order: """
.. module

.. moduleauthor
.. sectionauthor

.. versionadded

**Source code:**

.. testsetup

.. index


This'll be my first Python patch. Please let me know if I chose to go into too large of a scope and should split this, or if there's anything else I can improve. Also, since I'm modifying most of these files anyway, this might be a good opportunity to modify the header structure I mentioned above.

msg265904 - (view) Author: Yoni Lavi (Yoni Lavi) * Date: 2016-05-20 01:01
Friendly ping
msg266000 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-05-21 13:30
Thanks for the patch, Yoni. Like Guido said in msg257586, we only want to add links for pure Python modules. Can you update your patch to remove C modules please?

I'd prefer keep curses docs untouched. Lib/curses/* is basically a wrapper to _cursesmodule.c with some utility functions (sqlite3 is another example of this).

Another candidates for removal are distutils.core, wsgiref, ensurepip, formatted (it's deprecated), test (Lib/test is basically the test suite of CPython) modules.
msg266035 - (view) Author: Yoni Lavi (Yoni Lavi) * Date: 2016-05-22 00:34
Thank you for the review, Berker.
I removed source code links from all the modules you specifically mentioned, and all links to C code, including stat.rst which already had such a link before.

But I have to say that I don't quite understand the rationale here and (as a user) would personally prefer easy access to all source code, both C and Python. It is my opinion that a handy link to the source would help promote the conversion of library users to contributors.
msg266699 - (view) Author: Yoni Lavi (Yoni Lavi) * Date: 2016-05-30 15:10
Ping. Please advise on next step to get this submitted.
msg266835 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-06-01 18:43
[David, could this be a sprint issue?] 

Here is the difference between python and C coded modules.  The x.y docs are for all implementations of x.y.  The python-coded modules are intended for use by all implementations, perhaps with minor modifications.  In any case, the users of all implementation can presumably read Python code.  The C implementations used by CPython are specific to CPython, and not necessarily meaningful to user of Jython, IronPython, PyPy, or any other alternatives.

In any case, Guido has unambiguiously pronounced on linking to Python source files.  Doing that is enough for one issue.  I changed the title to be specific.
msg266837 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-06-01 20:17
You mean someone doing a file review at the sprints?  I can add it to the list.  No guarantees about it getting committed at the sprints, though.

If we are linking to the python version of modules that have both python and C versions, IMO the existence of the C version should be noted as a CPython implementation detail, since otherwise someone might think that the python code was what CPython was actually running.
msg266838 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-06-01 20:19
I meant full review, not file review.
msg266846 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-06-01 23:46
If someone does a full review (looks at each diff in Rietveld to see if it matches the intended format -- see Yoni's 4-19 message, with
:source:`Lib/` (lib2to3 is an exception) following **Source code:**
-- and then downloads patch, applies, builds new html doc, and tests a few of the new links, and find nothing wrong, and says 'ready for commit review) then I may take this, re-check, and commit.
msg266847 - (view) Author: Yoni Lavi (Yoni Lavi) * Date: 2016-06-01 23:59
Thank you for looking at this.

David, concerning modules that have multiple sources, are you referring to modules such as `stat` and `heapq`? In these, it's the Python source that tries to import the C module if it's available, so I'd consider it ok to leave just a link to the Python source.
msg266850 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-06-02 00:30
Yes, those should be OK.  I was thinking of _pyio and _pydecimal, but I don't know if you are linking to those.
msg266857 - (view) Author: Yoni Lavi (Yoni Lavi) * Date: 2016-06-02 01:20
Ah, thanks for the explanation. I think in this case we're ok too.
I linked io.rst to Lib/, which relies on the C version. And in the case of decimal.rst, it was already linked to Lib/ which very explicitly attempts to load the C version and falls back to _pydecimal.
msg267120 - (view) Author: Nathan Harold (nharold) * Date: 2016-06-03 17:52
I'll give this a go, per Terry's instructions in msg266846.

I've noticed up front that, due to other changes near the headers of a couple of files (specifically fcntl.rst and termios.rst), the patch doesn't apply cleanly anymore.  Is it permissible for me to fix that myself and upload a new version, or is it preferable for a patch's original author to handle such things?  In any case, there are plenty of other files for me to look over for now.
msg267149 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-06-03 19:42
For me, the issue is willingness, not permission ;-).  Reviewing a patch includes somehow indicating changes needed to apply it 'today', and a revised patch is the best way.  Please say whether the header changes are in 3.6 only (in which case a separate 3.5 patch is needed) or in both.  'Commit ready' means that you believe a core dev could (and should) download, apply without issue, and commit when satisfies.  I should do so within a week if no one else does.
msg267173 - (view) Author: Nathan Harold (nharold) * Date: 2016-06-03 22:08
I went through all the diffs in Rietveld, checking it against msg228576 as I went.  There were (naturally) more source links added than there are in that list from 2014, and there were also many more files with slight header reorganizations for consistency.  Anything in the diffs I don't mention here looked fine to me.  I also built the docs to test links in general and to see how some specific things here looked.

A number of the new links are to files, and others are to folders rather than individual source files.  I noticed a precedent for the former in the case of the http module; for the latter I'm less certain, but I can see that it makes sense in some cases, e.g. when is empty.  I think these are probably good.

One thing I particularly noticed was that in imp.rst and optparse.rst, the deprecation notice occurred below the source link, whereas the versionadded notices are always above it.  I think it would be more consistent and look more natural to have the deprecation notices above the source links.

This could be taken a step farther by suggesting that the source link always be the last visible thing before the horizontal line.  The two other cases this would affect here are asyncio.rst (the notice that the module is provisional is between the source link and line) and getopt.rst (the note, now moved above the line, is after the source link).  But this is all fairly subjective.

Other details for individual files:

marshal.rst - Here an index block a bit farther down into the text was moved up into the header.  I think these might have been intended to point right where they were, not to the module as a whole; those things occur only in the paragraph (or two, in the case of 'code object') immediately following where the index originally pointed.

unittest.mock-examples.rst - A versionadded notice was completely removed from this file.  In fact, that was the only change to this file.  I think I can see why you did this, since the actual API documentation for the library is in unittest.mock.rst, which has that same version notice.

Smaller details:
array.rst - -------------- was inserted above .. index:
fcntl.rst - Patch appears to apply correctly, but with fuzz.
os.path.rst - It might be good to put a space in "WindowsNT"
termios.rst - Patch fails to apply because some other text near the header has changed recently.
venv.rst - The source code link (to a folder) works, but is missing a trailing slash.

Terry, I got the same results when I tried applying it to 3.5, so I think a single patch will do.
msg267184 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-06-03 22:59
Great report.  It appears that the remaining questions are subjective decisions for a core developer: what is the best link, if any, for a package; should the link also be last thing in the section.
msg267495 - (view) Author: Yoni Lavi (Yoni Lavi) * Date: 2016-06-06 00:40
Thank you very much for the detailed review, Nathan. Attached is a new patch that resolves the merge issues, and some of the other issues you mentioned. 

Concerning the index reference in marshal.rst , ah, yes, I missed that it only related to a single paragraph. I reverted that change.

Concerning versionadded in unittest.mock-examples.rst, you're right, I saw that this was already available elsewhere and removed it for consistency with other similar intro files. I'll revert if you tell me.

I think I fixed all the smaller issues you mentioned, too. I wasn't sure about the fuzz; I didn't see any issue, so I assume it was automatically resolved by `hg update`. 

I don't have any input to give concerning the 'subjective' issues and will be happy to wait for a decision.
msg267540 - (view) Author: Nathan Harold (nharold) * Date: 2016-06-06 16:27
I went over the sections I above called "Other/Smaller details" as addressed in mywork3.patch.  I think they look fine.  The patch applies seamlessly to both 3.5 and 3.6.

For reference, since I noted them when I reviewed mywork2.patch last week, here are the packages at issue in the question about links to packages:

Package docs in which the patch in this issue adds a link to dbm.rst, email.rst, importlib.rst, json.rst, msilib.rst, tkinter.rst, unittest.rst, xml.dom.rst, xml.sax.rst

Package docs in which the patch in this issue adds a link to a directory: 2to3.rst, email.mime.rst, idle.rst, multiprocessing.rst, sqlite3.rst, urllib.rst, venv.rst, xml.rst

Also, I noticed that asyncio.rst provides a precedent for linking to a source directory.  So both ways have been done before.
msg268250 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-06-11 19:06
New changeset 222c1d461aa8 by Terry Jan Reedy in branch '3.5':
Issue #22558: Add remaining doc links to source code for Python-coded modules.

New changeset 2a01d7a488e9 by Terry Jan Reedy in branch 'default':
Merge Issue #22558.
msg268255 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-06-11 19:27
I committed the excellent patch.  It should appear in the upcoming 3.5 and 3.6 releases, and within a day online.  Thanks to both package author and also to reviewers.  I would like to see more patches like this.

I reverted the deletion of version added in mock examples.  It is a separate page, so the reminder is not inappropriate.

Minor nit: Yoni, you 'forgot' to run "python tools/scripts/" on a patched  repository.  Not too surprisingly, the editing left trailing whitespace that needed to be removed.  See the devguide for more.

A possible follow-on.  Some of the packages that document contained modules do so on the main page rather than in separate pages.  If the source link is to the package __init__, rather than to the package directory, then it would be appropriate to add links to the module sections.  The particular example I am thinking of is importlib, where we linked to .__init__.  There may be another.  Multiprocessing is not, because we linked to the directory.
msg268299 - (view) Author: Yoni Lavi (Yoni Lavi) * Date: 2016-06-12 00:12
Thank you Terry, Nathan and David for your work on this.
Sorry about the whitespace issue, I indeed forgot to run patchcheck when I prepared the 3rd patch.

Tery, would you like me to prepare a patch for the follow-on you suggested? If so, given that this one is now marked as fixed, we should open a new bug, right?
msg268301 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-06-12 00:23
Yes, new issue, make me nosy, assign to me if allowed, upload patch. Start with "This is a follow to #22558."
msg268418 - (view) Author: Yoni Lavi (Yoni Lavi) * Date: 2016-06-13 04:16
Created issue 27304 and uploaded a patch.
Date User Action Args
2016-06-13 04:16:21Yoni Lavisetmessages: + msg268418
2016-06-12 00:23:49terry.reedysetmessages: + msg268301
2016-06-12 00:12:22Yoni Lavisetmessages: + msg268299
2016-06-11 19:27:31terry.reedysetstatus: open -> closed
resolution: fixed
messages: + msg268255

stage: commit review -> resolved
2016-06-11 19:06:32python-devsetnosy: + python-dev
messages: + msg268250
2016-06-11 16:46:18terry.reedysetassignee: docs@python -> terry.reedy
stage: patch review -> commit review
2016-06-06 16:27:31nharoldsetmessages: + msg267540
2016-06-06 00:41:07Yoni Lavisetfiles: + mywork3.patch

messages: + msg267495
2016-06-03 22:59:37terry.reedysetmessages: + msg267184
2016-06-03 22:08:42nharoldsetmessages: + msg267173
2016-06-03 19:42:45terry.reedysetmessages: + msg267149
2016-06-03 17:52:40nharoldsetmessages: + msg267120
2016-06-02 17:05:51nharoldsetnosy: + nharold
2016-06-02 01:20:29Yoni Lavisetmessages: + msg266857
2016-06-02 00:30:09r.david.murraysetmessages: + msg266850
2016-06-01 23:59:06Yoni Lavisetmessages: + msg266847
2016-06-01 23:46:25terry.reedysetmessages: + msg266846
2016-06-01 20:19:48r.david.murraysetmessages: + msg266838
2016-06-01 20:17:17r.david.murraysetmessages: + msg266837
2016-06-01 18:43:46terry.reedysetmessages: + msg266835
title: Missing doc links to source code -> Missing doc links to source code for Python-coded modules.
2016-05-31 01:21:29gvanrossumsetnosy: - gvanrossum
2016-05-30 15:10:11Yoni Lavisetmessages: + msg266699
2016-05-22 00:34:28Yoni Lavisetfiles: + mywork2.patch

messages: + msg266035
2016-05-21 13:30:41berker.peksagsetnosy: + berker.peksag

messages: + msg266000
stage: needs patch -> patch review
2016-05-20 01:01:50Yoni Lavisetmessages: + msg265904
2016-04-20 02:28:28Yoni Lavisetfiles: + mywork.patch
keywords: + patch
messages: + msg263790
2016-04-19 15:33:42Yoni Lavisetnosy: + Yoni Lavi
2016-03-27 23:49:08martin.pantersettitle: Missing hint to source code - complete -> Missing doc links to source code
2016-01-06 12:05:35terry.reedysetversions: + Python 3.6, - Python 3.4
2016-01-06 12:05:24terry.reedysetmessages: + msg257602
2016-01-06 05:11:58gvanrossumsetmessages: + msg257586
2016-01-06 03:18:48terry.reedysetmessages: + msg257583
2016-01-06 01:19:19gvanrossumsetkeywords: + easy
nosy: + gvanrossum
messages: + msg257574

2014-11-11 16:56:41belopolskysetnosy: + belopolsky
messages: + msg231033
2014-11-02 15:56:00ezio.melottisetversions: + Python 3.4, Python 3.5
nosy: + ezio.melotti

messages: + msg230495

type: enhancement
stage: needs patch
2014-10-10 16:43:30terry.reedysetnosy: + terry.reedy
messages: + msg229015
2014-10-05 15:09:03Friedrich.Spee.von.Langenfeldsetfiles: +

messages: + msg228576
2014-10-05 14:38:12r.david.murraysetnosy: + rhettinger, r.david.murray
messages: + msg228571
2014-10-05 13:14:20SilentGhostsetnosy: + SilentGhost
messages: + msg228564
2014-10-05 13:05:37Friedrich.Spee.von.Langenfeldcreate