Title: Generate Idle help from Doc/library/idle.rst
Type: enhancement Stage: needs patch
Components: Documentation, IDLE Versions: Python 3.5, Python 3.4, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: Todd.Rovito, asvetlov, docs@python, eric.araujo, georg.brandl, ned.deily, python-dev, roger.serwy, terry.reedy, zach.ware
Priority: normal Keywords: patch

Created on 2013-01-08 16:45 by zach.ware, last changed 2014-12-17 04:18 by terry.reedy.

File name Uploaded Description Edit
issue16893_makefiles.diff zach.ware, 2013-01-16 18:35 Adjust the Doc makefiles to add idledoc target review
issue16893_makefiles.v2.diff zach.ware, 2013-02-18 04:59 Version 2 review
help.txt ned.deily, 2013-02-18 07:06 rendered help text
issue16893.v3.diff zach.ware, 2013-02-20 19:50 Version 3, with changes to Doc/library/idle.rst review
issue16893-v4.diff zach.ware, 2014-10-20 04:20 review
Messages (29)
msg179362 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-01-08 16:45
Lib/idlelib/help.txt and Doc/library/idle.rst contain almost exactly the same information, only formatted a bit differently.  This issue is to suggest the merger of the two files, by way of creating help.txt from idle.rst as a part of the build or install process.

This could be as simple as merely copying the one to the other, but that of course leaves in place all reST directives that mean nothing to a simple text document.  Thus, idle.rst should be processed to create help.txt.  This could be done with sphinx/docutils, but to avoid having the CPython build process (not just the doc build process) depend on those tools, I think a very simple dedicated script could be created to do the processing without much trouble.

As far as I can tell, this new script would only have to do a few simple steps:

- Remove comments (including index markers, etc.)
- Remove reST roles (replace ":role:`" with "`", possibly remove "`" as well)
- Remove backslash escapes
- Remove (or undouble) double colons
- Replace "#." with "1.", etc.

msg179615 - (view) Author: Todd Rovito (Todd.Rovito) * Date: 2013-01-11 02:16
I think getting this issue fixed makes sense, it would save time and remove duplication.
msg179708 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2013-01-11 17:32
I like the proposition in general, but files should be synchronized first.
msg179710 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-01-11 17:38
Note that Sphinx' "make text" output already should be suitable. We already update the pydoc topics with that on every release, so we could just as well do the same for the IDLE doc.  No need for another separate script.
msg179846 - (view) Author: Todd Rovito (Todd.Rovito) * Date: 2013-01-13 04:06
    Zachary and I worked on another issue together to sync idle.rst with help.txt here:

Issue 5066 is ready for commit if you have time by the way.  Thanks!
msg179954 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-01-14 16:52
> Note that Sphinx' "make text" output already should be suitable. We 
> already update the pydoc topics with that on every release, so we could 
> just as well do the same for the IDLE doc.  No need for another 
> separate script.

I take it this would mean generating help.txt and then checking it in?  Otherwise, users who built their own Python would likely run into issues with IDLE not finding its help file, or would be required to have sphinx available.  That's not quite what I had in mind, but if we already do it for pydoc topics, it sounds fine to me.
msg179959 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2013-01-14 17:31
Regenerating idle.txt and committing it is fine to me.
msg179960 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-01-14 18:00
> I take it this would mean generating help.txt and then checking it in? Otherwise, users who built their own Python would likely run into issues with IDLE not finding its help file, or would be required to have sphinx available.

Yes, it will be checked in.

> That's not quite what I had in mind, but if we already do it for pydoc topics, it sounds fine to me.

I don't think we have to worry about it getting out of date quickly.  Automatically generating the IDLE help at run time from a documentation source file is not posssible anyway, since the doc sources are not available in a standard location for installed Pythons.
msg179961 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-01-14 18:06
> I don't think we have to worry about it getting out of date quickly.  

Fair point :)

> Automatically generating the IDLE help at run time from a documentation 
> source file is not posssible anyway, since the doc sources are not 
> available in a standard location for installed Pythons.

This isn't quite what I meant either; I had intended it to be done at either Python build or install time.  Your method is much simpler, though, I like it.

That said, here's the diff between the (now) current Lib/idlelib/help.txt and the Sphinx ``make text`` output of Doc/library/idle.rst.  I've not found where we've automated the pydoc topics generation, else I'd try to provide a patch to do this as well.
msg179962 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-01-14 18:15
The unified diff is not very helpful; I think somebody has to put the files side by side and merge.

The pydoc topics are built with a custom Sphinx builder implemented in tools/sphinxext/ -- but if we just want the vanilla text builder output it should be enough to add a Doc/Makefile target:

idledoc: BUILDER = text
idledoc: SOURCES = library/idle
idledoc: build
	@echo "Build finished; now copying build/text/library/idle.txt to ../Lib/idlelib/help.txt."
	@cp build/text/library/idle.txt ../Lib/idlelib/help.txt

and I add a step to PEP 101 to run this.
msg179963 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-01-14 18:16
To make it actually work, replace "library/idle" by "library/idle.rst".
msg180096 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-01-16 18:35
Sorry, I wasn't as clear as I meant to be in my last message, I was suddenly rushed and didn't realize I'd left out what I meant to say, which was:

Now that Andrew has committed Todd's fix to issue 5066, idle.rst and help.txt are very well-aligned.  I believe the transition could be made now with no negative impact on the quality or usefulness of help.txt.

Here now is a patch to Doc/Makefile and Doc/make.bat to add an 'idledoc' target.  I haven't been able to test the Makefile change yet, but as it's a direct quote of your suggestion (with correction), Georg, I figure it ought to work :).  I have tested the make.bat change, though, and it works well for me.
msg180104 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-01-16 19:30
I did test the Makefile change, so this should be good to go.  I'll upate PEP 101 once it's in.
msg182115 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-02-14 18:31

If there are no objections, would anyone mind committing this?
msg182298 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-02-18 04:59
Actually, I have an objection myself.  In merging this patch with another I'm working on, I noticed that I failed to include the new 'idledoc' target in the Makefile usage message.  The attached patch fixes that oversight.
msg182300 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-02-18 07:04
There are a few problems with the proposed patch (v2).  I commented on those in Rietveld.  But, beyond that, I'm not convinced that the generated help.txt is an improvement over the original help.txt.  While it is formatted more consistently (a good thing), it is significantly longer (547 lines vs 369).  Some info is lost in the translation, for one, the menu div bars (---).  Some of the markup looks odd to me as plain text, for example:

    Upon startup with the ``-s`` option, IDLE will execute the file
    referenced by the environment variables ``IDLESTARTUP`` or
    ``PYTHONSTARTUP``. IDLE first checks for ``IDLESTARTUP``; if
    ``IDLESTARTUP`` is present the file referenced is run.  If 
    ``IDLESTARTUP`` is not present, IDLE checks for ``PYTHONSTARTUP``.

Also, I noticed that the deprecation notice, about running without a subprocess, near the end of the help text seems to be missing from the .rst.
msg182301 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-02-18 07:06
For comparison, here's a copy of the new rendered help.txt.
msg182546 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-02-20 19:50
Here's a new version of the patch, with the fixes that Ned pointed out.

I also tried to address concerns about lost information; menu divisions have been added to Doc/library/idle.rst, along with the blurb about running without a subprocess being deprecated.  Also, every instance of :kbd:`C-x` has been expanded to :kbd:`Control-x`, as that's how help.txt has commands written.  A rather unrelated change that I snuck in while I was editing idle.rst was to move the index markers for Class browser and Path browser to be above those entries rather than below.

The generated help.txt is significantly longer than the old version but I don't think that's all bad.  Most of the extra lines are new whitespace or things that had been a single line being broken up into two.  I personally thought the old help.txt was rather too dense in some places, though I might agree that the generated version is a bit sparse in others.

The paragraph about environment variables does have a rather unfortunate number of backticks, but I don't think it's unreadable.  It's also information that wasn't present in the original help.txt.

Thank you for the review, Ned :)
msg185706 - (view) Author: Todd Rovito (Todd.Rovito) * Date: 2013-04-01 04:25
I added roger.serwy to the nosy list.  Terry Reedy is already on the list.  I think this issue will help maintain the IDLE documentation now and in the future. Right now it has to manually be synced between help.txt and idle.rst. Only Python 3.4 is synced right now based on this issue:
Zach has done a great job working the issue so I think it should be committed.
msg185726 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-04-01 10:50
Zach, thanks for addressing most of the comments. The Makefile does now work as intended and more information is retained in the help.txt.

But I'm still troubled by the plaintext rendering, particularly of the inline code markup.  With the `` marks from the reStructuredText directives appearing in the help.txt, the help file as displayed is, IMO, more cluttered and, more importantly, more confusing.  Using `` to delimit displayed plaintext in lieu of more sophisticated text styling is unconventional at best.  I think changing the help file to include such marks would be a step backward.  But that's just one of the more obvious drawbacks of trying to use a plaintext format for help.

Ironically, the Tk text widget is perfectly capable of rendering richer text styles.  But, AFAIK, there hasn't been a use case up to now to support some sort of rich text format (be it rtf or html or rst directly) from a file in IDLE.  My initial thought was to suggest adding some sort of Tk-friendly Sphinx/Docutils builder.  That might be more generally useful but that will probably take a fair amount of work to define and implement.  Then it struck me that having an html-format help file would be able to represent all the existing Sphinx styles and, with the doc changes in the patch, we already have an html file that contains all the help text!  So rather than trying to sync the help file with the IDLE doc in the library, why not just display the library IDLE doc in a browser window?  IDLE already has a menu item and code to launch a web browser for the whole Python documentation set.  It would be easy to adapt that to change the IDLE help menu item to launch a web browser window with the IDLE page and get rid of help.txt.

There are a few issues that would need to be resolved.  The binary installers and most Unix-y distributions can optionally install local copies of the Python doc set to eliminate the need for network access to  There is platform-specific code in EditorWindow.__init__ to search locally for the doc files, falling back to if a local copy is not found.  The simplest approach would be to do the same for IDLE help, however that would have the drawback of there being no help available if the local copy had not been installed and there was no internet access.  Of course, that is the case today for the Python documentation.  If that is not acceptable for the help text, the docs build process could make a copy of a version of the library/idle.html into idlelib or, less desirable, the proposed plain text file could be the fallback.  Also, on Windows, AFAIK, the doc set is usually installed as a chm file which normally opens to the top-level index page in the help viewer.  A question is how to open directly to the IDLE page.  By inspection in the help viewer on Windows 7, it seems that a web call to a URL that appends '::/library/idle.html' to the existing pythonxxx.chm URL will open a browser window to the right page.  But there may be a better way to do that.

From the user's perspective, ISTM that having the help in the rich format possible with the existing doc html would be a huge advantage over what can be provided with any plain text format.  Comments?
msg185729 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-04-01 11:44
Great idea. Using a browser to display help text has become pretty common. For many games, for instance, this has superseded .pdf manuals, which supercedes paper. Unless there is already a tk extension to display html this seems like a good idea. We could even put a link to a youtube video tutorial if there is one or if someone makes one. (There seems to be some creative types interested in helping with Idle. This would be a good way for someone to contribute.
msg185732 - (view) Author: Todd Rovito (Todd.Rovito) * Date: 2013-04-01 11:59
   Using a web browser is a great idea!!!!  I like it because it removes code from IDLE making IDLE even simpler (and better).  Besides it would take us forever to duplicate some of the functionally that exists in today's modern web browser.  

   What do you think?

   What is this paper technology that you speak of?  A youtube video showing off IDLE would be fantastic.
msg186035 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-04-04 14:42
Defaulting to opening a browser window sounds great to me.  In those cases where there is no network access, though, I think we should keep a fallback help.txt, and I think the Sphinx text rendering is about the simplest and easiest fallback we can get.  Shall we open another issue for using a browser for Help, and should this issue wait on that one, or can this go ahead in the meantime?
msg186040 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-04-04 15:58
I just discovered that on Windows, the 3rd help option "Python Docs - F1" opens the local doc set whatever.chm in the Windows Help (HTML) Viewer. This is the same as the start menu choice. So on Windows, opening the same offline copy to the IDLE page, if possible, would be the thing to do. Perhaps the Windows code for opening the docs to the toc page can be reused to open them to the IDLE page.
msg229706 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-10-20 01:14
I have avoided editing the Idle docs because of this issue, but idle.rst needs some corrections and updates now, and will soon need some additions.  I want to only edit idle.rst. So I want to resolve this issue soon.

As I mentioned before, I think the best idea is Ned's: copy Doc/build/html/library/idle.html to idlelib (as help.html for now) and change the 'IDLE Help' code to display it in a browser.  This will be easy and will end the duplication and DRY violation that is the goal of this issue.

Zach, can you produce a makefiles.v4 patch (without .rst changes) that patches the makefiles to add a target do the copying?  The same patch (or another) should add idlelib/help.html to .hgignore so idle developers and testers can 'make' the new target without committing it.  Georg can then update PEP 101 with the new release step.

This change will provide a nicely formatted, always-available local copy that should match the installed micro-version.  Because of PEP 434, later online versions of the doc may contain changes that do not apply to the installed version.

After this is done, and anything in help.txt but not in idle.rst is moved, help.txt will be deleted.

issue16893.v3.diff mixes together changes for this issue and a separate issue of editing idle.rst.  I will separately apply some of your mising changes, but I don't think I like adding the new separators and even more whitespace.  Also, I assume that 'Control' in .txt was changed to 'C' in .rst because OSX uses 'Command' instead of 'Control' and 'C' stands for either.  That is aside from the issue of abbreviating something used repeatedly.  Anyway, I think this is a discussion for another tracker issue.
msg229712 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-10-20 04:20
Sure; here's a patch.  I suspect we may want to try giving Sphinx some different options to clean up the output a bit, but I'm not experienced enough with Sphinx to have any specific suggestions.
msg232105 - (view) Author: Roundup Robot (python-dev) Date: 2014-12-04 05:56
New changeset 6db65ff985b6 by Terry Jan Reedy in branch '3.4':
Issue #16893: For Idle doc, move index entries, copy no-subprocess section

New changeset feec1ea55127 by Terry Jan Reedy in branch '2.7':
Issue #16893: Update 2.7 version of Idle doc to match 3.4 doc as of the just
msg232233 - (view) Author: Roundup Robot (python-dev) Date: 2014-12-06 01:26
New changeset 283c364c372a by Terry Jan Reedy in branch '2.7':
Issue #16893: Update Idle doc chapter to match current Idle and add new

New changeset 2fc341c6e314 by Terry Jan Reedy in branch '3.4':
Issue #16893: Update Idle doc chapter to match current Idle and add new
msg232798 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-12-17 04:18
The goal of this issue is that Idle -> Help -> IDLE Help display a good-looking, accurate help page based on Doc/library/idle.rst.  The current Lib/idlelib/help.txt is neither good-looking nor accurate and must somehow be replaced, even if by temporary means, before another release (2.7.9 was missed, 3.4.3 is coming in January).

Ned's gave some comments on the alternatives in msg185726.  I am extending those (and in some places implicitly referring back to them) based on further thinking and investigation.  A summary of what I currently think might be best:

if Windows:
  open .chm to Idle page
    open offline docs to Idle page
  except failure:
    open generated .txt without `s.

The easiest alternative would be to display the online page in a browser.  It looks nice, has a sidebar index, and access to the rest of the docs.  An obvious problem is that this requires a browser and online access, either of which might be lacking in a classroom or when one is away from home base.  More subtle is that Idle features can be changed or added at any time.  The online docs, generated daily from the repository, should reflect the repository version of Idle.  Released versions need a doc frozen at the time of release.

Still easy, perhaps, would be building Doc/build/html/library/idle.html and copying it to idlelib/help.html as part of the release process*, and changing the code to display that instead 

However as Zack hinted ("clean up the output a bit"), this file is not a standalone file.  It has many links to files in '..', such as "Index" and "Copyright"; these would have to be removed completely as the text is useless without the link.  For cross-reference link, such as 'Tkinter' or 'sys.path', the text should be left but the linkage removed.  In both cases, clicking the link gives a File not Found page. The broken links to javascript formatting files in '../.source' appears to be ignored and the formatting simply not done.  So the page index is at the bottom of the page instead of the sidebar.

* The beginners who most need the help doc are running installed Python, not respository builds, so the latter concerns me less.

But why make an off-line page when there already is one in off-line docs?  I presume that the linux and osx doc urls just need 'index.html' replaced with '/library/idle.html'.  The 'standard location' for linux is x.y.z specific.  The python2 rpm location does not seem to be.  The osx location I will not guess about.

For Windows, startfile('x.chm') does the same as double-clicking x.chm in Explorer, which is to open the file with the Help Control, with a nice sidebar for contents, index, search, and favorites. Adding "::/library/idle.html" does not work because it makes the path invalid.  But with an special prefix

>>> os.startfile("mk:@MSITStore:c:/programs/Python34/Doc/python342.chm::/library/idle.html")

opens the page in InternetExplorer.  However, there is no sidebar and the page has to be clicked on the taskbar to make it visible.  In 1 of 4 tries, the above failed.

An answer to this StackOverflow question

prompted me to try

>>> subprocess.Popen("hh ms-its:c:/programs/Python34/Doc/python342.chm::/library/idle.html")

This opens the chm doc with the help control (and sidebar) to the Idle page, just as I wanted.  Since Doc/pythonxyz.chm is part of the installation, I think this is the solution for Windows.  The one glitch is that testing that the exact code in the repository works needs to be done with a pre-final install.  That is already true of testing that "Help -> Python docs F1" opens the .chm to the index page.  These both should be part of a new installed idle test file.

After generating html.txt from idle.rst, the backticks could be filtered out.  It would be close to what we have now.  Is there any need to keep it as a backup for non-Windows systems?  If not, I would be inclined to delete it.

For repository builds, users could go to the online Idle page.  I might even look for ../../Misc and if present, open it directly.  That page would be at most a day out of date.
Date User Action Args
2014-12-17 04:18:32terry.reedysetassignee: terry.reedy
messages: + msg232798
2014-12-06 01:26:50python-devsetmessages: + msg232233
2014-12-04 05:56:10python-devsetnosy: + python-dev
messages: + msg232105
2014-10-20 04:20:56zach.waresetfiles: + issue16893-v4.diff

messages: + msg229712
2014-10-20 01:14:41terry.reedysettitle: Create IDLE help.txt from Doc/library/idle.rst -> Generate Idle help from Doc/library/idle.rst
messages: + msg229706
stage: needs patch
2014-08-09 23:51:09BreamoreBoysetversions: + Python 3.5, - Python 3.3
2013-06-15 19:12:42terry.reedysetassignee: docs@python -> (no value)
versions: + Python 2.7, Python 3.3
2013-04-04 15:58:37terry.reedysetmessages: + msg186040
2013-04-04 14:42:27zach.waresetmessages: + msg186035
2013-04-01 11:59:04Todd.Rovitosetmessages: + msg185732
2013-04-01 11:44:22terry.reedysetmessages: + msg185729
2013-04-01 10:50:36ned.deilysetmessages: + msg185726
2013-04-01 04:25:40Todd.Rovitosetnosy: + roger.serwy
messages: + msg185706
2013-02-20 19:51:08zach.waresetfiles: - issue16893.diff
2013-02-20 19:50:53zach.waresetfiles: + issue16893.v3.diff

messages: + msg182546
2013-02-18 07:06:37ned.deilysetfiles: + help.txt

messages: + msg182301
2013-02-18 07:04:24ned.deilysetnosy: + ned.deily
messages: + msg182300
2013-02-18 04:59:10zach.waresetfiles: + issue16893_makefiles.v2.diff

messages: + msg182298
2013-02-14 18:31:03zach.waresetmessages: + msg182115
2013-01-16 19:30:22georg.brandlsetmessages: + msg180104
2013-01-16 18:35:48zach.waresetfiles: + issue16893_makefiles.diff

messages: + msg180096
2013-01-14 18:16:07georg.brandlsetmessages: + msg179963
2013-01-14 18:15:22georg.brandlsetmessages: + msg179962
2013-01-14 18:06:19zach.waresetfiles: + issue16893.diff
keywords: + patch
messages: + msg179961
2013-01-14 18:00:36georg.brandlsetmessages: + msg179960
2013-01-14 17:31:04asvetlovsetmessages: + msg179959
2013-01-14 16:52:38zach.waresetmessages: + msg179954
2013-01-13 04:06:17Todd.Rovitosetmessages: + msg179846
2013-01-11 17:38:00georg.brandlsetnosy: + georg.brandl
messages: + msg179710
2013-01-11 17:32:46asvetlovsetnosy: + asvetlov
messages: + msg179708
2013-01-11 16:22:17eric.araujosetnosy: + terry.reedy
2013-01-11 02:16:37Todd.Rovitosetmessages: + msg179615
2013-01-08 16:45:12zach.warecreate