classification
Title: Create IDLE help.txt from Doc/library/idle.rst
Type: enhancement Stage:
Components: Documentation, IDLE Versions: Python 3.4, Python 3.3, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Todd.Rovito, asvetlov, docs@python, eric.araujo, georg.brandl, ned.deily, roger.serwy, terry.reedy, zach.ware
Priority: normal Keywords: patch

Created on 2013-01-08 16:45 by zach.ware, last changed 2013-06-15 19:12 by terry.reedy.

Files
File name Uploaded Description Edit
issue16893_makefiles.diff zach.ware, 2013-01-16 18:35 Adjust the Doc makefiles to add idledoc target review
issue16893_makefiles.v2.diff zach.ware, 2013-02-18 04:59 Version 2 review
help.txt ned.deily, 2013-02-18 07:06 rendered help text
issue16893.v3.diff zach.ware, 2013-02-20 19:50 Version 3, with changes to Doc/library/idle.rst review
Messages (24)
msg179362 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-01-08 16:45
Lib/idlelib/help.txt and Doc/library/idle.rst contain almost exactly the same information, only formatted a bit differently.  This issue is to suggest the merger of the two files, by way of creating help.txt from idle.rst as a part of the build or install process.

This could be as simple as merely copying the one to the other, but that of course leaves in place all reST directives that mean nothing to a simple text document.  Thus, idle.rst should be processed to create help.txt.  This could be done with sphinx/docutils, but to avoid having the CPython build process (not just the doc build process) depend on those tools, I think a very simple dedicated script could be created to do the processing without much trouble.

As far as I can tell, this new script would only have to do a few simple steps:

- Remove comments (including index markers, etc.)
- Remove reST roles (replace ":role:`" with "`", possibly remove "`" as well)
- Remove backslash escapes
- Remove (or undouble) double colons
- Replace "#." with "1.", etc.

Thoughts?
msg179615 - (view) Author: Todd Rovito (Todd.Rovito) * Date: 2013-01-11 02:16
I think getting this issue fixed makes sense, it would save time and remove duplication.
msg179708 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2013-01-11 17:32
I like the proposition in general, but files should be synchronized first.
msg179710 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-01-11 17:38
Note that Sphinx' "make text" output already should be suitable. We already update the pydoc topics with that on every release, so we could just as well do the same for the IDLE doc.  No need for another separate script.
msg179846 - (view) Author: Todd Rovito (Todd.Rovito) * Date: 2013-01-13 04:06
Andrew,
    Zachary and I worked on another issue together to sync idle.rst with help.txt here:
http://bugs.python.org/issue5066

Issue 5066 is ready for commit if you have time by the way.  Thanks!
msg179954 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-01-14 16:52
Georg:
> Note that Sphinx' "make text" output already should be suitable. We 
> already update the pydoc topics with that on every release, so we could 
> just as well do the same for the IDLE doc.  No need for another 
> separate script.

I take it this would mean generating help.txt and then checking it in?  Otherwise, users who built their own Python would likely run into issues with IDLE not finding its help file, or would be required to have sphinx available.  That's not quite what I had in mind, but if we already do it for pydoc topics, it sounds fine to me.
msg179959 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2013-01-14 17:31
Regenerating idle.txt and committing it is fine to me.
msg179960 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-01-14 18:00
> I take it this would mean generating help.txt and then checking it in? Otherwise, users who built their own Python would likely run into issues with IDLE not finding its help file, or would be required to have sphinx available.

Yes, it will be checked in.

> That's not quite what I had in mind, but if we already do it for pydoc topics, it sounds fine to me.

I don't think we have to worry about it getting out of date quickly.  Automatically generating the IDLE help at run time from a documentation source file is not posssible anyway, since the doc sources are not available in a standard location for installed Pythons.
msg179961 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-01-14 18:06
> I don't think we have to worry about it getting out of date quickly.  

Fair point :)

> Automatically generating the IDLE help at run time from a documentation 
> source file is not posssible anyway, since the doc sources are not 
> available in a standard location for installed Pythons.

This isn't quite what I meant either; I had intended it to be done at either Python build or install time.  Your method is much simpler, though, I like it.

That said, here's the diff between the (now) current Lib/idlelib/help.txt and the Sphinx ``make text`` output of Doc/library/idle.rst.  I've not found where we've automated the pydoc topics generation, else I'd try to provide a patch to do this as well.
msg179962 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-01-14 18:15
The unified diff is not very helpful; I think somebody has to put the files side by side and merge.

The pydoc topics are built with a custom Sphinx builder implemented in tools/sphinxext/pyspecific.py -- but if we just want the vanilla text builder output it should be enough to add a Doc/Makefile target:

idledoc: BUILDER = text
idledoc: SOURCES = library/idle
idledoc: build
	@echo "Build finished; now copying build/text/library/idle.txt to ../Lib/idlelib/help.txt."
	@cp build/text/library/idle.txt ../Lib/idlelib/help.txt

and I add a step to PEP 101 to run this.
msg179963 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-01-14 18:16
To make it actually work, replace "library/idle" by "library/idle.rst".
msg180096 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-01-16 18:35
Sorry, I wasn't as clear as I meant to be in my last message, I was suddenly rushed and didn't realize I'd left out what I meant to say, which was:

Now that Andrew has committed Todd's fix to issue 5066, idle.rst and help.txt are very well-aligned.  I believe the transition could be made now with no negative impact on the quality or usefulness of help.txt.

Here now is a patch to Doc/Makefile and Doc/make.bat to add an 'idledoc' target.  I haven't been able to test the Makefile change yet, but as it's a direct quote of your suggestion (with correction), Georg, I figure it ought to work :).  I have tested the make.bat change, though, and it works well for me.
msg180104 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-01-16 19:30
I did test the Makefile change, so this should be good to go.  I'll upate PEP 101 once it's in.
msg182115 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-02-14 18:31
Ping!

If there are no objections, would anyone mind committing this?
msg182298 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-02-18 04:59
Actually, I have an objection myself.  In merging this patch with another I'm working on, I noticed that I failed to include the new 'idledoc' target in the Makefile usage message.  The attached patch fixes that oversight.
msg182300 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-02-18 07:04
There are a few problems with the proposed patch (v2).  I commented on those in Rietveld.  But, beyond that, I'm not convinced that the generated help.txt is an improvement over the original help.txt.  While it is formatted more consistently (a good thing), it is significantly longer (547 lines vs 369).  Some info is lost in the translation, for one, the menu div bars (---).  Some of the markup looks odd to me as plain text, for example:

    Upon startup with the ``-s`` option, IDLE will execute the file
    referenced by the environment variables ``IDLESTARTUP`` or
    ``PYTHONSTARTUP``. IDLE first checks for ``IDLESTARTUP``; if
    ``IDLESTARTUP`` is present the file referenced is run.  If 
    ``IDLESTARTUP`` is not present, IDLE checks for ``PYTHONSTARTUP``.

Also, I noticed that the deprecation notice, about running without a subprocess, near the end of the help text seems to be missing from the .rst.
msg182301 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-02-18 07:06
For comparison, here's a copy of the new rendered help.txt.
msg182546 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-02-20 19:50
Here's a new version of the patch, with the fixes that Ned pointed out.

I also tried to address concerns about lost information; menu divisions have been added to Doc/library/idle.rst, along with the blurb about running without a subprocess being deprecated.  Also, every instance of :kbd:`C-x` has been expanded to :kbd:`Control-x`, as that's how help.txt has commands written.  A rather unrelated change that I snuck in while I was editing idle.rst was to move the index markers for Class browser and Path browser to be above those entries rather than below.

The generated help.txt is significantly longer than the old version but I don't think that's all bad.  Most of the extra lines are new whitespace or things that had been a single line being broken up into two.  I personally thought the old help.txt was rather too dense in some places, though I might agree that the generated version is a bit sparse in others.

The paragraph about environment variables does have a rather unfortunate number of backticks, but I don't think it's unreadable.  It's also information that wasn't present in the original help.txt.

Thank you for the review, Ned :)
msg185706 - (view) Author: Todd Rovito (Todd.Rovito) * Date: 2013-04-01 04:25
I added roger.serwy to the nosy list.  Terry Reedy is already on the list.  I think this issue will help maintain the IDLE documentation now and in the future. Right now it has to manually be synced between help.txt and idle.rst. Only Python 3.4 is synced right now based on this issue:
http://bugs.python.org/issue5066
  
Zach has done a great job working the issue so I think it should be committed.
msg185726 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-04-01 10:50
Zach, thanks for addressing most of the comments. The Makefile does now work as intended and more information is retained in the help.txt.

But I'm still troubled by the plaintext rendering, particularly of the inline code markup.  With the `` marks from the reStructuredText directives appearing in the help.txt, the help file as displayed is, IMO, more cluttered and, more importantly, more confusing.  Using `` to delimit displayed plaintext in lieu of more sophisticated text styling is unconventional at best.  I think changing the help file to include such marks would be a step backward.  But that's just one of the more obvious drawbacks of trying to use a plaintext format for help.

Ironically, the Tk text widget is perfectly capable of rendering richer text styles.  But, AFAIK, there hasn't been a use case up to now to support some sort of rich text format (be it rtf or html or rst directly) from a file in IDLE.  My initial thought was to suggest adding some sort of Tk-friendly Sphinx/Docutils builder.  That might be more generally useful but that will probably take a fair amount of work to define and implement.  Then it struck me that having an html-format help file would be able to represent all the existing Sphinx styles and, with the doc changes in the patch, we already have an html file that contains all the help text!  So rather than trying to sync the help file with the IDLE doc in the library, why not just display the library IDLE doc in a browser window?  IDLE already has a menu item and code to launch a web browser for the whole Python documentation set.  It would be easy to adapt that to change the IDLE help menu item to launch a web browser window with the IDLE page and get rid of help.txt.

There are a few issues that would need to be resolved.  The python.org binary installers and most Unix-y distributions can optionally install local copies of the Python doc set to eliminate the need for network access to docs.python.org.  There is platform-specific code in EditorWindow.__init__ to search locally for the doc files, falling back to http://doc.python.org if a local copy is not found.  The simplest approach would be to do the same for IDLE help, however that would have the drawback of there being no help available if the local copy had not been installed and there was no internet access.  Of course, that is the case today for the Python documentation.  If that is not acceptable for the help text, the docs build process could make a copy of a version of the library/idle.html into idlelib or, less desirable, the proposed plain text file could be the fallback.  Also, on Windows, AFAIK, the doc set is usually installed as a chm file which normally opens to the top-level index page in the help viewer.  A question is how to open directly to the IDLE page.  By inspection in the help viewer on Windows 7, it seems that a web browser.open call to a URL that appends '::/library/idle.html' to the existing pythonxxx.chm URL will open a browser window to the right page.  But there may be a better way to do that.

From the user's perspective, ISTM that having the help in the rich format possible with the existing doc html would be a huge advantage over what can be provided with any plain text format.  Comments?
msg185729 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-04-01 11:44
Great idea. Using a browser to display help text has become pretty common. For many games, for instance, this has superseded .pdf manuals, which supercedes paper. Unless there is already a tk extension to display html this seems like a good idea. We could even put a link to a youtube video tutorial if there is one or if someone makes one. (There seems to be some creative types interested in helping with Idle. This would be a good way for someone to contribute.
msg185732 - (view) Author: Todd Rovito (Todd.Rovito) * Date: 2013-04-01 11:59
Ned,
   Using a web browser is a great idea!!!!  I like it because it removes code from IDLE making IDLE even simpler (and better).  Besides it would take us forever to duplicate some of the functionally that exists in today's modern web browser.  

Zach,
   What do you think?

Terry,
   What is this paper technology that you speak of?  A youtube video showing off IDLE would be fantastic.
msg186035 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-04-04 14:42
Defaulting to opening a browser window sounds great to me.  In those cases where there is no network access, though, I think we should keep a fallback help.txt, and I think the Sphinx text rendering is about the simplest and easiest fallback we can get.  Shall we open another issue for using a browser for Help, and should this issue wait on that one, or can this go ahead in the meantime?
msg186040 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-04-04 15:58
I just discovered that on Windows, the 3rd help option "Python Docs - F1" opens the local doc set whatever.chm in the Windows Help (HTML) Viewer. This is the same as the start menu choice. So on Windows, opening the same offline copy to the IDLE page, if possible, would be the thing to do. Perhaps the Windows code for opening the docs to the toc page can be reused to open them to the IDLE page.
History
Date User Action Args
2013-06-15 19:12:42terry.reedysetassignee: docs@python ->
versions: + Python 2.7, Python 3.3
2013-04-04 15:58:37terry.reedysetmessages: + msg186040
2013-04-04 14:42:27zach.waresetmessages: + msg186035
2013-04-01 11:59:04Todd.Rovitosetmessages: + msg185732
2013-04-01 11:44:22terry.reedysetmessages: + msg185729
2013-04-01 10:50:36ned.deilysetmessages: + msg185726
2013-04-01 04:25:40Todd.Rovitosetnosy: + roger.serwy
messages: + msg185706
2013-02-20 19:51:08zach.waresetfiles: - issue16893.diff
2013-02-20 19:50:53zach.waresetfiles: + issue16893.v3.diff

messages: + msg182546
2013-02-18 07:06:37ned.deilysetfiles: + help.txt

messages: + msg182301
2013-02-18 07:04:24ned.deilysetnosy: + ned.deily
messages: + msg182300
2013-02-18 04:59:10zach.waresetfiles: + issue16893_makefiles.v2.diff

messages: + msg182298
2013-02-14 18:31:03zach.waresetmessages: + msg182115
2013-01-16 19:30:22georg.brandlsetmessages: + msg180104
2013-01-16 18:35:48zach.waresetfiles: + issue16893_makefiles.diff

messages: + msg180096
2013-01-14 18:16:07georg.brandlsetmessages: + msg179963
2013-01-14 18:15:22georg.brandlsetmessages: + msg179962
2013-01-14 18:06:19zach.waresetfiles: + issue16893.diff
keywords: + patch
messages: + msg179961
2013-01-14 18:00:36georg.brandlsetmessages: + msg179960
2013-01-14 17:31:04asvetlovsetmessages: + msg179959
2013-01-14 16:52:38zach.waresetmessages: + msg179954
2013-01-13 04:06:17Todd.Rovitosetmessages: + msg179846
2013-01-11 17:38:00georg.brandlsetnosy: + georg.brandl
messages: + msg179710
2013-01-11 17:32:46asvetlovsetnosy: + asvetlov
messages: + msg179708
2013-01-11 16:22:17eric.araujosetnosy: + terry.reedy
2013-01-11 02:16:37Todd.Rovitosetmessages: + msg179615
2013-01-08 16:45:12zach.warecreate