Lib/idlelib/help.txt and Doc/library/idle.rst contain almost exactly the same information, only formatted a bit differently.  This issue is to suggest the merger of the two files, by way of creating help.txt from idle.rst as a part of the build or install process.

This could be as simple as merely copying the one to the other, but that of course leaves in place all reST directives that mean nothing to a simple text document.  Thus, idle.rst should be processed to create help.txt.  This could be done with sphinx/docutils, but to avoid having the CPython build process (not just the doc build process) depend on those tools, I think a very simple dedicated script could be created to do the processing without much trouble.

As far as I can tell, this new script would only have to do a few simple steps:

- Remove comments (including index markers, etc.)
- Remove reST roles (replace ":role:`" with "`", possibly remove "`" as well)
- Remove backslash escapes
- Remove (or undouble) double colons
- Replace "#." with "1.", etc.

Thoughts?

I think getting this issue fixed makes sense, it would save time and remove duplication.

I like the proposition in general, but files should be synchronized first.

Note that Sphinx' "make text" output already should be suitable. We already update the pydoc topics with that on every release, so we could just as well do the same for the IDLE doc.  No need for another separate script.

Andrew,
    Zachary and I worked on another issue together to sync idle.rst with help.txt here:
http://bugs.python.org/issue5066

Issue 5066 is ready for commit if you have time by the way.  Thanks!

Georg:
> Note that Sphinx' "make text" output already should be suitable. We 
> already update the pydoc topics with that on every release, so we could 
> just as well do the same for the IDLE doc.  No need for another 
> separate script.

I take it this would mean generating help.txt and then checking it in?  Otherwise, users who built their own Python would likely run into issues with IDLE not finding its help file, or would be required to have sphinx available.  That's not quite what I had in mind, but if we already do it for pydoc topics, it sounds fine to me.

Regenerating idle.txt and committing it is fine to me.

> I take it this would mean generating help.txt and then checking it in? Otherwise, users who built their own Python would likely run into issues with IDLE not finding its help file, or would be required to have sphinx available.

Yes, it will be checked in.

> That's not quite what I had in mind, but if we already do it for pydoc topics, it sounds fine to me.

I don't think we have to worry about it getting out of date quickly.  Automatically generating the IDLE help at run time from a documentation source file is not posssible anyway, since the doc sources are not available in a standard location for installed Pythons.

> I don't think we have to worry about it getting out of date quickly.  

Fair point :)

> Automatically generating the IDLE help at run time from a documentation 
> source file is not posssible anyway, since the doc sources are not 
> available in a standard location for installed Pythons.

This isn't quite what I meant either; I had intended it to be done at either Python build or install time.  Your method is much simpler, though, I like it.

That said, here's the diff between the (now) current Lib/idlelib/help.txt and the Sphinx ``make text`` output of Doc/library/idle.rst.  I've not found where we've automated the pydoc topics generation, else I'd try to provide a patch to do this as well.

The unified diff is not very helpful; I think somebody has to put the files side by side and merge.

The pydoc topics are built with a custom Sphinx builder implemented in tools/sphinxext/pyspecific.py -- but if we just want the vanilla text builder output it should be enough to add a Doc/Makefile target:

idledoc: BUILDER = text
idledoc: SOURCES = library/idle
idledoc: build
	@echo "Build finished; now copying build/text/library/idle.txt to ../Lib/idlelib/help.txt."
	@cp build/text/library/idle.txt ../Lib/idlelib/help.txt

and I add a step to PEP 101 to run this.

To make it actually work, replace "library/idle" by "library/idle.rst".

Sorry, I wasn't as clear as I meant to be in my last message, I was suddenly rushed and didn't realize I'd left out what I meant to say, which was:

Now that Andrew has committed Todd's fix to issue 5066, idle.rst and help.txt are very well-aligned.  I believe the transition could be made now with no negative impact on the quality or usefulness of help.txt.

Here now is a patch to Doc/Makefile and Doc/make.bat to add an 'idledoc' target.  I haven't been able to test the Makefile change yet, but as it's a direct quote of your suggestion (with correction), Georg, I figure it ought to work :).  I have tested the make.bat change, though, and it works well for me.

I did test the Makefile change, so this should be good to go.  I'll upate PEP 101 once it's in.

Ping!

If there are no objections, would anyone mind committing this?

Actually, I have an objection myself.  In merging this patch with another I'm working on, I noticed that I failed to include the new 'idledoc' target in the Makefile usage message.  The attached patch fixes that oversight.

There are a few problems with the proposed patch (v2).  I commented on those in Rietveld.  But, beyond that, I'm not convinced that the generated help.txt is an improvement over the original help.txt.  While it is formatted more consistently (a good thing), it is significantly longer (547 lines vs 369).  Some info is lost in the translation, for one, the menu div bars (---).  Some of the markup looks odd to me as plain text, for example:

    Upon startup with the ``-s`` option, IDLE will execute the file
    referenced by the environment variables ``IDLESTARTUP`` or
    ``PYTHONSTARTUP``. IDLE first checks for ``IDLESTARTUP``; if
    ``IDLESTARTUP`` is present the file referenced is run.  If 
    ``IDLESTARTUP`` is not present, IDLE checks for ``PYTHONSTARTUP``.

Also, I noticed that the deprecation notice, about running without a subprocess, near the end of the help text seems to be missing from the .rst.

For comparison, here's a copy of the new rendered help.txt.

Here's a new version of the patch, with the fixes that Ned pointed out.

I also tried to address concerns about lost information; menu divisions have been added to Doc/library/idle.rst, along with the blurb about running without a subprocess being deprecated.  Also, every instance of :kbd:`C-x` has been expanded to :kbd:`Control-x`, as that's how help.txt has commands written.  A rather unrelated change that I snuck in while I was editing idle.rst was to move the index markers for Class browser and Path browser to be above those entries rather than below.

The generated help.txt is significantly longer than the old version but I don't think that's all bad.  Most of the extra lines are new whitespace or things that had been a single line being broken up into two.  I personally thought the old help.txt was rather too dense in some places, though I might agree that the generated version is a bit sparse in others.

The paragraph about environment variables does have a rather unfortunate number of backticks, but I don't think it's unreadable.  It's also information that wasn't present in the original help.txt.

Thank you for the review, Ned :)

I added roger.serwy to the nosy list.  Terry Reedy is already on the list.  I think this issue will help maintain the IDLE documentation now and in the future. Right now it has to manually be synced between help.txt and idle.rst. Only Python 3.4 is synced right now based on this issue:
http://bugs.python.org/issue5066
  
Zach has done a great job working the issue so I think it should be committed.

Zach, thanks for addressing most of the comments. The Makefile does now work as intended and more information is retained in the help.txt.

But I'm still troubled by the plaintext rendering, particularly of the inline code markup.  With the `` marks from the reStructuredText directives appearing in the help.txt, the help file as displayed is, IMO, more cluttered and, more importantly, more confusing.  Using `` to delimit displayed plaintext in lieu of more sophisticated text styling is unconventional at best.  I think changing the help file to include such marks would be a step backward.  But that's just one of the more obvious drawbacks of trying to use a plaintext format for help.

Ironically, the Tk text widget is perfectly capable of rendering richer text styles.  But, AFAIK, there hasn't been a use case up to now to support some sort of rich text format (be it rtf or html or rst directly) from a file in IDLE.  My initial thought was to suggest adding some sort of Tk-friendly Sphinx/Docutils builder.  That might be more generally useful but that will probably take a fair amount of work to define and implement.  Then it struck me that having an html-format help file would be able to represent all the existing Sphinx styles and, with the doc changes in the patch, we already have an html file that contains all the help text!  So rather than trying to sync the help file with the IDLE doc in the library, why not just display the library IDLE doc in a browser window?  IDLE already has a menu item and code to launch a web browser for the whole Python documentation set.  It would be easy to adapt that to change the IDLE help menu item to launch a web browser window with the IDLE page and get rid of help.txt.

There are a few issues that would need to be resolved.  The python.org binary installers and most Unix-y distributions can optionally install local copies of the Python doc set to eliminate the need for network access to docs.python.org.  There is platform-specific code in EditorWindow.__init__ to search locally for the doc files, falling back to http://doc.python.org if a local copy is not found.  The simplest approach would be to do the same for IDLE help, however that would have the drawback of there being no help available if the local copy had not been installed and there was no internet access.  Of course, that is the case today for the Python documentation.  If that is not acceptable for the help text, the docs build process could make a copy of a version of the library/idle.html into idlelib or, less desirable, the proposed plain text file could be the fallback.  Also, on Windows, AFAIK, the doc set is usually installed as a chm file which normally opens to the top-level index page in the help viewer.  A question is how to open directly to the IDLE page.  By inspection in the help viewer on Windows 7, it seems that a web browser.open call to a URL that appends '::/library/idle.html' to the existing pythonxxx.chm URL will open a browser window to the right page.  But there may be a better way to do that.

From the user's perspective, ISTM that having the help in the rich format possible with the existing doc html would be a huge advantage over what can be provided with any plain text format.  Comments?

Great idea. Using a browser to display help text has become pretty common. For many games, for instance, this has superseded .pdf manuals, which supercedes paper. Unless there is already a tk extension to display html this seems like a good idea. We could even put a link to a youtube video tutorial if there is one or if someone makes one. (There seems to be some creative types interested in helping with Idle. This would be a good way for someone to contribute.

Ned,
   Using a web browser is a great idea!!!!  I like it because it removes code from IDLE making IDLE even simpler (and better).  Besides it would take us forever to duplicate some of the functionally that exists in today's modern web browser.  

Zach,
   What do you think?

Terry,
   What is this paper technology that you speak of?  A youtube video showing off IDLE would be fantastic.

Defaulting to opening a browser window sounds great to me.  In those cases where there is no network access, though, I think we should keep a fallback help.txt, and I think the Sphinx text rendering is about the simplest and easiest fallback we can get.  Shall we open another issue for using a browser for Help, and should this issue wait on that one, or can this go ahead in the meantime?

I just discovered that on Windows, the 3rd help option "Python Docs - F1" opens the local doc set whatever.chm in the Windows Help (HTML) Viewer. This is the same as the start menu choice. So on Windows, opening the same offline copy to the IDLE page, if possible, would be the thing to do. Perhaps the Windows code for opening the docs to the toc page can be reused to open them to the IDLE page.

I have avoided editing the Idle docs because of this issue, but idle.rst needs some corrections and updates now, and will soon need some additions.  I want to only edit idle.rst. So I want to resolve this issue soon.

As I mentioned before, I think the best idea is Ned's: copy Doc/build/html/library/idle.html to idlelib (as help.html for now) and change the 'IDLE Help' code to display it in a browser.  This will be easy and will end the duplication and DRY violation that is the goal of this issue.

Zach, can you produce a makefiles.v4 patch (without .rst changes) that patches the makefiles to add a target do the copying?  The same patch (or another) should add idlelib/help.html to .hgignore so idle developers and testers can 'make' the new target without committing it.  Georg can then update PEP 101 with the new release step.

This change will provide a nicely formatted, always-available local copy that should match the installed micro-version.  Because of PEP 434, later online versions of the doc may contain changes that do not apply to the installed version.

After this is done, and anything in help.txt but not in idle.rst is moved, help.txt will be deleted.

issue16893.v3.diff mixes together changes for this issue and a separate issue of editing idle.rst.  I will separately apply some of your mising changes, but I don't think I like adding the new separators and even more whitespace.  Also, I assume that 'Control' in .txt was changed to 'C' in .rst because OSX uses 'Command' instead of 'Control' and 'C' stands for either.  That is aside from the issue of abbreviating something used repeatedly.  Anyway, I think this is a discussion for another tracker issue.

Sure; here's a patch.  I suspect we may want to try giving Sphinx some different options to clean up the output a bit, but I'm not experienced enough with Sphinx to have any specific suggestions.

New changeset 6db65ff985b6 by Terry Jan Reedy in branch '3.4':
Issue #16893: For Idle doc, move index entries, copy no-subprocess section
https://hg.python.org/cpython/rev/6db65ff985b6

New changeset feec1ea55127 by Terry Jan Reedy in branch '2.7':
Issue #16893: Update 2.7 version of Idle doc to match 3.4 doc as of the just
https://hg.python.org/cpython/rev/feec1ea55127

New changeset 283c364c372a by Terry Jan Reedy in branch '2.7':
Issue #16893: Update Idle doc chapter to match current Idle and add new
https://hg.python.org/cpython/rev/283c364c372a

New changeset 2fc341c6e314 by Terry Jan Reedy in branch '3.4':
Issue #16893: Update Idle doc chapter to match current Idle and add new
https://hg.python.org/cpython/rev/2fc341c6e314

The goal of this issue is that Idle -> Help -> IDLE Help display a good-looking, accurate help page based on Doc/library/idle.rst.  The current Lib/idlelib/help.txt is neither good-looking nor accurate and must somehow be replaced, even if by temporary means, before another release (2.7.9 was missed, 3.4.3 is coming in January).

Ned's gave some comments on the alternatives in msg185726.  I am extending those (and in some places implicitly referring back to them) based on further thinking and investigation.  A summary of what I currently think might be best:

if Windows:
  open .chm to Idle page
else:
  try:
    open offline docs to Idle page
  except failure:
    open generated .txt without `s.

The easiest alternative would be to display the online python.org page in a browser.  It looks nice, has a sidebar index, and access to the rest of the docs.  An obvious problem is that this requires a browser and online access, either of which might be lacking in a classroom or when one is away from home base.  More subtle is that Idle features can be changed or added at any time.  The online docs, generated daily from the repository, should reflect the repository version of Idle.  Released versions need a doc frozen at the time of release.

Still easy, perhaps, would be building Doc/build/html/library/idle.html and copying it to idlelib/help.html as part of the release process*, and changing the code to display that instead 

However as Zack hinted ("clean up the output a bit"), this file is not a standalone file.  It has many links to files in '..', such as "Index" and "Copyright"; these would have to be removed completely as the text is useless without the link.  For cross-reference link, such as 'Tkinter' or 'sys.path', the text should be left but the linkage removed.  In both cases, clicking the link gives a File not Found page. The broken links to javascript formatting files in '../.source' appears to be ignored and the formatting simply not done.  So the page index is at the bottom of the page instead of the sidebar.

* The beginners who most need the help doc are running installed Python, not respository builds, so the latter concerns me less.

But why make an off-line page when there already is one in off-line docs?  I presume that the linux and osx doc urls just need 'index.html' replaced with '/library/idle.html'.  The 'standard location' for linux is x.y.z specific.  The python2 rpm location does not seem to be.  The osx location I will not guess about.

For Windows, startfile('x.chm') does the same as double-clicking x.chm in Explorer, which is to open the file with the Help Control, with a nice sidebar for contents, index, search, and favorites. Adding "::/library/idle.html" does not work because it makes the path invalid.  But with an special prefix

>>> os.startfile("mk:@MSITStore:c:/programs/Python34/Doc/python342.chm::/library/idle.html")

opens the page in InternetExplorer.  However, there is no sidebar and the page has to be clicked on the taskbar to make it visible.  In 1 of 4 tries, the above failed.

An answer to this StackOverflow question

https://stackoverflow.com/questions/11076952/open-chm-file-at-specific-page-topic-using-command-line-arguments?rq=1

prompted me to try

>>> subprocess.Popen("hh ms-its:c:/programs/Python34/Doc/python342.chm::/library/idle.html")

This opens the chm doc with the help control (and sidebar) to the Idle page, just as I wanted.  Since Doc/pythonxyz.chm is part of the installation, I think this is the solution for Windows.  The one glitch is that testing that the exact code in the repository works needs to be done with a pre-final install.  That is already true of testing that "Help -> Python docs F1" opens the .chm to the index page.  These both should be part of a new installed idle test file.

After generating html.txt from idle.rst, the backticks could be filtered out.  It would be close to what we have now.  Is there any need to keep it as a backup for non-Windows systems?  If not, I would be inclined to delete it.

For repository builds, users could go to the online Idle page.  I might even look for ../../Misc and if present, open it directly.  That page would be at most a day out of date.

If it's useful, I quickly threw together a routine that takes the HTML generated by Sphinx, and using html.parser, strips out the navigation stuff, and stuffs it into a Tk text widget, doing appropriate things with the subset of the formatting it uses (lists, headings, etc.).

If something like that seems appropriate to use, let me know and I can clean it up a bit and post it as a patch.

Trying to produce .txt from .rst or .html was the original idea but the result in the upload help.txt is not terribly good. We decided that it is reasonable these days to depend on a browser being present and use the webbrowser module to display the html in a browser. Some effort has been put into getting decent looking html from rst. On windows, the .chm installed with Python can be used as a backup should the browser not work.  

The reason for not simply starting a browser on the online Idle page is that the online docs reflect the repository version of Idle, which may have features not in the released version.  So a 'frozen' doc is needed.  (Note that the .chm is already frozen.) 

The steps and issues I see.

1. Should there be a separate idlehelp target, as in Zach's v4 patch, or should copying the one file be part of building the html files.  The issue here is making sure that the idlelib/help.html is rebuilt at the time of release.  Re-generating the html files is, I presume, part of the release process, whereas a separate step would have to somehow be added.

2. We already use webbrowser to start the online docs, I believe on all systems.  The code can be reused with a different target.

3. Is an exact copy ok, or should some nagivation be removed?  We need to try the file as is and see.

4. The html file should be added to .hgignore and not checked into the repository.

Ok. In case you're curious or might find it useful elsewhere, I've attached sphinxview.py, which renders the HTML into a text widget (with formatting, it's not converted to plain text).

This looks more maintainable than I expected, so I will definite try it out.

In msg185726 Ned said "Ironically, the Tk text widget is perfectly capable of rendering richer text styles." He was and is correct.  Mark Roseman offered to take up the challenge.  In spite of my skepticism, he went ahead and built sphinxview around HTMLParser.  This reduced the core of the problem to "for each tag, content, and end tag, what state change or text insertion is needed. The result is simple enough for me to understand and even edit.

I sent Mark a detailed comparison of the Firefox rendering of idle.html and his tk rendering.  He replied with a new version whose output is essentially identical to Firefox's (minus links), including the green example box.  I intend to use a third version and will work on a full patch.

When running in a repository clone, Idle could access ../../Doc/build/html/library/idle.html.  But for installations, idle.html needs to be in idlelib.  I guess the only was to guarantee this everywhere (excepting overt downstream removals) is to commit it into the repository, unlike most other files built from .rst files). 

To help make sure this happens, I would like to add a comment at the top of idle.rst.  The devguide sentence on comments is unclear to me. Will the following work?
.. # After editing this, also build html, copy idle.html to idlelib,
   test, and upload or commit the html diff also.

It would be nice if the copy and commit process were added to the release process, but I will go ahead even without that.  A nicely rendered, slightly out-of-date .html will be a great improvement over the current situation.

I'd be happy to work on making an 'idledoc' target for the Doc/ makefiles that works with Mark's viewer.  I can't guarantee any kind of timeframe, but the idledoc target wouldn't be necessary right off the bat; it could be done after the main bulk of the patch is committed.

As far as reST comments, the syntax is:

"""
.. This is a comment.
   This is still the same comment.
   And more of the same comment.

This is no longer part of the comment
"""

See http://docutils.sourceforge.net/docs/user/rst/quickref.html#comments

New changeset 4038508240a1 by Terry Jan Reedy in branch '2.7':
Issue #16893: Replace help.txt with idle.html for Idle doc display.
https://hg.python.org/cpython/rev/4038508240a1

New changeset 52510bc368f8 by Terry Jan Reedy in branch '2.7':
Issue #16893: include new files
https://hg.python.org/cpython/rev/52510bc368f8

New changeset 2d808b72996d by Terry Jan Reedy in branch '3.4':
Issue #16893: Replace help.txt with idle.html for Idle doc display.
https://hg.python.org/cpython/rev/2d808b72996d

New changeset e66fbfa282c6 by Terry Jan Reedy in branch '2.7':
Issue #16893: whitespace in idle.html.
https://hg.python.org/cpython/rev/e66fbfa282c6

New changeset 9b79a4901069 by Terry Jan Reedy in branch '3.4':
Issue #16893: whitespace in idle.html.
https://hg.python.org/cpython/rev/9b79a4901069

New changeset 1107e3ee6103 by Terry Jan Reedy in branch '2.7':
Issue #16893: whitespace in help.py.
https://hg.python.org/cpython/rev/1107e3ee6103

I committed a patch that works well enough for release as it. I attached the diff for changed files in case anyone wants to view in Rietveld. 

Still to do for this issue:

* Automate getting the 2.7 and earliest 3.x idle.html copied to idlelib and whitespace-normalized. (Commit/push should stay manual.) Several lines have 1 to many blanks at the end, while the file lacks a final \n.  See e66fbfa282c6. (These anomalies come from Sphinx; idle.rst has been normalized to be checked in.)

I have a .bat file that builds any and copies as needed.  I will to add a pass through Idle's trailing-whitespace deleter, which did just what was needed to commit. (Patchcheck.normalize_docs_whitespace does a here unneeded comparision (always unequal), creates an unwanted and nuisance .bak, and does not appear to work when the last list has no '\n'.)

When I have automation working for me, I would like to add make targets for others to use (Zach's patch 4).

* Edit idle.rst

I will push NEWS and idlelib.NEWS entries separately, along with others for Idle.

I opened new issue #25198 for further viewer improvements (and bug fixes, if any).

New changeset e749080fa0f9 by Terry Jan Reedy in branch '2.7':
Issue #16893: finish deprecation.
https://hg.python.org/cpython/rev/e749080fa0f9

New changeset ff0270e9bdfb by Terry Jan Reedy in branch '3.4':
Issue #16893: finish deprecation.
https://hg.python.org/cpython/rev/ff0270e9bdfb

New changeset 855484b55da3 by Terry Jan Reedy in branch '2.7':
Issue #16893: Add idlelib.help.copy_strip() to copy-rstrip Doc/.../idle.html.
https://hg.python.org/cpython/rev/855484b55da3

New changeset df987a0bc350 by Terry Jan Reedy in branch '3.4':
Issue #16893: Add idlelib.help.copy_strip() to copy-rstrip Doc/.../idle.html.
https://hg.python.org/cpython/rev/df987a0bc350

New changeset f08437278049 by Terry Jan Reedy in branch '3.5':
Issue #16893: Add idlelib.help.copy_strip() to copy-rstrip Doc/.../idle.html.
https://hg.python.org/cpython/rev/f08437278049

New changeset 09ebed6a8cb8 by Terry Jan Reedy in branch 'default':
Issue #16893: Add idlelib.help.copy_strip() to copy-rstrip Doc/.../idle.html.
https://hg.python.org/cpython/rev/09ebed6a8cb8

NEWS entries are done.

I added a function, copy_strip, to help.py to rstrip (in bytes mode) idle.html as it copies to help.html.  (I changed the name to better remember which is which.)  In my .bat file, with Doc as current directory, I use
  ..\pcbuild\python_d.exe -c "from idlelib.help import copy_strip; copy_strip()"
I could change the if __name__ block so that
  ..\pcbuild\python_d.exe -m idlelib.help copy_strip
would work.  Either way, I would like to add 'idlehelp' to makefiles.  But not tonight.

The tests seem to have grown a new Deprecation Warning (triggered when run with python -bWall). Looks like this may be a side effect of revision 2d808b72996d.

[160/392] test_idle
/media/disk/home/proj/python/cpython/Lib/idlelib/EditorWindow.py:88: DeprecationWarning: EditorWindow.HelpDialog is no longer used by Idle.
It will be removed in 3.6 or later.
It has been replaced by private help.HelpWindow

  helpDialog = HelpDialog()  # singleton instance, no longer used

New changeset 26e819909891 by Terry Jan Reedy in branch '2.7':
Issue #16893: Move idlelib.EditorWindow.HelpDialog deprecation warning
https://hg.python.org/cpython/rev/26e819909891

New changeset c607004a98bf by Terry Jan Reedy in branch '3.4':
Issue #16893: Move idlelib.EditorWindow.HelpDialog deprecation warning
https://hg.python.org/cpython/rev/c607004a98bf

Thank you for reporting this. My general problem is that idlelib was only informally private until PEP434 and I only started adding 'private' and 'deprecated' to the code a week ago.  So I am trying to make changes in existing versions while not breaking even unlikely uses of the current API. The specific problem is the the class init method *is* used -- to create an unused instance -- on import.

  helpDialog = HelpDialog()  # singleton instance, no longer used

The patch fixes the problem by moving the deprecation to the show method, which Idle really does not now call.

Through the contributions of several people, Zack's key idea, that Idle documentation should have a single source, idle.rst, has come to fruition. So I think it time to close this.  I already mentioned #25198 for improving the tkinter viewer. I also opened #25218 for revising Zack's patch and possibly using an 'idledoc' target in the release process.