This is a straight-forward project.

Pydoc currently generated 1990's style html which mixes content and presentation, making it very difficult for users to customize the appearance of the output.

It is full of html like:

<table width="100%%" cellspacing=0 cellpadding=2 border=0 summary="heading">
<tr bgcolor="%s">
<td valign=bottom> <br>
<font color="%s" face="helvetica, arial"> <br>%s</font></td

<td align=right valign=bottom
<font color="%s" face="helvetica, arial">%s</font></td></tr></table>
''' % (bgcol, fgcol, title, fgcol, extras or ' ')

    def section(self, title, fgcol, bgcol, contents, width=6,
                prelude='', marginalia=None, gap=' '):
        """Format a section with a heading."""
        if marginalia is None:
            marginalia = '<tt>' + '&nbsp;' * width + '</tt>'
        result = '''<p>
<table width="100%%" cellspacing=0 cellpadding=2 border=0 summary="section">
<tr bgcolor="%s">
<td colspan=3 valign=bottom>&nbsp;<br>
<font color="%s" face="helvetica, arial">%s</font></td></tr>
    ''' % (bgcol, fgcol, title)
        if prelude:
            result = result + '''
<tr bgcolor="%s"><td rowspan=2>%s</td>
<td colspan=2>%s</td></tr>
<tr><td>%s</td>''' % (bgcol, marginalia, prelude, gap)

Please convert it to simple, validated HTML with the formatting moved to a simple, validated default style sheet. Liberally apply div/span elements with class/id attributes as necessary.

I believe Ron has done something along these lines already. See bpo-2001. Committed css file is empty, (see r86971) but I am adding Ron to "nosy" in case he has something in the pipeline.

See also the calendar module for an example of a "modern" html.

I uploaded the css file I used in an experimental version of pydoc. It may give some useful starting values.

Before this is done, the old server code should be removed (also for 3.3). (another issue?) There are two files in the tools/scripts directory that may need attention as well. pydocgui.pyw and pydoc3.

Those two scripts are just three-liners pydoc launchers.

I disagree about the CSS reset. Default values and user-set values have a reason for being there, and I share the viewpoint that they should not be reset.

I'm looking for a deeper fix, all the in-line styling replaced by a stylesheet. Can you guys work together on bring this to fruition?

Eric, most of what's in that file is what I figured out by trial and error in order to get it to work on the different browsers at that time. (about 3 years ago.) You are probably more experienced with css than I am, so you are more than welcome to update and change anything in there. :-)

What do you think about starting with a set of static html pages to get the css and html to work nice, and then only after that is done, edit pydoc to generate those pages. That should get us mostly there, and simplify what needs to be done in pydoc.

A good procedure is to start afresh with an empty, embedded style sheet and replace the html styling attributes one at a time, while keeping the overall look and feel the same. At the end of the process the embedded style sheet can be converted to external.

I think that's how I ended up with the style sheet I uploaded. It works, but it can be a slow process.

Another factor is the pydoc server will reread an external style sheet on browser refreshes. So you can see the results of style sheet changes without restarting pydoc. If it's embedded, you need to edit the program and restart everything each time.

I was thinking that I may still be able to make a set of static html files that go along with that style sheet. Then we can adjust the css class names and make other changes, then use that as a guide for replacing the html.

Eric, what do you think?

The uploaded stylestyle is *much* bigger than I expected. There should be no more than a one-to-one correspondence. With inheritance, perhaps even fewer entries.

Ok, I just looked at them again, I didn't remember how different it was. They probably won't be much help other than maybe seeing how some things could be done. Here's a zip file of some saved pages, so you can take a look.

The CSS also has a validation error, some warnings and could be written better IMHO (specifically I don't like the placement of the {}).

I also don't believe that extensive CSS resets are useful -- they usually just lead to lot of redefining. I usually limit myself to * { margin: 0; padding: 0; } and possibly specify specific values when necessary. Minor differences with different browsers can be perfectly fine in lot of cases.

[Raymond]

I'm looking for a deeper fix, all the in-line styling replaced by a
stylesheet. Can you guys work together on bring this to fruition?
When I talked about the CSS reset, I was referring to a precise part of the file proposed by Ron, so we’re already discussing together :-)

I wonder how desirable it is to preserve the look and feel of the pages. We all agree on externalizing the CSS and add a way for the users to add their own stylesheet; why not take the opportunity to also improve the style? Huge blocks of colors are not that attractive to me :)

Regarding workflow, I’d find easier to start from bare HTML that works nice (yes, I test with w3m) and then add style. Technically, I’d like to maintain the HTML as a small set of files (in pydoc_data) containing fragments of HTML with placeholders ({} or $): That’s easy to edit, to validate (a very simple script can put the fragments together) and to use.

I agree that the CSS file should be as short as possible (in content), but not too short (in file size). For example, trailing commas in properties and brackets on their own line will allow nice diffs, just like Python.

(I won’t have much time for Python in December, but we have a lot of time before 3.3b1 :)

There's no rush on this one.
There desired properties are:

• separate content from presentation
• validated CSS and HTML
• simple CSS so that people can *easily* customize it
• possibly offer two premade stylesheets
  1. a default beautiful one
  2. one the closely approximates the current look and feel

If the current look cannot be recreated, it should be taken as a sign that the content/presentation split wasn't done well.

Here is a tentative start on this. (css_v1.diff)

The css file is much better. It's shorter, simpler and validated.

The header and navbar panel use it in the new server.

Added a markup call to the topic page contents. (The same markup call is already used for the doc strings.)

Extra items in the css are what I come up with from testing different ways of doing things.

The HtmlDoc class has methods that take colors. Can this be changed or does it need to be depreciated first?

    def heading(self, title, fgcol, bgcol, extras=''):
        """Format a page heading."""
        return '''
<table width="100%%" cellspacing=0 cellpadding=2 border=0 summary="heading">
<tr bgcolor="%s">
<td valign=bottom>&nbsp;<br>
<font color="%s" face="helvetica, arial">&nbsp;<br>%s</font></td
><td align=right valign=bottom
><font color="%s" face="helvetica, arial">%s</font></td></tr></table>
    ''' % (bgcol, fgcol, title, fgcol, extras or '&nbsp;')

For the interactive server, I can override these methods with no problem, but the generated docs won't benefit from this until the HtmlDoc class is replaced. Any suggestions?

Well, you could reuse these arguments to mean CSS classes, and have styles like ".red { color: red }" :)

It may be useful to change those to 'id=' and 'class=' if possible.

It isn't clear to me how much of pydoc is still part of the public api in python 3.x. pydoc.__all__ is set only to ['help'].

Entering help(pydoc) just gives the basic help and command line arguments along with.

DATA
__all__ = ['help']
help = <pydoc.Helper instance>

There is nothing in the official (online) docs on importing pydoc or any of it's classes or methods.

But dir(pydoc) shows all the methods, and the HTMLDoc class is still importable even though they aren't listed in __all__.

Here is a new diff which updates all the new pydoc pages to use the css file.

The css file is simpler and cleaner. I also made a few adjustments to the url handler error handling, and changed the titles in the head sections so they say "Pydoc" instead of "Python" as they are PyDoc pages about Python.

None of these changes effect any of the old pydoc code yet. This is about as far as we can go without removing the old tk panel and server.

Time for some feed back. And how close do we really need it to be to the original?

:-)

A few comments about css_v2.diff:

1. when the value is '0', there's no need to specify the unit (e.g. 0em);
2. when the color is specified the background-color should be specified as well (and vice versa);
3. hex colors (e.g. #00FF00) should be preferred to named colors (e.g. gray);
4. some selectors and properties don't work with older browsers (e.g. E > F or min-width);
5. there are a few empty dl.*{} that I would remove (unless you plan to fill them later);
6. the style I prefer for CSS is:
   selector {
   property: value;
   }

Regarding the HTML:

1. using an HTML 4.01 strict doctype would be better;
2. all the style-related attributes and elements should be removed (e.g bgcolor, valign, <font>);
3. using .red { color: red; } is not a good idea. Classes' names should describe the role of the element (e.g. header, entry) and not their style (otherwise when you want to change the page with a blue theme you'll end up with a .red { color: blue; }). If the colors are passed directly to the HTML they should be removed and left to the CSS(s) only. I don't know the code well enough to say if this is doable and/or if it requires a deprecation first;
4. the <li>s in html_header() should be closed, same for all the other elements that support a closing tag, even if optional (e.g. <dt>, <dd>);

There are also some minor incontinences in the indentantion, e.g.:
+ link_list = ['<a href="%s.html">%s</a>' % (name, name)
+ for name in sys.builtin_module_names
+ if name != '__main__']
+ contents = [html.index_columns('Built-in modules',
+ link_list, css_class='section modules')]
+ link_list = ['<a href="%s.html">%s</a>' % (name, name)
+ for name in sorted(Helper.keywords.keys())]
(the "contents" one is indented correctly), and some extra space after the '('.

If the colors are passed directly to the HTML they should be
removed >and left to the CSS(s) only. I don't know the code
well enough to say if this is doable and/or if it requires a
deprecation first;

We may have to do dome depreciating when it comes to the old HTMLDoc class methods. Would it be possible to depreciate the whole class instead of the separate methods? If so, it would be good if we can squeeze that into 3.2. <hint hint> Or else we may not be able to finish this until Python 3.4.

The empty css elements will be used later.

Here is the css_v3.diff for review. I'd like to hear any thoughts about the general html structure and class/id names.

Beside validating them, I test with firefox and chromium. I don't have easy access to ms-explorer or the current mac browser.

These arguments should not really be of concern. If we indeed deem them public API, they can stay but be ignored.

To go forward I can create a new private api instead of changing HTMLDoc, that would be preferable.

Should the -w option also use the new html pages? Or do we need a new option for that?

A reminder: Check for instances where html.escape is not called on data inserted into the html pages.

I'll update the patch as the non-css (error handling) parts made it into python 3.2. :-)

I added a few comments for ccs_v3.diff on rietveld.

Thanks for the update, Berker.
Would you be able to remove the 'easy' keyword from this ticket?

I'll keep myself in the nosy list as I'm interested to learn about this anyway :)

When rhettinger created this issue, the goal was

"Pydoc currently generated 1990's style html which mixes content and presentation, making it very difficult for users to customize the appearance of the output. ... Please convert it to simple, validated HTML with the formatting moved to a simple, validated default style sheet."

Apparently, the statement of this issue is no longer correct. However, in order to do so, one apparently needs to have access to a private list mentioned in a comment (https://mail.python.org/mailman/private/core-mentorship/2016-August/003622.html)

One thing that motivated to submit my tentative "solution" to this issue here almost two years ago (solution which, I thought, met the stated goal) was a discussion at Pycon as to how one could encourage new contributors. Is this the purpose of the core-mentorship list? Does one need to join the list if one wishes to contribute?

Hi Andre,

You can subscribe to the core mentorship mailing list here:

https://mail.python.org/mailman/listinfo/core-mentorship

Once you subscribed, you'll get access to the said discussion :)

It is not required, but it's a great place to ask questions or get help from existing core developers.

HTH

I've removed the easy tag. (Aside for those who can do the operation: to remove all tags from an issue one must select '-no selection-'; just ctrl-clicking the last tag to remove its highlight doesn't work...or at least that's what I remember, I didn't actually test it this time.)

I see that pydoc still useful, and have 6 points I'd like to work on with pydoc. Is this too much for one issue? Should I break bits out to another issue(s)? Although it would make sense to do it all in one go.

-----------------------------------------------
Pydoc is a good way to get users (especially beginners) used to writing docstrings, and leads nicely into doctest and similar tools.
Other languages, such as Perl and Java have similar tools that generate text or html documentation from code.
Some have used pydoc classes for their own uses.
It is not intended that pydoc would compete with Sphinx. Sphinx is not core.

pydoc changes
-----------------------------------------------

1. Fix .\ (or ./) problem
   Currently, documenting a single module requires the prefix of .\
   TODO: Change to check:
   If it's a directory, document all .py files in that directory.
   If it's a Python file, document that.
   If it's one of the special keywords, display that documentation.
   Otherwise, display an error.

Currently:
user@server:/mnt/c/course$ pydoc -w helloFunc.py
no Python documentation found for 'helloFunc.py'

user@server:/mnt/c/course$ pydoc -w ./helloFunc.py
wrote helloFunc.html

TODO: Update the help strings.

2. Fix command line run error message
   Currently, passing an argument (that is not a keyword and is not a filename that exists) displays an error message that isn't relevant to this context:

   C:\course> py -3 "C:\Program Files (x86)\Python35-32\Lib\pydoc.py" notthere.py
   No Python documentation found for 'notthere.py'.
   Use help() to get the interactive help utility.
   Use help(str) for help on the str class.

This changed from Python2:
C:\course> py -2 "C:\Python27\Lib\pydoc.py" notthere.py
No Python documentation found for 'notthere.py'.

TODO: Remove the last two lines of the help output from Py3 pydoc when run from the command line.

3. Add "-h" command-line argument.
   TODO: Add "-h This help summary" to the output of pydoc -h
   All the existing options are single letter, so make it just -h
   Continue the current behaviour of "no command arguments or switches" displays help.

4. TODO: Unknown command line option should give an error.
   Currently, pydoc with an unknown command line option doesn't give an error, but just displays the standard help:
   C:\course> py -3 "C:\Program Files (x86)\Python35-32\Lib\pydoc.py" -j
   pydoc - the Python documentation tool

   pydoc ...
   Show text documentation on something. may be the name of a
   Python keyword, topic, function, module, or package, or a dotte
   ...

This can add confusion. It would be more useful to display a specific error, and refer users to more help, like ls does:
user@server:~$ ls -j
ls: invalid option -- 'j'
Try 'ls --help' for more information.

5. TODO: Improve HTML output
   Oh yes.
   There's a few good contributions on this, but I think Frédéric Jolliton's summary is the most succinct.

6. Additional command line argument
   To allow certain command line arguments.
   Sidenote: Both pod (http://perldoc.perl.org/pod2html.html) and javadoc (http://download.java.net/jdk7u2/docs/technotes/tools/solaris/javadoc.html) have many command line arguments. I'm not proposing for significant changes, but I think the below options would add significantly to the usability, and are modeled on pod and javadoc.
   It is not intended that pydoc would compete with Sphinx.
   TODO:
   + --help Same as -h for help
   + --css= Override the built-in stylesheet
   + --output or -d or --dir Output directory name
   + --verbose More verbose output

You sound like you think we want to get rid of pydoc, which is certainly not true :). As I understand it it is only the *html* output of pydoc that we are considering deprecating.

Yes, you should open separate issues for your proposed changes. 1 and 2 should be separate issues, 3 and 4 should be combined into a proposal to convert pydoc to using argparse. 5 you can skip, and 6 I don't think will be accepted, although you are welcome to propose it along with some motivating use cases.

I disagree with deprecating HTML output. Pydoc has builtin HTTP server, it provides the documentation with hyperlinks. It would be nice if IDLE use this feature for displaying interactive help.

I think we should enhance HTML output not just by making it looking better, but by adding more interactivity. For example add the ability to collapse classes, methods, etc, add more hyperlinks.

That sounds like a reasonable argument to me.

To clarify something in my last post: 5 should be skipped in terms of opening new issues because it is already covered by this issue.

If Terry would like to see pydoc enhanced to support idle, then I think that would decide the issue.

Since David added me as nosy, I will address the IDLE issues raised above.

But first, a question. Does 'deprecating HTML output' merely mean deprecating the old output in favor of new and better HTML output, or deprecating having any html output, which would also mean deprecating the html doc server? I am strongly against the latter.

Currently, IDLE, in both pyshell and run.py, imports pydoc and sets "pydoc.pager = pydoc.plainpager". Bypassing pydoc.getpager this way is not documented, but seems perhaps necessary. Other than this, I believe 'help(xyz)' is treated by IDLE just like any other user code.

Serhiy, msg278126 > "It would be nice if IDLE use this feature"

There are actually two features: hmtl output, and the html server, which uses the default browser to display the output.

Since a year ago, IDLE help displays the html IDLE doc produced by sphinx in a subclass of tkinter Text that feeds the page to a subclass of HTMLParser. The code is in help.py. Sphinx's html seems to follow Frédéric's guidelines in msg246917. I suspect the CSS files are ignored. We might want to do something similar with pydocs html output.

Raymond: is sphinx's pydoctheme anything like the css you are looking for? Could it be used as a starting point?
<link rel="stylesheet" href="../_static/pydoctheme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />

There have been requests that 'long' help output (for modules and classes) be displayed separately from Shell itself so that it does not obscure history and be easier to refer to. If we do this, using html instead of plain text would be nice, especially if new features were added.

Serhiy, cont. > "I think we should enhance HTML output not just by making it looking better, but by adding more interactivity. For example add the ability to collapse classes, methods, etc, add more hyperlinks."

I mainly use pydoc server for tkinter. I would *love* have the repetitive 'inherited methods' section for each class collapsed by default. Ditto for when I use help interactively. To me, this is the single worst feature of pydoc output.

David msg278127 > "If Terry would like to see pydoc enhanced to support idle, then I think that would decide the issue."

I am not sure what you mean by 'the issue' and I can't currently think of anything IDLE-specific that pydoc should do, other than be consistent. However, producing modern, decent looking html output would make html use possible. Incorporating the current output, as displayed in the server, might be a net negative PR move.

Side note: thinking about how to make a clickable link in a Text widget, I realized that that IDLE already does some syntax tagging for colors (keywords, builtins, def and class identifiers). The same tags could be bound to double-click or right-click to display help popups, such as
----------------------------------------------------------
The "if" statement is used for conditional execution:

if_stmt ::= "if" expression ":" suite
( "elif" expression ":" suite )*
["else" ":" suite]

It selects exactly one of the suites by evaluating the expressions one
by one until one is found to be true (see section Boolean operations
for the definition of true and false); then that suite is executed
(and no other part of the "if" statement is executed or evaluated).
If all expressions are false, the suite of the "else" clause, if
present, is executed.

Related help topics: TRUTHVALUE
--------------------------------------------------------------------
with the grammar chunk highlighted and the related topics clickable. What IDLE would need here is consistency in the format of the help texts.

My memory of the thread (I haven't gone back and checked) was that the thought was to drop html output altogether because the sphinx docs were more useful for anyone wanting to use html, and network connectivity was so universal in this day and age that having a local html server for offline uses was of minimal utility.

So your vote that it be kept because it is useful to idle and can be made more useful by improving the html seems to me to decide the issue in favor of making the improvements (given someone willing to do the patch review and any additional needed coding).

I've found the HTML to be useful (-w mode, not running a server) for generating quick documentation (much lighter weight commitment than using sphinx). I show this in my intro classes and the engineers are usually impressed with it.

This is really easy and requires neither configuration files or reST markup:

    $ python -m pydoc -w statistics
    wrote statistics.html

So, my vote it to keep it.

I would still like modern HTML 5 with CSS however.

On Sat, Mar 23, 2019 at 6:27 PM Raymond Hettinger <report@bugs.python.org>
wrote:

Raymond Hettinger <raymond.hettinger@gmail.com> added the comment:

I've found the HTML to be useful (-w mode, not running a server) for
generating quick documentation (much lighter weight commitment than using
sphinx). I show this in my intro classes and the engineers are usually
impressed with it.

This is really easy and requires neither configuration files or reST
markup:

$ python -m pydoc -w statistics
wrote statistics.html

So, my vote it to keep it.

I would still like modern HTML 5 with CSS however.

I submitted code to do this in January 2015 on the bug tracker but, except
for some interest by a single person, my submission was essentially
ignored. I do not claim it is perfect, but I thought it was a significant
improvement as it did use HTML 5 with CSS.

André Roberge

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue10716\>

---

I stumbled upon this issue because I think this is a potential improvement. One of the reasons I don't use it is due to the outdated UI but since it also generates HTML doc for installed packages in a virtual environment I find this to be a useful feature like rdoc (ruby), rustdoc (rust) and are actively maintained helping documentation of third party packages too for local reference.

I have less knowledge about CSS but I tried updating css_v3.diff with master and added boostrap.css from CDN to pydoc which adds spacing along with improving layout and font. Attached is a screenshot that is looks better with the CSS patch and bootstrap. Including bootstrap is not lightweight option in the Python distribution but can be seen as a proof of improvement.

Thanks @aroberge for the patch and your efforts on improving it https://aroberge.blogspot.com/2015/01/scratching-itch-improving-pydoc.html

Hey all, I'd like to note my interest in this issue getting addressed. I have a use case where I'd like to generate documentation for a module and using non-core modules to do that requires some approval that I would like to avoid.

I understand there are other more important issues but this issue has had a patch waiting for review for ten years. Can we move this forward somehow?

Someone would need to take the patch, apply it to current main branch, fix issues and open a PR.

On second thought, this issue is not purely programming, but also has a UI component. Better HTML and separated CSS would be good, but before that I think we’d need someone with design chops to mock up a better page layout and style (better than big coloured rectangles!), then I think have a discussion on python-dev (not to discuss all details in committee, but to validate the overall change).

Tried myself at it.

I'm not a front-end guy, fasten your seatbelts.

I tried to keep it minimal, but after some work the change is not that small, I focused on the HTML modernization only, not the design (to try to land at least a first step toward modernization).

Does someone want to review #28651?

New changeset c91b6f5 by Julien Palard in branch 'main':
bpo-10716: Migrating pydoc to html5. (GH-28651)
c91b6f5

@JulienPalard The issue seems to have been solved. I think we can issue the problem.