Why does e.g. configparser.ConfigParser contain doc strings with Sphinx incompatible markup?

The markup starts with back-tick, but ends with a single quote.

Example:
https://github.com/python/cpython/blob/master/Lib/configparser.py?ts=2#L26

Sphinx writes these messages:
D:\git\PoC\py\lib\ExtendedConfigParser\__init__.py:docstring of lib.ExtendedConfigParser.ExtendedConfigParser.read_file:3: WARNING: Inline interpreted text or phrase reference start-str
ing without end-string.

Note: ExtendedConfigParser is class derived from configparser.ConfigParser. Inherited methods get documented too.

------------------------
Btw. I have some improvements for this class, how can I contribute them? Who is the maintainer for this class? Please contact me: Patrick.Lehmann@tu-dresden.de

The improved version is available at GitHub: https://github.com/Paebbels/ExtendedConfigParser?ts=2

How can I supply a fix?

I have a branch with lots of fixes.
https://github.com/Paebbels/cpython/tree/paebbels/issue-28710?ts=2

Why don't you accept pull requests via GitHub?


Kind regards
    Patrick

We will accept github pull requests in the future (the transition is underway).

For now, you can create a diff file (using hg diff by preference, but git diff will work) and uploaded it to the issue.  For this issue, please only upload the docstring changes.  For other enhancement suggestions, open separate issues.  (This would be true even if we were accepting pull requests).

The existing docstring markup is probably a remnant of the days when the documentation was written in LaTeX.

Lukasz Langa is the current maintainer of this module.

Here is the patch file created with:
PS> git diff > docstring_markup.patch

This patchfile effects all files with this markup in the CPython repository.


Kind regards
    Patrick

As far as I looked, the patch changes `xyz' in docstrings and quotes to ``xyz``.  A rst expert should verify that this is correct.  In printed strings, `zyz' is changed to 'xyz', which I consider to be correct.

Before applying this, I would want to review in Rietveld, with side by side diff and changes color marked.  However, Rietveld does not like the patch and there is no 'review' button.  I thought it might be the git format, but I found another git patch that did have a review button.  When I downloaded and tried to apply (to default), hg says that there is no diff.  I don't see the problem, but we cannot currently use a patch that does not apply in hg.

Patrick, in order to use patches, we require Contributor Agreements.
https://www.python.org/psf/contrib/
There is an electronic form that makes submission easy.
https://www.python.org/psf/contrib/contrib-form/

I think that we do not generally use ReST markup in our docstrings.  So replacing `x' with 'x' would be more correct, I think.  In many cases the quotes could just be omitted entirely.

The patch command says:

  patch unexpectedly ends in middle of line
  patch: **** Only garbage was found in the patch input.

Hello,

I used this regexp on all files:
--------------------------------------
match pattern: `([A-Za-z0-9_]+)'
replace pattern ``\1``
--------------------------------------
I assumed that only identifiers where quoted in such way. I think my editor found around 139 matches in the whole CPython repository.

I found some of these markup in non docstring strings, which I reverted as far as I found them by manually reviewing all changed files.

For a colored diff, see my Git branch: https://github.com/Paebbels/cpython/commit/6d3f348f71b5b0ae9fbfcb8fdbba72dc5fac428a?ts=2
There is also a PR-, commit-, and line-based comment feature box GitHub.

How you solve it is up to you, but I would like to get rid of hundreds of warnings in my Sphinx runs, when modules are inherting code (and docstrings) from Python.

Kind regards
    Patrick

.... I signed the CLA.

I also found some docstrings using double back-tick plus double single quotes.

For example: ``x.y = v'' in builtins.py in function setattr(...).

Marking as a stdlib-wide issue since it's not configparser-specific.

FYI, as you probably noticed, the `term' syntax predates Sphinx and is used in lots of places. While I think it would be nice to fix it, it's a big diff.

Any progress on that issue?

1.5 years passed by and it should be possible to fix the Python documentation in that time, right?

Thanks for coming back to this.

We're accepting PRs via github now, so the next step now would be to make it into a PR.  

Sometimes things just get forgotten and you have to nudge them to get them moving (see the devguide for guidelines about when it is appropriate to nudge an issue).

I agree with David about replacing `x' with 'x' instead of ``x``, so please make this change.  Do you know what ``x.y = v'' is supposed to mean?

David, do you want one PR with 139+ changes, or should we start with fewer?  Patrick, are these usages scattered over numerous files or concentrated in a few?

Against what branch should I create the PR?

I was a huge number of changes.
I think I'll create multiple PRs to ease the review.

master, and maybe backport from there, unless the PR never touches master.

I wonder if we should bring this up on pydev.  Without discussion and agreement, someone might object just because there are so many changes.  Or someone might say that they want the 'illegal' markup kept.

I see that the one docstring you linked to, 
https://github.com/python/cpython/blob/master/Lib/configparser.py?ts=2#L26
had 9 occurences, 1 for each parameter.

I never see any warnings.  It must be that our build system somewhere discards them.

This should probably be discussed on python-dev.  In particular, changing from single backticks to double backticks is something that makes Sphinx happy but will uglify the docstrings for all other ways of viewing the docs.

Also, note that most of our docs aren't autogenerated from the docstrings so we've never had any particular need to impose Sphinx markup requirements for docstrings.

In my last comment above, I forgot that this issue is about *docstrings*.  We do not officially process docstrings with Sphinx, so there are no warnings to be suppressed.

PEP 8, which covers style for the stdlib, refers to https://www.python.org/dev/peps/pep-0257/
Neither says anything about markup and last I remember, there should not be any for the stdlib. Pep-0257 gives this example:

def complex(real=0.0, imag=0.0):
    """Form a complex number.

    Keyword arguments:
    real -- the real part (default 0.0)
    imag -- the imaginary part (default 0.0)
    """
Here, parameter names are indicated by the formatting, not by markup.  If `' is used in such lists, it should just be deleted.  I believe 'name' is sometimes used in running text.

The help() function prints a docstring as is.

Right, my opinion is that we shouldn't be putting markup in docstrings.  They are (from our point of view) pure text.

I don't know if discussion on python-dev is warranted, it seems like a fairly uncontroversial change, since it brings the docstrings in question into compliance with our general practice in the majority of the stdlib.  Unless my impression about that is wrong :)

I don't have an opinion on multiple versus single PR for this.

By the way, in case anyone is curious, I'm pretty sure that markup is left over from the days when tex/latex was what docs were *written* in.

What is the problem? Docstrings are not written in the reStructuredText format in general. If Sphinx complains about docstrings in the stlib, don't run Sphinx for the stdlib files or report a Sphinx bug. Even if remove all `x', docstrings still will not be valid reStructuredText. For example the last three lines will be joined in a single paragraph for the docstring in msg319848.

`name' does not have special meaning in (La)TeX. AFAIK `name' is common writing of quotes in English texts. It predates reStructuredText and was widely used in Usenet and mailing lists (as two spaces after sentence-ending punctuation, double hyphen for a dash, _n_a_m_e_ to simulate underscoring, etc). It is just a part of old computer-typing culture (or may be even pre-computer).

No, it is (somewhat) unique to tex.  If you write `word' tex would turn that into a pair of opposing quotes in the typeset document, as opposed to 'word' (the convention in regular text/emails/posts/etc) where you'd just see ascii quotes.  tex would render 'word' as a closing quote both before and after word, which looks weird in typeset text.

There's no bug here; as you say we aren't interested in making the docstrings parseable as restructured text (at least, I'm not).  For me, this is about getting rid of the now-odd-looking tex leftovers and making the ascii styling consistent with the bulk of our docstrings.

It's not a big deal, though.

> AFAIK `name' is common writing of quotes in English texts

I don't remember ever seeing it before.  It looks like a typo to me, and I am sure it will to most readers.  I think it should be corrected as if it were a typo.

Oh, you are right. I didn't write much English TeX, and used other types 
of quotes.

I'm sure that I seen `such quotation' in non-TeX files, but maybe my 
memory fools me. In any case using this writing here is likely an 
artifact of copying from a TeX documentation.

Having single quotes in docstrings is also ok for Sphinx documentation.

Btw. ReStructured text (docutils) was invented to document Python sources, why is it not used by Python?

RestructuredText, DocUtils, and Sphinx were developed independently, by people other than the pydev/cpython group.  (The proposal to include DocUtils in the stdlib was rejected.)  We converted to .rst for the Python documentation sources files about a decade ago.  Sphinx turns them into the .html files you can see online.  RestructuredText markup can now also be used in PEP, news/changelog, and What's New sources.  These are things that most people only view online or in other processed forms, not in source form.

On the other hand, stdlib docstrings are mostly viewed unprocessed in .py sources or as output from help() in text consoles or text widgets.  So markup in stdlib docstrings would impact everyone, not just core developers.

Marking this as closed.  Though well-intentioned, the suggestion is predicated on a misperception of the role of Sphinx for CPython and existing PEP recommendations on docstring practices.

I would still like to see the legacy tex markup removed from the docstrings, so I think closing this issue is not appropriate, but it's not a big enough deal that I'll push for it if Raymond wants to keep it closed.

I agree with David.  I would like the ugly markup changed independent of how Sphinx treats it.  I was thinking of changing the title to "Change obsolete tex markup in stdlib docstrings".  The intent of `word' was for people to see balanced quotes, which in ascii text means 'word'.