classification
Title: `python setup.py check --restructuredtext --strict --metadata` fails with: `warning: check: Could not finish the parsing.` if the RST document uses code or code-block directives.
Type: behavior Stage: resolved
Components: Distutils Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Marc.Abramowitz, berker.peksag, dstufft, merwok, posita, python-dev
Priority: normal Keywords: patch

Created on 2014-12-16 07:58 by Marc.Abramowitz, last changed 2016-12-15 16:08 by posita. This issue is now closed.

Files
File name Uploaded Description Edit
cfabe07bc98f.diff berker.peksag, 2014-12-16 11:52 review
9b8f6812ff69.diff merwok, 2014-12-16 18:16 review
issue23063.patch Marc.Abramowitz, 2014-12-16 18:17 review
Repositories containing patches
https://bitbucket.org/msabramo/cpython/#fix_distutils_rst_check
Messages (9)
msg232720 - (view) Author: Marc Abramowitz (Marc.Abramowitz) * Date: 2014-12-16 07:58
`python setup.py check --restructuredtext --strict --metadata` fails with:

    warning: check: Could not finish the parsing.

if the RST document uses `code` or `code-block` directives.

This is annoying because the document is valid, but it appears to be invalid
and confuses people. For example, see
https://github.com/ionelmc/pytest-benchmark/pull/4#issuecomment-66940307

How to reproduce this bug
-------------------------

Clone a repo that has a `README.rst` with `code-block` directives in it. E.g.:

    $ git clone git@github.com:ionelmc/pytest-benchmark.git
    $ cd pytest-benchmark
    $ git checkout ab0b08f6fccb06a7909905a8409f8faa8b01e0d8

   Observe that it has "code-blocks" in it:

       $ grep 'code-block' README.rst
       .. code-block:: python
       .. code-block:: python

Observe that RST document is valid, according to `rst2html.py`:

       $ rst2html.py --halt=1 README.rst > README.html && echo "RST was OK."
       RST was OK.

       $ head -n 3 README.html
       <?xml version="1.0" encoding="utf-8" ?>
       <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
       <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">

Observe that `python setup.py check --restructuredtext --strict --metadata` fails:

    $ python setup.py check --restructuredtext --strict --metadata
    running check
    warning: check: Could not finish the parsing.

    error: Please correct your package.

    $ echo $?
    1

**What was expected**: `python setup.py check --restructuredtext --strict
--metadata` should succeed with no warnings, just as `rst2html.py did`, because
`README.rst` is a valid RST document.

**What actually happened**: `python setup.py check --restructuredtext --strict
--metadata` prints a warning and an error and fails, unlike `rst2html.py`

The error is coming from here:
https://github.com/python/cpython/blob/master/Lib/distutils/command/check.py#L142

It's happening because of this line:
https://github.com/python/cpython/blob/master/Lib/distutils/command/check.py#L125

    :::python
    settings = frontend.OptionParser().get_default_values()

If this is changed to:

    :::python
    settings = frontend.OptionParser(components=(Parser,)).get_default_values()

then things work much better (this is how `tools/quicktest.py` in docutils does it for example -- see https://sourceforge.net/p/docutils/code/HEAD/tree/trunk/docutils/tools/quicktest.py#l196)

The attached patch prevents the failure from happening and also adds more information to the error when things go south.
msg232721 - (view) Author: Marc Abramowitz (Marc.Abramowitz) * Date: 2014-12-16 08:09
Testing a few common cases:

$ python setup.py check --restructuredtext --strict --metadata
running check
error: The docutils package is needed.

$ python setup.py check --restructuredtext --strict --metadata
running check
warning: check: Cannot analyze code. Pygments package not found. (line 66)

warning: check: Cannot analyze code. Pygments package not found. (line 99)

error: Please correct your package.

$ python setup.py check --restructuredtext --strict --metadata && echo "RST was OK."
running check
RST was OK.
msg232722 - (view) Author: Marc Abramowitz (Marc.Abramowitz) * Date: 2014-12-16 08:14
Note that this patch does 2 things:

1. Improves the error message
2. Prevents check from failing when there are code-blocks

If I only did #1 and not #2, then output looks like this:

$ python setup.py check --restructuredtext --strict --metadata && echo "RST was OK."
running check
warning: check: Could not finish the parsing: 'Values' object has no attribute 'syntax_highlight'.

error: Please correct your package.
msg232734 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2014-12-16 11:55
Could you add a test? See Lib/distutils/tests/test_check.py.
msg232753 - (view) Author: Marc Abramowitz (Marc.Abramowitz) * Date: 2014-12-16 18:14
OK, I added a test.

See:

* https://bitbucket.org/msabramo/cpython/commits/9b8f6812ff6981b5f195b6bf73cefb0fea46fba6
* https://bitbucket.org/msabramo/cpython/pull-request/1/fix-distutils-setuppy-check/diff

If you want, I can also update the diff attached here, but maybe it's easier to just look at my branch on Bitbucket? Whatever is most convenient for you...
msg232754 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2014-12-16 18:18
The Mercurial integration lets you update the patch with one click.  Decentralized VCSes are nice :‑)

Patch looks good to me.  Thank you.
msg232756 - (view) Author: Marc Abramowitz (Marc.Abramowitz) * Date: 2014-12-16 18:41
What's the next step?
msg232943 - (view) Author: Éric Araujo (merwok) * (Python committer) Date: 2014-12-19 17:34
An active core developer needs to see this and decide to commit the patch.  You can see if there’s someone on IRC.  If there’s no action in a week or two, feel free to ask on python-dev (maybe listing more than one waiting patch, or offering reviews in exchange for core dev time).
msg234051 - (view) Author: Roundup Robot (python-dev) Date: 2015-01-15 05:00
New changeset db09d760b965 by Benjamin Peterson in branch '3.4':
fix parsing reST with code or code-block directives (closes #23063)
https://hg.python.org/cpython/rev/db09d760b965

New changeset 38826e21f0db by Benjamin Peterson in branch '2.7':
fix parsing reST with code or code-block directives (closes #23063)
https://hg.python.org/cpython/rev/38826e21f0db

New changeset 731a36c13629 by Benjamin Peterson in branch 'default':
merge 3.4 (#23063)
https://hg.python.org/cpython/rev/731a36c13629
History
Date User Action Args
2016-12-15 16:08:02positasetnosy: + posita
2015-01-15 05:00:39python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg234051

resolution: fixed
stage: commit review -> resolved
2014-12-19 17:34:27merwoksetmessages: + msg232943
stage: patch review -> commit review
2014-12-16 18:41:20Marc.Abramowitzsetmessages: + msg232756
2014-12-16 18:18:49merwoksetmessages: + msg232754
2014-12-16 18:17:42Marc.Abramowitzsetfiles: + issue23063.patch
2014-12-16 18:16:50merwoksetfiles: + 9b8f6812ff69.diff
2014-12-16 18:16:31merwoksethgrepos: - hgrepo286
2014-12-16 18:16:17merwoksethgrepos: + hgrepo288
2014-12-16 18:14:48Marc.Abramowitzsetmessages: + msg232753
2014-12-16 11:55:49berker.peksagsetnosy: + berker.peksag
messages: + msg232734
2014-12-16 11:53:23berker.peksagsetstage: patch review
versions: - Python 3.2, Python 3.3, Python 3.6
2014-12-16 11:52:40berker.peksagsetfiles: + cfabe07bc98f.diff
keywords: + patch
2014-12-16 08:14:05Marc.Abramowitzsetmessages: + msg232722
2014-12-16 08:09:59Marc.Abramowitzsetmessages: + msg232721
2014-12-16 07:58:05Marc.Abramowitzcreate