This issue is to start running (at least some of) the doctests in the Doc/ folder's *.rst files as part of regrtest.

We will need a whitelist mechanism to tell regrtest which files to process since currently many doctests are not runnable (e.g. some kind of "doctest-safe" marker that can go in a file as suggested by Nick).

See also the discussion thread started here:

http://mail.python.org/pipermail/python-dev/2012-August/121304.html

Might it be simpler to run doctest over the rst file from the relevant unittest?  (Perhaps with help from test.support.)

One option for whitelisting files would be to subclass doctest.DocTestParser and pass an instance as the parser argument to doctest.DocFileSuite:

http://docs.python.org/dev/library/doctest.html#doctest.DocFileSuite

The custom parser could look for the per-file marker before falling back to the usual parsing logic.  This would give us the functionality we need without having to make changes to doctest.

> Might it be simpler to run doctest over the rst file from the relevant unittest?  (Perhaps with help from test.support.)

If I understand correctly, do you mean for example that Lib/test/test_textwrap.py could be responsible for loading the doctests in Doc/library/textwrap.rst?

One problem with this approach is that the *.rst files are not in one-to-one correspondence with unittest files in Lib/test, etc.  For example, there are folders of *.rst files that may not correspond to any unit test files (e.g. the tutorial folder).  It also introduces more coupling because the unit tests would now have to know about the existence of the Doc/*.rst files (though this could be a nice addition as it would let you run the doctests for a module along with its unit tests if you wanted to run tests only for a particular module).

Or did you mean something else?

I'm not sure running these tests as part of regrtest is a good idea.
Fixing and/or marking the doctests is a good idea, and having a `make doctest` that reports no error is a desirable goal.  When `make doctest` works we could think about adding a regrtest option to run the doctests too, and if these tests don't take too much extra time to run we might even consider to run them by default (or at least when -uall is specified).
Having an option to run individual doctests would be nice too, and would encourage developers to test their doctests.
You should also see if there are more doctest-safe or more doctest-unsafe tests, and mark the ones in the smallest group.

> I'm not sure running these tests as part of regrtest is a good idea.

Can you provide reasons?

> If I understand correctly, do you mean for example that 
> Lib/test/test_textwrap.py could be responsible for loading
> the doctests in Doc/library/textwrap.rst?

One advantage of this is that I could explicitly add a test in test_textwrap.py that loads the doctests in textwrap.rst and runs them.
This would allow me to decide if those tests should be run or not, and which ones are runnable.  If I know that textwrap.rst has no runnable tests I won't add the test, so by simply running test_textwrap I would be running all the tests I want/need to run.

OTOH this has disadvantages too.  If the tests are loaded explicitly it means that every test file that wants to load the doctests should be modified.  Even if foo.rst has no runnable doctests and no test to load them is added to test_foo.py, a `make doctest` might still include the file unless we mark it in some way (either by a module-wide directive, or by marking all the individual snippets of code).

> Can you provide reasons?

Regrtest takes already a few minutes to run, adding these tests will make it even slower and the tradeoff might not be worthwhile (I'm assuming that most of the doctests are basic examples that are already well tested, and the occasional typos can be found by simply run `make doctest` every once in a while).
The tests currently have many failures and output, so adding them now will make the regrtest output more confusing.  Integrating this with regrtest requires some work, for example you would have to count successes/failures, add skips for certain tests and resources to control what tests should be run (there's currently a turtle going around on my screen when I run `make doctest`).  This will make regrtest even more complex.
People might not care to run these doctests if they are not touching the docs, and if they are, they might prefer to check only the specific doc file they changed.

One possibility to avoid tons and tons of failures that have to be fixed is to convert working doctests to the explicit syntax

.. doctest::

   blah

and set doctest_test_doctest_blocks to False in the Sphinx conf.py.

> the occasional typos can be found by simply run `make doctest` every once 
> in a while).

But doesn't "make doctest" attempt to run the doctests using Python 2.x (because Sphinx does not support Python 3.x).

An example from the output of "make doctest" which fails because Python 2.x is being used:

**********************************************************************
File "library/multiprocessing.rst", line 453, in default
Failed example:
    print(p, p.is_alive())
Expected:
    <Process(Process-1, initial)> False
Got:
    (<Process(Process-1, initial)>, False)
**********************************************************************

Yeah, once we switch around to using the build Python for this...

Using Python 3 to run Sphinx is tracked in #10224.  I thought there was already an issue to make the doctests runnable but can't find it, maybe it was only a discussion on a mailing list a year or two ago.

The point of running the doctests in the docs to to make sure the *docs* are correct, not to make sure Python is correct.  So IMO adding them to regrtest is much more trouble than the tiny benefit it would produce for test coverage.  On the other hand, having 'make doctest' run when '-uall' is specified would be cool, if the tests can thus be run by the buildbots.

But I think should be 'make doctest', and not regrtest using doctest to process .rst files.  The reason is that Sphinx has extra facilities that allow docttests to work without having to have "boilerplate" code in places where doing so would disrupt the narrative flow.  I could be mis-remembering (and Georg will correct me if I am misremembering again :), but I don't think those work when doctest is used directly against the .rst file.

Another idea would be to make a notation that looks exactly* like doctests for documentation purposes, but that doctest would not run.  Then, non-runnable doctests could be skipped, and runnable ones could be run. This would help keep the runnable code in the documentation validated, but would be on a test-by-test basis, rather than file by file.

That's exactly what Georg's suggestion is about.  Sphinx does have a way to mark doctest snippets as "run this", "don't run this".  I believe that requires using 'make doctest' as the runner, but I already think that is the way to go, as I said before.

I think implementing this is very important, and +1 to Georg's suggestion, because no one is suddenly going to convert KLOCs of code samples to be testable (many code samples are partial, and will need to be completed in one way or another to be actually runnable).

However, do I understand correctly that this is blocked on #10224?

I would like the chance to try implementing this without depending on Sphinx and "make doctest".  I don't think it would take much work, and it would let us leverage and hook into regrtest's existing options (like test selection by name).  I am also okay with exposing this via an option rather than including the doctests by default, which seemed to be the main concern expressed by others.

On Windows, .rst files are not part of the install, so running doctests on .rst files cannot be a required part of the standard regrtest.

Attached is a patch that adds to regrtest the ability to run doctests in both library modules and documentation files (specifically, in the non-test modules in Lib/ and in the *.rst files in Doc/).

The syntax to run all doctests is--

    python -m test --doctests --docdir <path-to-doc>

If --docdir is left out, only module doctests are run.  You can run the doctests for individual files and modules by including test names of the form "fdoc:library/builtins.rst" and "mdoc:html.parser" (for file doctests and module doctests, respectively).

Doctests for a module are only run if the module has a variable `_doctest_safe` that evaluates to True.

I haven't yet added a mechanism to mark a documentation file as "doctest safe" or to skip importing a library module entirely (when running all tests).

I am attaching an updated version of the patch for feedback.

The latest version of the patch I developed and used in the real-world example of fixing both the module and documentation file doctests for the ipaddress module (and its HOWTO) in issue 15888.

The patch includes the following additional features:

(1) the user is warned if a user-selected test file contains no tests
(2) tests to run can be specified using one or more globs (e.g. fdoc:howto*)
(3) the doc directory can be specified relatively (e.g. --docdir Doc)
(4) passing --docall adds all doctests to the test run selection
(5) passing --docunsafe runs doctests for all selected tests, whether or not the test files are marked "doctest safe"

The patch also includes usage documentation towards the beginning.

> You should also see if there are more doctest-safe or more doctest-unsafe tests, and mark the ones in the smallest group.

Running the doctests for all .rst files in the Doc/ directory using the latest patch gave me the following:

299 tests [files] OK.
126 tests failed
(and 6 that I manually skipped)

Following the suggestion above from Ezio, we should have a doc file marker to skip a file as opposed to including a file.  This means that doctests would be included by default, which I think would be a good way to go.  How do people feel about something like the following for a marker?

.. doctest-skip-file (to distinguish from skipping an individual test)

Or perhaps (to go with something like Nick's original suggestion)

.. doctest-unsafe

> Integrating this with regrtest requires some work, for example you would have to count successes/failures, add skips for certain tests and resources to control what tests should be run (there's currently a turtle going around on my screen when I run `make doctest`).

Fortunately, we get a lot of this for free when integrating with regrtest because the infrastructure is already there for the other tests.  For example, the test counts for my previous comment were automatic, operating in a clean temp cwd is something else taken care of, a command-line interface is already present, etc.

The main doctest-specific things that needed to be added were (1) doctest discovery (which involves walking the Lib/ and Doc/ directory hierarchies), and (2) creating TestSuite instances differently (by invoking DocTestSuite and DocFileSuite as opposed to calling TestLoader.loadTestsFromModule()).

I would still like to decouple the code a bit more and perhaps even add some unit tests for some of the newly added code.

> Sphinx does have a way to mark doctest snippets as "run this", "don't run this".  I believe that requires using 'make doctest' as the runner,

The doctest module supports this natively (i.e. without the need for Sphinx) via its "skip" directive ("#doctest: +SKIP"):

http://docs.python.org/dev/library/doctest.html#doctest.SKIP

Sphinx will even hide doctest directives like this when rendering the HTML, so the additional annotation won't be visible to the end user/reader and clutter up the interactive examples (cf. issue 12947).

Also, it looks like the Sphinx "doctest" directive gets most or all of its functionality via the options already exposed by doctest:

http://sphinx.pocoo.org/ext/doctest.html?#directive-doctest

> The reason is that Sphinx has extra facilities that allow docttests to work without having to have "boilerplate" code in places where doing so would disrupt the narrative flow.

Yes, this seems to be true.  Sphinx has a "testsetup" directive which lets you avoid adding boilerplate to interactive examples:

http://sphinx.pocoo.org/ext/doctest.html?#directive-testsetup

I did a quick search, though, and it looks like this is used in only 7 documentation files out of the 430+ -- one of which is library/turtle.rst. :)

Similarly, the "testcode" directive (a variant of the doctest format) seems to be used in just 5 files.  In any case, we could still benefit from running the doctests from regrtest that don't require use of testsetup.

turtle uses it because that was the file that I made work when I was playing with 'make doctest'.  I think being able to use the testsetup directive would be good.  It could also them be used (I think!) to put resource directives in the docs that would control whether or not (eg) turtle was run via regertest '-u', making it not run by default.  What make doctest does is to write the doctests out to to a file and then run them with the normal doctest tools, so there ought to be a way to integrate with it.  I guess how easy that is depends on how easy it is to run sphinx on just one file...which might not be easy.

Attached is a file of doctest statistics (counts of example failures, exceptions, and successes) for almost every file in the Doc directory when running the doctests with vanilla doctest.

I did this to get a sense of which files it would be easiest to get into a "passing" state, and to see how many files actually have doctest examples.

It was actually non-trivial to get fine-grained test result data like this because doctest only exposes the ability to create a unittest.TestCase for each file (i.e. an entire file either registers as a success or failure).  From the API, you can't get results down to the level of individual doctest examples.

See also issue 15938 that I recently created which would be a small step in this direction (getting a total example count independent of successes and failures).

We discussed about this on IRC, and I'll try to summarize the outcomes.

If we want devs to run doctests, the syntax should be as simple as possible, ideally there shouldn't be any extra syntax -- at least for the common case.

Note that there are two kind of doctests:
  * the ones in the *.rst files in the Doc dir;
  * the ones in the docstrings of the *.py files in the Lib dir;

Some of the use cases are:
  1) being able to run all the doctests at once;
  2) being able to run individual doctests files;
  3) being able to discover/run doctests in a specified dir;

The use case 1) likely needs an extra flag (e.g. --docall), and could be done by default with -uall.

The use case 2) can be already done with `./python -m doctest path/to/doctest.rst`,  but, if doctest support is added to regrtest, it would be nice to have this too.
The following issues were discussed:
  * should regrtest accept them with no extra syntax (e.g. `./python -m test path/to/doctest.rst` -- this would be nice but probably tricky/hackish) or should they marked somehow (e.g. `fdoc:path/to/doctest.rst`, or `--doctests path/to/doctest1.rst path/to/doctest2.rst`)?
  * should we use paths (e.g. Doc/library/textwrap.rst -- easier) or dotted names (e.g. Doc.library.textwrap -- more consistent with unittests)?
  * should we require the full path/name, or just the filename like we do with regular unittests (e.g. `-m test test_textwrap`)?
  * should we use the fact that Doc/Lib or rst/py are in the name to distinguish the doctests, or can we search for the name in Lib/test/ first, and fallback on Doc/ and Lib/ if it's not there?

The use case 3) is less common, it can be provided with an additional flag but it's not strictly necessary IMHO (might be handy when we want to run all the doctests in a package).

Other issues that we haven't discussed are:
  * it would be nice to have something like run_doctests(['path/to/doctest.rst', 'path/to/module.py']) to be added in the test_module and be called automatically by `-m test test_module`;
  * which way should be used to mark the files/snippets as doctest-compatible;
  * how to have a doctest equivalent to setUp/tearDown (this is optional, we can always skip the test if it fails without initialization);

(FTR: I was trying the patch on 3.2 and since there were a few conflicts I uploaded the resulting patch in case anyone needs it -- this doesn't mean it should go on 3.2 though.)

I think I'm -1 to run doctests as part of regrtest. Documentation examples are meant to be read, not executed: it is fine for them to omit unimportant information or trivial boilerplate (such as imports).

Any complication in the way doc examples must be written will deter us from actually writing examples.

Furthermore, spreading unit tests in the doc files instead of Lib/test files will make maintenance of tests more tedious as well.

> Documentation examples are meant to be read, not executed

I agree, but several of them already run, and others run with the addition of doctests directives (that are hidden in the html output).  If the code needs to be changed in a way that is less readable, the test can just be skipped.

> Furthermore, spreading unit tests in the doc files instead of Lib/test
> files will make maintenance of tests more tedious as well.

Tests should be kept in Lib/test, this is mostly to make sure that the examples are correct.  The examples in the docs should still be written with the goal to be clear for the user -- being valid doctests is a secondary goal.

I've given this some more thought, and I'm leaning toward Antoine's POV here.

The point of running the doctests in the docs is not to test python, but to test the docs.  Sphinx has a facility to do that: make doctest.  So I think it is better to use that doc-specific infrastructure to test the docs.  This also allows us to use the full Sphinx facilities to work with those doctests, which includes hiding the not-necessary-for-docs bits.

I think we should be moving *away* from having special infrastructure in regrtest.  As much stuff as makes sense should be moved to unittest, and we've been slowly doing that.  Correspondingly, we should use Sphinx's native test facilities, not add special stuff to regrtest.  If Sphinx doesn't have the ability to run individual files, we should add that ability to Sphinx, not regrtest.  (Note: I personally do not use the ability recently added to regrtest to select unit tests from the command line, instead I use the unittest CLI directly, and I think that's the better way to do it.  IMO regrtest should be focused on running the test *suite*, not on running individual tests.)

Once we've refined the docs so that 'make doctest' runs cleanly, we should automate the testing of the docs in some fashion.  One way to do that would be to define a test module that handles the interfacing with Sphinx, and is controlled by a resource that is *not* included in -uall, and is then explicitly enabled on certain buildbots.

Doctests in module files could be handled the same way: build a test module that does whatever autodiscovery is needed, and control it via a resource not included in -uall.  (I am, however, open to the argument that this one should be included in -uall.)

Again, there already exists a way to invoke individual files, using the native doctest facilities.  So adding hooks to regrtest is not really needed, IMO.  Also again, if the doctest CLI is lacking, the improvement should go into doctest.

IMO :)

> I think we should be moving *away* from having special infrastructure
> in regrtest.  As much stuff as makes sense should be moved to
> unittest, and we've been slowly doing that.  Correspondingly, we
> should use Sphinx's native test facilities, not add special stuff to
> regrtest.  If Sphinx doesn't have the ability to run individual files,
> we should add that ability to Sphinx, not regrtest.  (Note: I
> personally do not use the ability recently added to regrtest to select
> unit tests from the command line, instead I use the unittest CLI
> directly, and I think that's the better way to do it.  IMO regrtest
> should be focused on running the test *suite*, not on running
> individual tests.)

The main reason to add it to regrtest was to allow special test modes
with it (such as -R or -F). (and, also, the unittest CLI's poor online
help makes it rather unusable for me :-)).
But I agree on the principle that unittest should be expanded to better
accomodate the needs of regrtest.

RDM’s arguments make a lot of sense.

We should improve the unittest help, then :)  I will also note that the couple of times I tried it I couldn't figure out how to use the regrtest test selection :)

What we probably need for regrtest is a way to pass through a "selection string" from the regrtest command line to the unittest test discovery, and then run the selected tests under regrtest control.  The same could apply to Sphinx doctests and module doctests.  This may require adding APIs to those packages (unittest may already have the necessary API).  

I think the Sphinx case is the one that will take the most work to accomplish, but I also think it is worth the effort to do that rather than adding CPython-specific stuff to regrtest.

I should mention that http://bugs.python.org/issue10967 exists as a meta issue about trying to gut regrtest in favour of using other things in unittest (and doctest in this case) when possible.

It appears, especially from the later messages, that this is not wanted.  However noting msg170616 should we close this with #10967 to supersede it?

The issue has been fixed by bpo-35240.

commit a22df4896f6b83c8741203118790ae281716bca5
Author: Victor Stinner <vstinner@redhat.com>
Date:   Wed Nov 28 10:24:08 2018 +0100

    bpo-35240: Add "doctest" job to Travis CI (GH-10753)
    
    Create a new "doctest" job in Travis CI to run "make doctest".