I just had a problem with doctests. It manifested as an error that occurred when I did
>>> import StringIO
in my doctest. 

After some investigation, I realised that my actual problem was that the doctest library documentation https://docs.python.org/3/library/doctest.html left me unclear about three aspects of multi-line doctest fixtures.

1. There is no example of a multiple-line doctest fixture: specifically, a fixture which involves at least one line to set up state, and another line to be the example that tested.  I suggest adding such an example as a new section, "26.3.2. Usage: multi-line tests".  Note that the example in 26.3.3.2 doesn't count: that is a Compound Statement, and I'm talking about a test consisting of multiple interactive statements.

2. Section "26.3.3.2. How are Docstring Examples Recognized?" does not talk about when to use a PS1 prefix (>>>) and when to use a PS2 prefix (...). It should say to use the PS1 at the start of each Python "interactive statement", that is, a statement list ending with a newline, or a Compound Statement. And, to use the PS2 to prefix each continuing line within a Compound Statement. 

3. Section "26.3.3.3. What’s the Execution Context?" is not clear about the crucial difference between state that accumulates within a single docstring from Example to Example, and that is reset from one docstring to another. The phrase "so that one test in M can’t leave behind crumbs that accidentally allow another test to work" can be interpreted as applying within a single docstring. The phrase "names defined earlier in the docstring being run" is easy to miss.

A StackOverflow Question and Answer describing my problem and my insight about what the library documentation didn't tell me is at http://stackoverflow.com/questions/41070547/why-is-importing-a-module-breaking-my-doctest-python-2-7 . A blog post of mine, a source for documentation improvements, is at http://blog.jdlh.com/en/2017/01/31/multi-line-doctests/ .

I actually encountered this problem in a Python 2.7 context, with https://docs.python.org/2/library/doctest.html . But it also applies to Python 3.6 https://docs.python.org/3/library/doctest.html and to the current dev https://docs.python.org/dev/library/doctest.html . In the spirit of fixing the new stuff first, I am filing this issue against Python 3.7. 

Unless someone tells me otherwise, I'll get around to filing similar issues against Python 3.6 and Python 2.7.
Then it's my intention to write improvements to the documentation for Python 3.7, 3.6, and 2.7, and submit each of those. It looks like they will be pretty similar, because the library documentation is pretty similar in each version.

I'm new to Python issue filing and doc fixing, so I would appreciate correction if (when?) I start to blunder.

> I just had a problem with doctests. It manifested as an
> error that occurred when I did
> >>> import StringIO in my doctest.

Maybe you are running the doctest with Python 3. The StringIO module is no more available in Python 3, so your doctest will fail on Python 3 and will pass on Python 2.

> 1. There is no example of a multiple-line doctest fixture:
> specifically, a fixture which involves at least one line to
> set up state, and another line to be the example that tested.
> I suggest adding such an example as a new section, "26.3.2.
> Usage: multi-line tests".
> Note that the example in 26.3.3.2 doesn't count: that is a
> Compound Statement, and I'm talking about a test consisting
> of multiple interactive statements.

The example in section 26.3.3.2 [1], before the compound statement (the if/else), contains two simple statements. I also think the example is complete and straightforward for its purpose, and it does not have to be complicated adding avanced features. If you want a more complex control over your tests, for instance adding fixtures, you can find how to do it later, in [2].

> 2. Section "26.3.3.2. How are Docstring Examples Recognized?" does
> not talk about when to use a PS1 prefix (>>>) and when to use a
> PS2 prefix (...). It should say to use the PS1 at the start of each
> Python "interactive statement", that is, a statement list ending with
> a newline, or a Compound Statement. And, to use the PS2 to prefix each
> continuing line within a Compound Statement. 

I am -1 on this. IMHO there is no need to add extra description to a such self-explaining, clear and concise example. But this I think is also clear from the beginning. Infact, the first line of the doc says: "The doctest module searches for pieces of text that look like interactive Python sessions"

> 3. Section "26.3.3.3. What’s the Execution Context?" is not
> clear about the crucial difference between state that accumulates
> within a single docstring from Example to Example, and that is reset
> from one docstring to another. The phrase "so that one test in M can’t
> leave behind crumbs that accidentally allow another test to work"
> can be interpreted as applying within a single docstring. The phrase
> "names defined earlier in the docstring being run" is easy to miss.

Also this section IMO is well written and clear. I do not know what to say... If someone else thinks it is not enough understandable, maybe the better think to do is to add a very small example, with two functions, that shows their docstrings have independent globals.

> Unless someone tells me otherwise, I'll get around to filing similar
> issues against Python 3.6 and Python 2.7.
> Then it's my intention to write improvements to the documentation
> for Python 3.7, 3.6, and 2.7, and submit each of those.

Keep in mind that writing a patch is not enough. The patch has to be reviewed, and that means other people have to find the time to work on it. I suggest you to watch this talk of Raymond Hettinger, before going on:

https://www.youtube.com/watch?v=voXVTjwnn-U


[1] https://docs.python.org/3/library/doctest.html#how-are-docstring-examples-recognized
[2] https://docs.python.org/3/library/doctest.html#unittest-api

It does appear that "test" is being used ambiguously in the docs.  In most places it means a single statement, but in execution context it appears to be being used as a synonym for "docstring".  In that section it should be replaced by "docstring".  It would probably be worth checking the rest of the doc to make sure it is used consistently elsewhere as well.

As for the prompts, I agree with Marco: I don't think there is any need to add words, since it clearly says it is supposed to look like an interactive session and the reader gets reminded of the rules for that by the example provided.  IMO it is not appropriate to re-document those rules here.  However, if you come up with a really concise addition that clarifies it, we'll consider it.

I don't think an additional example of using previous state is required, and certainly not an additional section.  The execution context section makes it clear that you can use names defined earlier, as does the fact that doctests emulate interactive sessions.  Both of these make it clear you can do multi line examples that use shared state.  However, if you come up with an example that adds value to the documentation, we'll certainly consider it.

So in summary, I think we *need* a one word change in the execution context section, and we'll consider any other enhancements you want to propose.

And thanks for wanting to improve the docs!

Marco, thank you for your prompt comment. 

However, 
> Maybe you are running the doctest with Python 3. The StringIO module is no more available in Python 3, so your doctest will fail on Python 3 ....

Please let me be clear. This issue is not about the StringIO module or about any of my doctests. It is about the documentation of the doctests module, and the ways in which that documentation is less helpful than it could be.

Marco, thank you for the suggestion to watch Raymond Hettinger's talk. 

> I suggest you to watch this talk of Raymond Hettinger, before going on:
>
> https://www.youtube.com/watch?v=voXVTjwnn-U

That video is 63 minutes long, and a lot of those minutes are taken up by stories. I will watch the rest when I get time. I wish his insights were summarised in writing; I'll bet that would take much less than 63 minutes to read. I find that written text is a much faster way to deliver information, though maybe it takes the teacher more time to write well than to give a talk.

I understand your point about the need for a review, and reviews being the bottleneck. The "Lifecycle of a Patch" document[1] is clear about that.

[1] https://docs.python.org/devguide/patch.html#reviewing

Marco and David, thank you again for your prompt replies. 

Let me respond to both of your comments on the doctest module documentation itself.


[Marco] > The example in section 26.3.3.2 [1], before the compound statement (the if/else), contains two simple statements. I also think the example is complete and straightforward for its purpose, and it does not have to be complicated adding avanced features.

You are correct, it contain two simple statements. I missed this when I first read the example.  Part of the problem is that the example is contrived. It is an actual test. I think it would be better to replace these two lines by a two-line test, one which sets up state and another which exercises the module under test.

You believe "the example is complete and straightforward for its purpose, and it does not have to be complicated adding avanced features." I am giving you feedback that it was not complete and straightforward for me, as an ordinary software developer trying to use the documentation. 


[Marco] > If you want a more complex control over your tests, for instance adding fixtures, you can find how to do it later...

No, control over tests is not what I am bringing up in this issue. I an concerned with documentation of doctest "Examples".


[Marco] > I am -1 on this [changing 26.3.3.2]. IMHO there is no need to add extra description to a such self-explaining, clear and concise example. But this I think is also clear from the beginning. Infact, the first line of the doc says: "The doctest module searches for pieces of text that look like interactive Python sessions"

[David] > As for the prompts, I agree with Marco: I don't think there is any need to add words, since it clearly says it is supposed to look like an interactive session and the reader gets reminded of the rules for that by the example provided.  IMO it is not appropriate to re-document those rules here.


My feedback to you is that for me, as an ordinary software developer trying to use the doctest module, this section is not "self-explaining, clear and concise", and that "How Docstring examples are recognised" is not "clear from the beginning". 

Perhaps we have a difference of opinion on how clear and effective this documentation is.  The best way to resolve that would be to gather data: having several ordinary software developers reading the document, then measuring their comprehension. That would be hard to arrange. At least this bug report will serve as a data point.

I believe that a library module documentation should at some point clearly specify how the module works. I believe section Section "26.3.3.2. How are Docstring Examples Recognized?" does poorly at that. To say, "In most cases a copy-and-paste of an interactive console session works fine, but doctest isn’t trying to do an exact emulation of any specific Python shell." is not a specification. It's a blurry generalisation. The list of "fine print" is at a better level of detail, but still doesn't mention how the parser treats statements, nor how prefixes relate to statement boundaries.

I am giving you feedback that for me, as a developer trying to use the doctest module, this documentation failed to give me the comprehension I needed to be effective.


[Marco] >Also this section [26.3.3.3, execution context] IMO is well written and clear. I do not know what to say... If someone else thinks it is not enough understandable, maybe the better think to do is to add a very small example, with two functions, that shows their docstrings have independent globals.

[David] > It does appear that "test" is being used ambiguously in the docs.  In most places it means a single statement, but in execution context it appears to be being used as a synonym for "docstring".  In that section it should be replaced by "docstring".  It would probably be worth checking the rest of the doc to make sure it is used consistently elsewhere as well.

[David] > The execution context section makes it clear that you can use names defined earlier, as does the fact that doctests emulate interactive sessions.  Both of these make it clear you can do multi line examples that use shared state. 

Again, I am giving you feedback that this document was not "clear" for me. I am taking the time to write this issue because I think the documentation has weaknesses, and would like to help improve it.

I agree with David, that one problem in this section is that the word "test" is used with three different meanings. Some instances of the word "test" should be changed to "docstring" or similar. That would help. 


[David] > So in summary, I think we *need* a one word change in the execution context section, and we'll consider any other enhancements you want to propose.

All right, I'll work on a patch that makes that one-word change, and offers some other concise improvements. I expect it will be more that you think necessary, given that we disagree on how "clear" the current documentation is.

Given that reviews are a bottleneck, I fear my patch may grind to a halt waiting for review. But this is my first contribution to Python. We'll see. I clearly have a lot to learn.


[David] > And thanks for wanting to improve the docs!

You are welcome!

Thank you again for your prompt responses. And thank you for all your work to build this marvelous Python language, which helps so many.


[1] https://docs.python.org/3/library/doctest.html#how-are-docstring-examples-recognized

IMO if you would like to apply big changes to the current doc, as you wrote before, maybe is worth considering to propose a separate howto, instead of wide changes to the current page. I am suggesting this to you, because looking at the other modules of the standard library, I see they just explain the API, extending with examples when needed. When is worth adding extra documentation, in a different style, I see the usual solution is to extend with an howto. Look at the argparse module and related howto, for instance:

https://docs.python.org/3/library/argparse.html
https://docs.python.org/3/howto/argparse.html

I also think an howto in this case is appropriate, because there is more to say than only the API. If you want to follow the howto way, I would like to contribute. In any case, I am disposed to review your patch

Marco, thank you for the suggestion of a howto. That is a good idea. 

In parallel, I was thinking of some howto content that I would like to see documented. 

• How to decide what to test with doctests and what with unittests. I have a feeling that the sweet spot of doctests is functionality which you can invoke in one line, and which produces one line of output which you can match against a reference string. If it takes many lines to set up, or invoke, or checking for correctness is more than a test for string equality, then maybe unittest is a better tool.

• An effective way to write doctests. I suspect that some people write doctests by exercising the functionality in Python's interactive mode, then copy and paste the transcript from that session to the doctests. I don't do it that way, I author in the docstring. Maybe I'm not doing it the best way.

• How to author doctests so that they both prove the module correct, and provide clear and readable documentation. I imagine some effective tests for edge cases and error conditions are important to have, but are not readable documentation. Maybe such tests belong in a unittest framework instead of as doctests.

• How to run doctests from a unittest harness (your earlier note about the unittest API would be a part of this).

A problem for me is that I think I don't have the experience and wisdom to give good advice in these areas. I would be happy to start such a howto, and to accept feedback, and to edit it into good prose. I would need wiser people to contribute good ideas for the howto.

Also, Marco, thank you for being willing to review a patch.  That is helpful.

My next step is to check out the documentation source, to be in a position to make a patch.

> In parallel, I was thinking of some howto content that I would like to
> see documented.

Great :-)

> How to decide what to test with doctests and what with unittests.
> I have a feeling that the sweet spot of doctests is functionality
> which you can invoke in one line, and which produces one line of
> output which you can match against a reference string. If it takes
> many lines to set up, or invoke, or checking for correctness is
> more than a test for string equality, then maybe unittest is a
> better tool.

All your questions come up every time I speak about doctest, and these doubts are the main reason for my previous sentence: "there is more to say [about doctest] than only the API". That is why IMO an howto could be really helpful. What doctest checks is not one line of output, but the output as a string, and of course it could have multiple lines. Try to execute the doctest (with the -v option) for this docstring:

def foo():
    """ 
    >>> a = 33
    >>> a
    33
    >>> for i in range(2):
    ...   print(i)
    ...
    0
    1
    """

Three tests will be executed: one for `a=33`, expecting nothing, one for `a`, expecting 33 as output, and one test for the whole `for` block, expecting one output composed of two lines.

For several reasons IMO doctest is not a substitute for unittest. They are different tools for different purposes, as thier name point up. To have a wide view, usually it is a good idea to start from the beginning:

https://groups.google.com/forum/#!msg/comp.lang.python/DfzH5Nrt05E/Yyd3s7fPVxwJ

Anyway, IMO this is not the right place to discuss and explain the objectives of doctest. I propose you to open a repository on github, work and discuss there, and when all is ready, we can ask for a review. Please write here the link to the github repo, so anyone else wants to contribute can find it. Thanks again for your time

Marco:

> To have a wide view, usually it is a good idea to start from the beginning:
> 
> https://groups.google.com/forum/#!msg/comp.lang.python/DfzH5Nrt05E/Yyd3s7fPVxwJ

Thank you, that is a very interesting thread. Clearly the creator of doctests, Tim Peters, is a well-regarded person in the Python world. 

On the other hand, I was surprised to see some of the examples used in the present-day documentation in Tim's original 1999 message. I find it remarkable that in 18 years since then, no rewrite or editing has replaced those examples with better ones.

I've drafted some fairly restricted changes to the doctest documentation page. They are in my Github branch, https://github.com/JDLH/cpython/tree/Issue29428_doctest_docs . The diffs are at https://github.com/JDLH/cpython/commit/223ef8f8a6d2fbec6db774912028abb4d2ff88b6 . 

There currently is no official Python github repot against which to make a pull request. In a few days, once it appears, I'll make a pull request.

In the meantime, I can take review comments and improve it. If you are interested, please review.

Pull Request https://github.com/python/cpython/pull/45 submitted to new Github repo. 

I would appreciate a review.

I attempted to balance all the different opinions in the discussion below: stay concise, but also improve the clarity.

Bump. Could I get a few more eyes looking at the current state of https://github.com/python/cpython/pull/45/ ? 

* @bitdancer made some suggestions. I accepted some, and replied to others where I think we should keep looking for common ground. I'd like to see replies.

* I've made changes in response to other comments.

I think pull/45 represents a targeted set of changes to those parts of the module documentation which are particularly unclear. It is not a wholesale re-write; that's a task for a different time. I believe it is responsive to comments so far. I'd appreciate more eyes to help arrive at consensus on those improvements where we can. 

Thank you!

I've closed the associated PR https://github.com/python/cpython/pull/45 with a recommendation to resubmit a smaller, more atomic PR if desired.