http://docs.python.org/py3k/library/base64.html?highlight=base64
the examples given doesn't work

The example can be fixed by placing a "b" before the two string literals.

However, pretty much the whole document refers to "strings" and should refer to "byte sequences" or the "bytes" type.

I thought there were automated tests that exercised the documentation examples.  Am I wrong about that?

See issue 4769, which would partially fix this problem if implemented correctly.  The docs should still be given a thorough review to use the appropriate bytes/string language.

There is a way to do automated testing of the code in the docs, but nobody has done the work to automate it yet.  Sphinx only recently got 3.x support, before which it wasn't even possible.  To run it by hand, do 'make doctest' in the Doc subdirectory of the checkout...the first step will be to fix all the failures....

PATCH
Specifically, in section 17.6. base64..., near bottom, example should be

>>> import base64
>>> encoded = base64.b64encode(b'data to be encoded') #hang
>>> encoded
b'ZGF0YSB0byBiZSBlbmNvZGVk'
>>> data = base64.b64decode(encoded)
>>> data
b'data to be encoded'

with the first and third 'b' prefixes added.

I confirmed that doctest works with above.

I am a bit puzzled about Sphinx and 3.x comment, as doctest just need a plain ascii file and does not care how it was produced. (I used browser Save As text file function.) However, moot now.

As an experiment, I ran doctest on 17.2 json saved as .txt, See #9767
4 failures, 2 obvious doc faults, 2 unclear to me.
Their were 2 similar doc faults in non-interactive code examples, so doctest is not enough to catch all bad code.

We clearly need to do this for the entire doc, preferably before 3.2 is released. A master issue is the wrong format, at least by itself. What I think is needed is a repository doc like Misc/maintainers.rst, call it Misc/doctests or Misc/docdoctests. It should have a line for each doc section with current status (blank for unchecked, n/a for no interactive example, issue number for fixes in progress). A master issue could then be a place where non-committers can report changes that committers could apply. What do you others think?

It seems there are three ways of testing the docs:

1) ./python -m doctest Doc/library/json.rst
2) make doctest (a.k.a. sphinx-build -b doctest)
3) http://sphinx.pocoo.org/ext/doctest.html

Manually running 1) or 2) and fixing things seems okay for a first step, and when everything is fixed, we could add automation to prevent regressions. Georg is in the best place to say how to do it (through a thin test_docs.py integration layer between doctest and regrtest, with sphinx.ext.doctest, or by editing the test target of the makefile to run make doctest -C Doc).

Note also that some docs (turtle) require running Tcl, which may be unwanted on headless machines like buildbots.

Generally: +1 on making sure examples in the docs are up to date.  If someone wants to do the tedious work of making sure that a "make doctest" succeeds, I'm all for it, it may involve adding a few (in HTML output invisible) testsetup blocks.

Eric: I'm not sure what the difference between your methods 2 and 3 is :)

As Terry already mentioned, by far not all example code is covered by that, and I don't think there's anything we can do about it.  We can of course try converting more doc examples to doctest format, however:

a) for longer examples and especially function/class definitions this really hurts readability and usability (think copy/paste)

b) many examples are also not easily runnable (Tk is a good example, many more can be found in those for network services)

> Generally: +1 on making sure examples in the docs are up to date.  If
> someone wants to do the tedious work of making sure that a "make
> doctest" succeeds, I'm all for it, it may involve adding a few (in
> HTML output invisible) testsetup blocks.

I'm not sure that's a good idea. It may add a lot of spurious imports
which only make the examples longer and less readable.

2) works without changing anything, 3) requires using specific directives IIUC.

>> Generally: +1 on making sure examples in the docs are up to date.  If
>> someone wants to do the tedious work of making sure that a "make
>> doctest" succeeds, I'm all for it, it may involve adding a few (in
>> HTML output invisible) testsetup blocks.
> 
> I'm not sure that's a good idea. It may add a lot of spurious imports
> which only make the examples longer and less readable.

That's why I said to use "testsetup" directives -- they are not visible in the HTML/PDF/... output, but used when running the tests.

> 2) works without changing anything, 3) requires using specific directives IIUC.

No.  The doctest extension is what "make doctest" calls.

Thanks for clearing this misunderstanding of mine.

I hope the trivial 2-byte fix does not get lost in the general issue.

Fixed in r85642.

That fixes the example code, but what about the numerous text that reads  "strings" that should read "byte sequences", "bytes", or similar?

I reviewed the doc and tightened up the wording (which was already mostly correct) in r85672.  Also fixed one typo and changed it to consistently use 'byte string' (rather than 'bytestring' which was used in one or two places).

On Fri, Sep 03, 2010 at 10:57:17PM +0000, Georg Brandl wrote:
> That's why I said to use "testsetup" directives -- they are not
> visible in the HTML/PDF/... output, but used when running the tests.

Do you already have such a directive in sphinx? I think, it would be a
good idea to have doctests succeed. And having examples in the docs
working 'directly out of text'.

Yes, Georg mentioned the directive because it exists :)

See the turtle docs for some examples, I think.  I seem to remember using it when I made those doctests pass on 2.7 (warning: it writes weird stuff on your screen :)