classification
Title: doctest: add cmdline parameters
Type: enhancement Stage: resolved
Components: Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: barry, ezio.melotti, haypo, python-dev, r.david.murray, techtonik
Priority: normal Keywords: patch

Created on 2011-03-03 20:33 by techtonik, last changed 2013-06-25 12:13 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
README.txt techtonik, 2011-03-03 20:33
doctest_cli.patch r.david.murray, 2013-06-20 15:41 review
doctest_cli.patch r.david.murray, 2013-06-22 15:30 review
doctest_term_dummy.patch haypo, 2013-06-25 00:14 review
Messages (16)
msg130002 - (view) Author: anatoly techtonik (techtonik) Date: 2011-03-03 20:33
doctest module need to parse -h/--help parameters, and accept flags like ELLIPSIS from command line. Otherwise it very hard to debug tests like contained in the attached README.

It is also worth to make it parameter compatible with unittest.main()

Usage: tests.py [options] [test] [...]

Options:
  -h, --help       Show this message
  -v, --verbose    Verbose output
  -q, --quiet      Minimal output
  -f, --failfast   Stop on first failure
  -c, --catch      Catch control-C and display results
  -b, --buffer     Buffer stdout and stderr during test runs
msg191521 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-06-20 14:45
Here is a patch that implements the ones I'm interested in (-o for any option, and -f as a shortcut for -o FAIL_FAST).  It's a good question whether doctest should support other unittest options, but doing so is more complicated than supporting -o and -f.

Adding a couple people to nosy who have an interest in testing and doctest, hoping for a patch review :)
msg191522 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-06-20 14:47
Oh, a documentation update is still needed here, so I guess I'm not really ready for review yet :(
msg191528 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-06-20 15:41
Patch updated with documentation.
msg191639 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-06-22 13:53
I will probably commit this tomorrow if there are no objections.
msg191645 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-06-22 15:30
Patch updated per Barry's review comments.
msg191716 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-23 18:24
New changeset ae802dc4dcd4 by R David Murray in branch 'default':
#11390: convert doctest CLI to argparse and add -o and -f options.
http://hg.python.org/cpython/rev/ae802dc4dcd4
msg191717 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-06-23 18:25
Committed.  Thanks for the review, Barry.
msg191824 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-06-24 23:59
The new tests added in changeset ae802dc4dcd4 are failing on some buildbots. I can reproduce the issue on my Linux box (Fedora 18). First failure of "./python -m test test_doctest":

[1/1] test_doctest
**********************************************************************
File "/home/haypo/prog/python/default/Lib/test/test_doctest.py", line 2627, in test.test_doctest.test_CLI
Failed example:
    rc1, out1, err1
Expected:
    (0, b'', b'')
Got:
    (0, b'\x1b[?1034h', b'')

The b'\x1b[?1034h' sequence it written by the readline module when it is imported. The readline module is imported by pdb, which is used by doctest.

The following article proposes set to TERM environment variable to "linux" to workaround the issue:
http://reinout.vanrees.org/weblog/2009/08/14/readline-invisible-character-hack.html

It works:

$ echo $TERM  # yeah, Fedora supports 256 colors!
xterm-256color
$ ./python -c 'import readline'|hexdump -C
00000000  1b 5b 3f 31 30 33 34 68                           |.[?1034h|
00000008
$ TERM=linux ./python -c 'import readline'|hexdump -C

This issue was reported to OpenSuse in 2009:
http://lists.opensuse.org/opensuse-bugs/2009-05/msg01633.html

The issue is not specific to Python. Just one another example with Octave (issue closed as "not a bug"):
https://bugzilla.redhat.com/show_bug.cgi?id=304181

The issue was also reported upstream (bug-readline mailing list) a few days ago:
http://lists.gnu.org/archive/html/bug-readline/2013-06/msg00000.html
msg191825 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-06-25 00:14
Attached doctest_term_dummy.patch: workaround the issue by setting TERM env var to "dummy".

Why is test_CLI() implemented using a doctest? Why not reusing the unittest module to benefit from test.support and test.regrtest? (Hint: I don't like doctests :-))
msg191826 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-06-25 00:19
test_doctest.test_CLI() is also failing on Windows:

http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%203.x/builds/1078/steps/test/logs/stdio

It looks like an issue with Windows newline (\r\n) versus UNIX newline (\n). test_CLI() decodes the bytes output from UTF-8, but do not normalize newlines.

The universal_newlines=True option can be passed to subprocess, which is not possible currently using script_helper. Is there function somewhere else to normalize newlines?
msg191843 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-06-25 10:28
It is implemented as a doctest because everything else in that test module is implemented as a doctest.  And clearly this reveals a bug of some sort.  Perhaps the dummy trick needs to be done by doctest itself?  (I will need to read your post over more carefully and understand the issue better.)

As for the windows failures, my apologies.  I knew I would need to check windows after the commit when I wrote the code, but I forgot to actually do it.
msg191844 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-06-25 10:46
> As for the windows failures, my apologies.  I knew I would need to check windows after the commit when I wrote the code, but I forgot to actually do it.

No problem, it's so easy to forget Windows :-) And we have buildbots
for this task (detect regressions).

2013/6/25 R. David Murray <report@bugs.python.org>:
>
> R. David Murray added the comment:
>
> It is implemented as a doctest because everything else in that test module is implemented as a doctest.  And clearly this reveals a bug of some sort.  Perhaps the dummy trick needs to be done by doctest itself?  (I will need to read your post over more carefully and understand the issue better.)
>
> As for the windows failures, my apologies.  I knew I would need to check windows after the commit when I wrote the code, but I forgot to actually do it.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue11390>
> _______________________________________
msg191846 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-06-25 10:50
> Perhaps the dummy trick needs to be done by doctest itself?

The issue really comes from the readline module. The workaround is to set an environment variable, which affects the whole process (all threads).

Only setting TERM for test_CLI() is fine, because it's the only test running doctest in a subprocess.

Hum, we may set TERM=dummy in script_helper directly, because this module tries to run Python in a fresh environment. It uses "python -E" for example.
msg191850 - (view) Author: Roundup Robot (python-dev) Date: 2013-06-25 12:11
New changeset 8f22e03f5f07 by R David Murray in branch 'default':
#11390: fix test failures due to readline and windows lineneds.
http://hg.python.org/cpython/rev/8f22e03f5f07
msg191851 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-06-25 12:13
Fixed for this test.  It would probably be worthwhile to improve script_helpers, I'll open a new issue for that.
History
Date User Action Args
2013-06-25 12:13:38r.david.murraysetstatus: open -> closed
resolution: fixed
messages: + msg191851
2013-06-25 12:11:45python-devsetmessages: + msg191850
2013-06-25 10:50:32hayposetmessages: + msg191846
2013-06-25 10:46:50hayposetmessages: + msg191844
2013-06-25 10:28:43r.david.murraysetmessages: + msg191843
2013-06-25 00:19:54hayposetmessages: + msg191826
2013-06-25 00:14:46hayposetfiles: + doctest_term_dummy.patch

messages: + msg191825
2013-06-24 23:59:36hayposetstatus: closed -> open

nosy: + haypo
messages: + msg191824

resolution: fixed -> (no value)
2013-06-23 18:25:14r.david.murraysetstatus: open -> closed
resolution: fixed
messages: + msg191717

stage: patch review -> resolved
2013-06-23 18:24:26python-devsetnosy: + python-dev
messages: + msg191716
2013-06-22 15:30:11r.david.murraysetfiles: + doctest_cli.patch

messages: + msg191645
2013-06-22 13:53:32r.david.murraysetmessages: + msg191639
2013-06-20 15:41:31r.david.murraysetfiles: - doctest_cli.patch
2013-06-20 15:41:09r.david.murraysetfiles: + doctest_cli.patch

messages: + msg191528
stage: needs patch -> patch review
2013-06-20 14:47:05r.david.murraysetmessages: + msg191522
stage: patch review -> needs patch
2013-06-20 14:45:30r.david.murraysetfiles: + doctest_cli.patch

versions: + Python 3.4, - Python 3.3
keywords: + patch
nosy: + barry, ezio.melotti, r.david.murray

messages: + msg191521
stage: needs patch -> patch review
2011-03-11 21:53:12eric.araujosetstage: needs patch
type: enhancement
versions: + Python 3.3, - Python 2.7, Python 3.2
2011-03-03 20:33:46techtonikcreate