classification
Title: regrtest: always run tests in verbose mode, but hide the output on success
Type: Stage:
Components: Tests Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eric.araujo, ezio.melotti, georg.brandl, ned.deily, pitrou, python-dev, r.david.murray, vstinner
Priority: normal Keywords: patch

Created on 2011-06-24 20:45 by vstinner, last changed 2011-07-01 13:40 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
regrtest_verbose.patch vstinner, 2011-06-24 20:45 review
regrtest_verbose3.patch vstinner, 2011-06-27 23:12 review
Messages (26)
msg138981 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-06-24 20:45
Sporadic issues are difficult to analyze on the buildbots because sometimes a test fails, but when it runs again in verbose mode... it doesn't fail anymore. Typical example:

===============================
[ 99/356/1] test_smtplib
Re-running test test_smtplib in verbose mode
(...)
testTimeoutValue (test.test_smtplib.GeneralTests) ... ok
testBasic (test.test_smtplib.DebuggingServerTests) ... test test_smtplib failed -- multiple errors occurred; run in verbose mode for details
ok
testHELP (test.test_smtplib.DebuggingServerTests) ... ok
testNOOP (test.test_smtplib.DebuggingServerTests) ... ok
testNotImplemented (test.test_smtplib.DebuggingServerTests) ... ok
(...)
----------------------------------------------------------------------
Ran 32 tests in 23.664s

OK
===============================

I propose to always run the tests in verbose mode, write the output into a buffer, but only display the output on failure (or ignore the output on success).

Attached patch simplifies regrtest.py usage: replace 4 verbose levels (no verbose option, -v, -w, -W) by only 2 (no verbose option, -v). Always display the full output in case of an error (as we used -w before): no more "multiple error occured" without output trap, no more "oh, rerunning the test doesn't fail" trap. The -v option works as before (directly write the output in verbose mode).

regrtest is much more verbose than before in case of a failure, but I prefer too much information than no information when I try to debug something. If it is too much verbose by default, I can try to use 3 verbose levels (e.g. don't display the full output if only one test failed).

If the patch is accepted, the devguide should also be patched.
msg139251 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-06-27 09:40
> Typical example: (... smtplib ...)

Another example (yesterday):
------
[355/356/2] test_subprocess
...
Re-running test test_subprocess in verbose mode
...
Ran 228 tests in 322.313s

OK (skipped=20)
------
msg139336 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-06-27 23:12
New patch updated according to bitdancer's comment on IRC: only change the -W option. With the patch, -W only runs the test once but captures the output.

The main difference with my patch is that all output is written to stderr, even if the captured output was written into stdout. I don't want to capture stdout and stderr in two differents stream, I prefer to keep the original output order. In case of a failure, the output only concerns a failure.
msg139405 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-29 11:04
New changeset 949e0c0f5a38 by Victor Stinner in branch '3.2':
Issue #12400: regrtest -W doesn't rerun the tests twice anymore, but captures
http://hg.python.org/cpython/rev/949e0c0f5a38

New changeset 0172d40fdcd4 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: regrtest -W doesn't rerun the tests twice anymore,
http://hg.python.org/cpython/rev/0172d40fdcd4
msg139408 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-29 11:23
New changeset af799df478c6 by Victor Stinner in branch 'default':
Issue #12400: oops, remove debug code...
http://hg.python.org/cpython/rev/af799df478c6
msg139409 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-29 11:24
New changeset 69e102bfed2b by Victor Stinner in branch '3.2':
Issue #12400: remove unused variable
http://hg.python.org/cpython/rev/69e102bfed2b

New changeset 9d53612333c7 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: remove unused import
http://hg.python.org/cpython/rev/9d53612333c7
msg139410 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-06-29 11:34
I will not backport the change in Python 2.7, because regrtest already captures stdout (but not in verbose mode), and it checks that the output is empty.
msg139411 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-29 11:43
New changeset 80c7d63325f6 by Victor Stinner in branch 'default':
Issue #12400: fix test_faulthandler if regrtest captures sys.stderr
http://hg.python.org/cpython/rev/80c7d63325f6
msg139412 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-29 12:59
New changeset 74c6afbb524a by Victor Stinner in branch '3.2':
Issue #12400: don't use sys.stderr in test_kqueue because it may be replaced by
http://hg.python.org/cpython/rev/74c6afbb524a

New changeset c0afc8d00c0a by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: don't use sys.stderr in test_kqueue because it may be
http://hg.python.org/cpython/rev/c0afc8d00c0a
msg139414 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-29 13:26
New changeset 6c54c334ea7a by Victor Stinner in branch '3.2':
Issue #12400: runtest() reuses the same io.StringIO instance for all calls
http://hg.python.org/cpython/rev/6c54c334ea7a

New changeset 8897c755633b by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: runtest() reuses the same io.StringIO instance for
http://hg.python.org/cpython/rev/8897c755633b
msg139415 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-29 13:38
New changeset 2d07b1b4466a by Victor Stinner in branch '3.2':
Issue #12400: regrtest, force verbose mode to True with option -W
http://hg.python.org/cpython/rev/2d07b1b4466a

New changeset 561974b760eb by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: regrtest, force verbose mode to True with option -W
http://hg.python.org/cpython/rev/561974b760eb
msg139416 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-29 13:54
New changeset d8ba59de860a by Victor Stinner in branch '3.2':
Issue #12400: test.support.run_doctest() doesn't change sys.stdout anymore
http://hg.python.org/cpython/rev/d8ba59de860a

New changeset bb7c9308c846 by Victor Stinner in branch 'default':
Issue #12400: test.support.run_doctest() doesn't change sys.stdout anymore
http://hg.python.org/cpython/rev/bb7c9308c846
msg139420 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-06-29 14:25
Hmm.  I thought Antoine got rid of the output checking when he added -j support.  Well, he was the one who did it, anyway, so there may be a change set you could backport to get rid of the output checking if you want to backport this to 2.7.
msg139427 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-29 15:20
New changeset fdb98a7ef9be by Victor Stinner in branch '3.2':
Issue #12400: Add missing import (os) to test_kqueue
http://hg.python.org/cpython/rev/fdb98a7ef9be

New changeset 37877d19c2c9 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: Add missing import (os) to test_kqueue
http://hg.python.org/cpython/rev/37877d19c2c9
msg139429 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-29 15:30
New changeset 7bd23115b6c4 by Victor Stinner in branch '3.2':
Issue #12400: runtest() truncates the StringIO stream before a new test
http://hg.python.org/cpython/rev/7bd23115b6c4

New changeset fc831c49216d by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: runtest() truncates the StringIO stream before a new
http://hg.python.org/cpython/rev/fc831c49216d
msg139431 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-29 15:33
One more time, commit messages using the present tense are ambiguous: when you write “runtest() truncates the StringIO stream before a new test“, it’s not clear at all whether you describe the previous, incorrect behavior or the new, fixed one.  These phrasings are better:

“X used to do Y” (old behavior)
“X now does Z” (new)
“make X do Z” (new)
msg139433 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-06-29 15:40
> One more time, commit messages using the present tense are ambiguous:
> when you write “runtest() truncates the StringIO stream before a new
> test“, it’s not clear at all whether you describe the previous,
> incorrect behavior or the new, fixed one.

You can guess by reading the patch.
msg139434 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-29 15:42
It’s irrelevant, please provide unambiguous commit message.  This was discussed two or three times already.
msg139436 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-29 16:12
New changeset bc2f3fbda7e3 by Victor Stinner in branch '3.2':
Issue #12400: test_zipimport_support doesn't restore original sys.stdout
http://hg.python.org/cpython/rev/bc2f3fbda7e3

New changeset ff5abf93db80 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: test_zipimport_support doesn't restore original
http://hg.python.org/cpython/rev/ff5abf93db80
msg139438 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-06-29 16:48
Eric is right. I don't see the patch when reading "hg log" output, or looking up revisions given by "hg annotate".
msg139441 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-06-29 17:52
> Eric is right. I don't see the patch when reading "hg log" output

I'm using hg log -p.

> or looking up revisions given by "hg annotate".

hg annotate doesn't display the changelog, only the commit number or
hash, which option do you use?
msg139442 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-29 18:03
New changeset 450209efe272 by Victor Stinner in branch '3.2':
Issue #12400: regrtest.runtest() uses stream.seek(0) before .truncate()
http://hg.python.org/cpython/rev/450209efe272

New changeset 3ce22f1b9540 by Victor Stinner in branch 'default':
(merge 3.2) Issue #12400: regrtest.runtest() uses stream.seek(0) before .truncate()
http://hg.python.org/cpython/rev/3ce22f1b9540
msg139448 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-06-29 20:13
> Eric is right. I don't see the patch when reading "hg log" output

... or when looking at the issue tracker.  Yes, one can probably figure out the intent by looking at the whole patch but it's kind of rude to force people to do that.  That's what the commit message is there for: to give a brief, unambiguous summary of what the change is about.
msg139450 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-29 21:26
New changeset c7bed90e4f31 by Victor Stinner in branch 'default':
Issue #12400: test_faulthandler now uses sys.__stderr__
http://hg.python.org/cpython/rev/c7bed90e4f31
msg139453 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-29 22:00
New changeset 6edb7c153105 by Victor Stinner in branch 'default':
Issue #12400: test_cprofile now restores correctly the previous sys.stderr
http://hg.python.org/cpython/rev/6edb7c153105
msg139580 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-07-01 13:40
Ok, no more regression / bug related to this issue, let close it.
History
Date User Action Args
2011-07-01 13:40:22vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg139580

versions: - Python 2.7, Python 3.2
2011-06-29 22:00:59python-devsetmessages: + msg139453
2011-06-29 21:26:32python-devsetmessages: + msg139450
2011-06-29 20:13:06ned.deilysetnosy: + ned.deily
messages: + msg139448
2011-06-29 18:03:36python-devsetmessages: + msg139442
2011-06-29 17:52:16vstinnersetmessages: + msg139441
2011-06-29 16:48:32georg.brandlsetmessages: + msg139438
2011-06-29 16:12:59python-devsetmessages: + msg139436
2011-06-29 15:42:47eric.araujosetmessages: + msg139434
2011-06-29 15:40:55vstinnersetmessages: + msg139433
2011-06-29 15:33:52eric.araujosetnosy: + eric.araujo
messages: + msg139431
2011-06-29 15:30:20python-devsetmessages: + msg139429
2011-06-29 15:20:40python-devsetmessages: + msg139427
2011-06-29 14:25:45r.david.murraysetmessages: + msg139420
2011-06-29 13:54:07python-devsetmessages: + msg139416
2011-06-29 13:38:49python-devsetmessages: + msg139415
2011-06-29 13:26:02python-devsetmessages: + msg139414
2011-06-29 12:59:37python-devsetmessages: + msg139412
2011-06-29 11:43:49python-devsetmessages: + msg139411
2011-06-29 11:34:13vstinnersetmessages: + msg139410
2011-06-29 11:24:55python-devsetmessages: + msg139409
2011-06-29 11:23:39python-devsetmessages: + msg139408
2011-06-29 11:04:23python-devsetnosy: + python-dev
messages: + msg139405
2011-06-27 23:12:37vstinnersetfiles: + regrtest_verbose3.patch

messages: + msg139336
2011-06-27 09:40:15vstinnersetmessages: + msg139251
2011-06-24 20:45:58vstinnercreate