classification
Title: Compact output for regrtest
Type: enhancement Stage: resolved
Components: Tests Versions: Python 3.5
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, berker.peksag, brett.cannon, chris.jerdonek, ezio.melotti, flox, ned.deily, pitrou, python-dev, r.david.murray, vstinner
Priority: normal Keywords: patch

Created on 2013-03-26 21:02 by ezio.melotti, last changed 2019-06-25 10:24 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
regr.diff ezio.melotti, 2013-03-26 21:02 Proof of concept against default. review
regroutput.txt ezio.melotti, 2013-03-26 21:02 Sample output.
patched_regrtest_output.txt ned.deily, 2013-03-29 07:21
issue17554-urlfetch.diff ezio.melotti, 2013-04-03 21:53 review
Messages (17)
msg185302 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-26 21:02
The attached patch is a proof of concept that changes the output of regrtest so that the "[xxx/yyy] test_zzzzzz" is updated in place instead of printing each test on a separate line.  Failures, skips, and the final report are printed normally, making it easier to find what failed without having to scroll for 400+ lines of output.
The patch works with and without -j, but I haven't tested it with other flags.

If people like the idea, this could be the default behavior, otherwise a new flag to enable the feature could be added (or it could be enabled with the already existing -q/--quiet).

Also attached a sample output of a full test run.
msg185306 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-26 21:22
I would like both the normal (non-overwritten) progress list and the summary at the end :)

I would not like the overwriting mode to be the default, but otherwise it looks good.
msg185307 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-26 21:23
Oh, by 'looks good' I was referring to the output, not the patch, which I haven't looked at yet.
msg185330 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-03-27 13:18
So are you saying you want both the short-form output while running but then the whole output upon completion, David? I can understand wanting the summaries still in order to sort the skipped tests vs. failures, but not outputting every test that succeeded since you can figure that out through a process of elimination w/o much issue.

I think instead it would serve us better to have a command to write the tests that were run and what happened to a file for use with the -f with successes commented out, skips commented out along with the skipped message, and then only failures left uncommented for easier re-running of the tests in the same order.
msg185363 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-27 18:44
No, what I'd prefer is that the current "print it as it runs" behavior not change, but that the list of skip reasons be displayed at the end. after all the tests have completed.  This is probably not a realistic request, so I'm fine with just having Ezio's version as an option.  

To be clear: I don't have any use for the list of passed tests other than it keeping me informed of the test run progress, and for that use I want it to scroll up my display, not overwrite on one line.

That said, I don't have *strong* feelings about this :)
msg185365 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2013-03-27 18:55
I say move forward and put it behind a flag (w/ quiet is fine, maybe some arg to specify quietness or -qq much like -vv?).
msg185464 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-28 19:48
I would prefer to have it by default with a flag to disable it, since IMHO the full list of passed tests is just noise, but other devs don't seem to like the idea.
Another option would be to save the flags in a config file or in an envvar, so that it won't be necessary to repeat them every time, but that's another issue (and on a related note, people don't seem to specify -Wd either when they run the tests, even if they should)...
msg185465 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-03-28 19:49
One reason I think the full list is better is that it's easier for beginners to understand and review what's happening (and also get a taste of the contents of the test suite). Experienced contributors wouldn't care that much admittedly.
msg185466 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-28 19:55
They will still see the test names and the counter that increases.  I would also say that having the whole output fitting in a screen is less intimidating than a few screens filled with a wall of text.  It also makes easier to notice skipped and failing tests, and that might mitigate the common question "why it says that test_xxx was skipped?" (when the test is actually skipped, a reason is usually presented, but that's lost in the output, and not repeated at the end).
Maybe we should ask on core-mentorship what version they like more?
msg185493 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-03-29 07:21
There definitely needs to be an option to continue to include the normal progress list in the output.  This is essential for comparing test runs.  I'm +0 on whether it be the default or not.  Actually, what is really needed for automated analysis of test runs is getting the complete output of a regrtest run in a standard serialized format, perhaps YAML, including the test name, test result, stdout, and stderr for each test. But that's a separate issue.  I've attached a the output from a current OS X run with the proof-of-concept patch.  One issue that I see is that there are a number of tests that normally produce messages of various sorts to stderr and/or stdout without the test failing.  Suppressing the passed test names makes it difficult to figure out from which tests these messages are coming from.
msg185503 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-29 17:35
I fixed all the tests that were producing extra output on my machine, but there are probably some left on other machines.  Using -uall also produces extra output, and I haven't decided how to deal with that yet (maybe it should be silenced as well).
Getting this right might require to introduce e.g. a new regrtest/support.print() function though...
msg185510 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-29 18:22
-uall shouldn't produce more output by itself, it just runs more tests.  Do you mean that some *output* is actually conditionalized on whether a certain resource is enabled?
msg185511 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-29 18:27
The tests enabled by -uall (and probably -unetwork) use open_urlresource, and open_urlresource has a "print('\tfetching %s ...' % url, file=get_original_stdout())".  This should probably be printed only when -v is passed.
msg185962 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-04-03 21:53
Attached patch shows the "Fetching <url> ..." only in verbose mode.
msg230937 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2014-11-10 07:12
issue17554-urlfetch.diff LGTM.
msg232407 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-12-10 00:34
New changeset 8c337b4a8811 by Berker Peksag in branch 'default':
Issue #17554: Print "fetching <url> ..." messages only in verbose mode.
https://hg.python.org/cpython/rev/8c337b4a8811
msg346502 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-06-25 10:24
This issue has no activity for 5 years. regrtest output evolved a lot since that one. I like "verbose" output on buildbots to be able to debug when something goes wrong.
History
Date User Action Args
2019-06-25 10:24:25vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg346502

stage: patch review -> resolved
2014-12-10 06:01:16Arfreversetnosy: + Arfrever
2014-12-10 00:34:04python-devsetnosy: + python-dev
messages: + msg232407
2014-11-10 07:12:36berker.peksagsetmessages: + msg230937
stage: needs patch -> patch review
2014-07-05 17:16:58floxlinkissue15974 superseder
2014-07-05 17:15:01floxsetnosy: + flox
2014-07-05 16:44:12berker.peksagsetnosy: + berker.peksag

versions: + Python 3.5, - Python 3.4
2013-04-03 21:53:11ezio.melottisetfiles: + issue17554-urlfetch.diff

messages: + msg185962
2013-03-29 18:27:25ezio.melottisetmessages: + msg185511
2013-03-29 18:22:17r.david.murraysetmessages: + msg185510
2013-03-29 17:35:10ezio.melottisetmessages: + msg185503
2013-03-29 07:22:00ned.deilysetfiles: + patched_regrtest_output.txt
nosy: + ned.deily
messages: + msg185493

2013-03-28 19:55:21ezio.melottisetmessages: + msg185466
2013-03-28 19:49:45pitrousetmessages: + msg185465
2013-03-28 19:48:18ezio.melottisetmessages: + msg185464
2013-03-27 18:55:20brett.cannonsetmessages: + msg185365
2013-03-27 18:44:03r.david.murraysetmessages: + msg185363
2013-03-27 13:18:00brett.cannonsetmessages: + msg185330
2013-03-26 21:23:12r.david.murraysetmessages: + msg185307
2013-03-26 21:22:36r.david.murraysetmessages: + msg185306
2013-03-26 21:02:49ezio.melottisetfiles: + regroutput.txt
2013-03-26 21:02:15ezio.melotticreate