Issue47133
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2022-03-26 20:56 by ethan.furman, last changed 2022-04-11 14:59 by admin.
Messages (9) | |||
---|---|---|---|
msg416090 - (view) | Author: Ethan Furman (ethan.furman) * | Date: 2022-03-26 20:56 | |
When running unittest with the -v flag, if a test has errors, and has a docstring, the test name is shown on one line, and the docstring is shown on the next line -- and the ERROR word is shown with the docstring. What this means is that: - the test name is easily missed by anyone who isn't aware of that (I only recently found out when someone else pointed it out to me) - grepping for ERROR doesn't reveal the test name, necessitating a search through the testing code to find the test that failed There are two possible solutions, and the selection of which one should be discussed on python-dev: - print the name and docstring, and ERROR, all on one line; or - provide a command-line switch to select that behavior --- For the experienced developers who see this: please leave the issue for one of the new developers subscribed to core-mentorship. Thank you. |
|||
msg416094 - (view) | Author: Itay Yeshaya (itay.yeshaya) | Date: 2022-03-26 22:16 | |
I would like to work on this. |
|||
msg416097 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * | Date: 2022-03-26 23:19 | |
Issue14265, issue46126. |
|||
msg416099 - (view) | Author: Steven D'Aprano (steven.daprano) * | Date: 2022-03-27 00:12 | |
I don't think this change of behaviour should be accepted without discussion, and I'm not sure why you think Python-Dev is the right place, rather than here. Messing around with unittest and docstrings has already caused issues in the past: https://docs.python.org/3/library/unittest.html#unittest.TestCase.shortDescription so perhaps we shouldn't break what's not broken? And frankly, I don't see why the current behaviour is a problem, although maybe that's because the way I use unittest is different from the way you use it. Grepping the output for errors? Why run verbose mode if you aren't going to read the whole output? Seems very odd. Ethan says: "the test name is easily missed by anyone who isn't aware of that" Surely the solution to that is education, not changing unittest? Now you are aware of it, and the test name is no longer "easily missed". If it ever was. To me, it is plain as day. Here is some typical output: [steve ~]$ python3.10 -m unittest -v mathlib/testing/test_utils.py test_broken_doc (mathlib.testing.test_utils.TestMinmax) This is a docstring ... ERROR test_broken_nodoc (mathlib.testing.test_utils.TestMinmax) ... ERROR test_empty_iterable (mathlib.testing.test_utils.TestMinmax) ... ok [additional passing tests ommitted for brevity] ====================================================================== ERROR: test_broken_doc (mathlib.testing.test_utils.TestMinmax) This is a docstring ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/steve/Desktop/mathlib/testing/test_utils.py", line 20, in test_broken_doc self.assertEqual(1, x) NameError: name 'x' is not defined ====================================================================== ERROR: test_broken_nodoc (mathlib.testing.test_utils.TestMinmax) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/steve/Desktop/mathlib/testing/test_utils.py", line 24, in test_broken_nodoc self.assertEqual(1, x) NameError: name 'x' is not defined ---------------------------------------------------------------------- Ran 10 tests in 0.004s FAILED (errors=2) Errors are listed twice, once in the summary, and the second time in the details section showing the traceback. Note that if you just grep for ERROR you get the result you want: [steve ~]$ python3.10 -m unittest -v mathlib/testing/test_utils.py |& grep ERROR This is a docstring ... ERROR test_broken_nodoc (mathlib.testing.test_utils.TestMinmax) ... ERROR ERROR: test_broken_doc (mathlib.testing.test_utils.TestMinmax) ERROR: test_broken_nodoc (mathlib.testing.test_utils.TestMinmax) So I am having trouble seeing what the problem here is. |
|||
msg416100 - (view) | Author: Ethan Furman (ethan.furman) * | Date: 2022-03-27 00:35 | |
I proposed a discussion to python-dev to increase the odds that folks with an interest could chime in. Not everyone follows the new-bugs list. I find having the output on two lines counter-intuitive -- I was expecting one line per test, and indeed my experience seemed to support that (out of hundreds of tests showing on one line, it's easy to miss that the rare verbose error is on two). And I'm not the only one, since another developer also thought there was a problem and spent most of issue46126 "fixing" it. Another reason for posting to python-dev is that issue 46126 had been active for three months and had a PR merged before another dev mentioned that verbose tests with docstrings just printed on two lines instead of one. Thank you for pointing out the possible disruption caused by unittest output changes. |
|||
msg416103 - (view) | Author: Steven D'Aprano (steven.daprano) * | Date: 2022-03-27 04:07 | |
I have no problem with pinging Python-Dev, or any other forum, asking for interested parties. I just don't think we should be intentionally spliting the discussion more than it's going to get split organically. (If this is interesting to people, they will discuss it on other forums no matter what we say or do.) Aside from the conservative position "If it ain't broke, don't break it", also known as "Chesterton's Fence": https://matt-rickard.com/chestertons-fence/ I'm not specifically opposed to this change. Nor am I in favour. Why do we print the first line of the doctests in the output of failing tests? If we understood the rationale for why we do that, it might give some insight into the consequences of removing it. If we don't know why it was shown in the first place, maybe we shouldn't remove it until we have some idea. I am -1 on another option flag unless there is no other choice. Each boolean flag option doubles the number of tests of unitttest itself needed for full coverage. Another option might be to leave the docstring line alone, and just *add* the 'ERROR' to the method name line: # current output test_broken_doc (mathlib.testing.test_utils.TestMinmax) This is a docstring ... ERROR # enhancement test_broken_doc (mathlib.testing.test_utils.TestMinmax) ERROR This is a docstring ... ERROR That seems to me to be less risky than trying to cram them all on one line. test_broken_doc (mathlib.testing.test_utils.TestMinmax) This is a docstring ... ERROR By the way, I presume we do the same thing for FAIL lines, which have the same behaviour as ERROR lines: test_fail_doc (mathlib.testing.test_utils.TestMinmax) This is a docstring ... FAIL test_fail_nodoc (mathlib.testing.test_utils.TestMinmax) ... FAIL |
|||
msg416105 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * | Date: 2022-03-27 06:42 | |
This is already the third (at least) issue asking for such feature. Many people have problems with the current output. I would prefer to have a full qualified name of the test which I can copy by a double click and insert in a command that reruns specified tests. And full path to the test's source which I can copy to run in editor. pytest output looks more user-friendly to me. But it may break other's test output parsers so we perhaps need an option for this. Options for unittest and libregrtest. I am not sure that it is a task for an average new contributor. |
|||
msg416138 - (view) | Author: Steven D'Aprano (steven.daprano) * | Date: 2022-03-27 22:29 | |
Rather than an option for this, it would be cleaner to deprecate the current output in 3.11, and fix it in 3.12 or 3.13. Otherwise we have to maintain the old (bad?) output and the new output both forever. |
|||
msg416642 - (view) | Author: Gregory P. Smith (gregory.p.smith) * | Date: 2022-04-03 20:00 | |
A more fundamental issue behind this: unittest's plain text verbose output is not intended to be parsable and consumable by machines. It is 22 years old and was intended for use in 80 column wide terminals as a progress report while watching it run as much it is a final result. Consider that the legacy use case or format if you wish. Allowing an option for output to one line per test with description could be more likely to require horizontal scrolling or reading past a line wrap in CI setups where text is scrolling regions embedded in scrolling regions. Yuck. BUT not necessarily bad. Just different. The ideas expressed in this thread of making a more clear text format with the important PASS/FAIL/ERROR/SKIP status up front and a very clear pastable test module.class.test_name seem wise. I expect most Python projects with nicer test output formatting desires have simply adopted pytest and never looked back. --- Other python unittest derivative concepts to get clear output of results: We add a reliably machine parsable XML report of test outcomes in absl-py https://github.com/abseil/abseil-py/tree/main/absl/testing. It's IMNSHO a rather annoying implementation - we wish unittest made plugging in additional output formats easier. We use that generated XML format for results from unittest systems in all of our languages at work, it is picked up and parsed by our internal CI system and used to display nicely formatted results when available. I assume pytest also provides machine parsable reporting capabilities but I haven't looked. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:57 | admin | set | github: 91289 |
2022-04-03 20:00:29 | gregory.p.smith | set | nosy:
+ gregory.p.smith messages: + msg416642 |
2022-03-27 22:29:01 | steven.daprano | set | messages: + msg416138 |
2022-03-27 06:42:08 | serhiy.storchaka | set | messages: + msg416105 |
2022-03-27 04:07:45 | steven.daprano | set | messages: + msg416103 |
2022-03-27 00:35:24 | ethan.furman | set | messages: + msg416100 |
2022-03-27 00:12:10 | steven.daprano | set | nosy:
+ steven.daprano messages: + msg416099 |
2022-03-26 23:19:32 | serhiy.storchaka | set | nosy:
+ serhiy.storchaka messages: + msg416097 |
2022-03-26 22:16:59 | itay.yeshaya | set | nosy:
+ itay.yeshaya messages: + msg416094 |
2022-03-26 21:19:13 | iritkatriel | set | keywords: + easy |
2022-03-26 21:01:09 | ethan.furman | set | assignee: ethan.furman |
2022-03-26 20:56:32 | ethan.furman | create |