This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Allow better verbosity / output control in test cases
Type: enhancement Stage: needs patch
Components: Library (Lib) Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: berker.peksag, ezio.melotti, martin.panter, michael.foord, pitrou, r.david.murray, rbcollins, serhiy.storchaka, zach.ware
Priority: normal Keywords:

Created on 2014-08-14 15:39 by pitrou, last changed 2022-04-11 14:58 by admin.

Messages (12)
msg225304 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-08-14 15:39
Currently, test cases have no control over output and verbosity. I would propose two possible enhancements:
- give the TestCase read access to the verbosity value (as e.g. `self.verbosity`), to allow for conditional output in tests
- allow test methods to force output buffering (rather than only let the user specify it on the command line), because you would like some stuff to only be printed out on failure; a decorator would IMO be fine
msg225653 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2014-08-22 09:50
This seems like a reasonable improvement, I'd be in favour. I'd be *slightly* concerned that a test can override an explicit verbosity setting of the user, but I guess that annoyance is up to the test writer (they should check for an explicit setting first).
msg225664 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-08-22 14:27
Oh, I'm only proposing read-only access to the verbosity information. And forcing output buffering would only last for the current test method, of course.
msg225671 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2014-08-22 14:49
Yep. I have no qualms about allowing test cases to switch on buffering.
msg227621 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-09-26 15:10
There is the verbose attribute of the test.support module.
msg227622 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2014-09-26 15:20
That only works for the CPython test suite (and it's not a public API).

FWIW I'm +1 on the idea, but I would have to see how it will get implemented in a patch.
msg227624 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-09-26 15:29
Usages of test.support.verbose should be replaced by self.verbosity.

As for output buffering, may be replace sys.stdout by file-like object which 
flushes its buffered content to original stdout on failure and discard it on 
success. Or add the self.log file-like object with such behavior and redirect 
all verbose output to it.
msg227626 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2014-09-26 15:35
> As for output buffering, may be replace sys.stdout by file-like object
> which flushes its buffered content to original stdout on failure and
> discard it on success.

This is what the --buffer option is already supposed to do (I only found out about it thanks to this issue, the name is not very indicative of what it does...).  IIUC what Antoine is suggesting is having a more fine-grained control of the buffering, and the ability to set it from individual test cases rather than using a global command line flag or unittest.main(buffer=True) (which is only used while executing the test file directly).
msg227628 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-09-26 15:56
Indeed, I'm sorry for suggesting two features in one issue :-)

Feature #1 is "self.verbosity" (as a read-only variable) on test cases. Sounds like a no-brainer, IMHO :-)

Feature #2 is selective enabling of the buffering feature in test cases. This rather misnamed features only prints out stdout when the test fails, which is useful when you want permanent debug statements that only pollute stdout when there is a test failure.
msg248939 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-08-21 01:32
There's a few interacting things here. If I can suggest some design thoughts.

buffering within a test is I think really something we should offer a test servicing API for. There are many thirdparty ones (e.g. I have one in fixtures) - but it should be a dedicated API, even if the implementation happens to be shared with the buffer runner feature.

I think it is reasonable in some cases to change TestCase behaviour based on how much debugging is desired - even in CI environments, some outputs are much more expensive than others to generate (e.g. taking a full dump of a database). To enable that we do need a CLI -> runtime environment communication mechanism.

There are many related APIs which we could look at for doing this.
* TestCase's load API (__init__) - we'd get at this via TestLoader
* TestCase's execution API (run, debug) - we'd get at this through TestRunner (for more parameters) or perhaps TestResult (by adding some attribute/query method).
* TestCase's test servicing API (assert*) - we could add a query method or a well known attribute to abstract out where the info is actually coming from
* TestCase's user supplied API (setUp, test methods)
* TestResult - constructed by the runner, it is in some ways the natural means for passing stuff like this around - its how result does its buffering today: but IMO the implementation is in the wrong spot there [it precludes parallelisation vs a test service API that uses a mutex only when needed], so I'd like to not repeat that here.

TestCases support being run multiple times, so to me that rules out using the load API.

I very much want to take TestResult in the direction of being forward-only: feed-backwards APIs on it are much more complex to reason about in the case of Y joins in the result chain. So I think its time we introduced a runner -> test API.

Something like

def run(self, result=None, runner=None):
   ...
   dunder_set(self, '__unittest__runner__', runner)
   try:
   ...
   finally:
       dunder_set(self, '__unittest__runner__', None)
   
def debug(self, runner=None):
   ... same ...

with a helper to pull that out without folk's own code triggering name mangling.

And runner would then be the home for things like aborting a test run, providing configuration like verbosity, arbitrary end user options and so on.
Something like:
runner.config = {}
runner.config['verbosity'] = ...
runner.config['user'] = {... marshalled from CLI ... }
msg250249 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2015-09-08 18:06
Using the runner as a "context" passed to test cases (and accessible from tests) for this kind of configuration seems like a good approach to me.
msg250251 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-09-08 18:24
I like that approach as well, it is the kind of API I've been finding myself writing a fair bit lately (a context passed to class instances that otherwise don't store global state themselves).
History
Date User Action Args
2022-04-11 14:58:06adminsetgithub: 66393
2015-09-08 18:24:26r.david.murraysetmessages: + msg250251
2015-09-08 18:06:24michael.foordsetmessages: + msg250249
2015-08-21 01:32:34rbcollinssetmessages: + msg248939
2015-02-13 01:25:19demian.brechtsetnosy: - demian.brecht
2014-09-27 00:28:17martin.pantersetnosy: + martin.panter
2014-09-26 15:56:06pitrousetmessages: + msg227628
2014-09-26 15:35:02ezio.melottisetmessages: + msg227626
2014-09-26 15:29:31serhiy.storchakasetmessages: + msg227624
2014-09-26 15:21:50r.david.murraysetnosy: + r.david.murray
2014-09-26 15:20:44ezio.melottisetmessages: + msg227622
stage: needs patch
2014-09-26 15:10:14serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg227621
2014-09-26 12:05:17pitrousetnosy: + rbcollins
2014-08-22 14:49:58michael.foordsetmessages: + msg225671
2014-08-22 14:27:13pitrousetmessages: + msg225664
2014-08-22 09:50:49michael.foordsetmessages: + msg225653
2014-08-20 12:53:55berker.peksagsetnosy: + berker.peksag
2014-08-18 18:31:35demian.brechtsetnosy: + demian.brecht
2014-08-14 15:50:35zach.waresetnosy: + zach.ware
2014-08-14 15:39:01pitroucreate