Issue2241
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2008-03-05 16:24 by bcwhite, last changed 2022-04-11 14:56 by admin. This issue is now closed.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
unittest-diff25.py | bcwhite, 2008-03-05 16:24 | diff -u unittest.py |
Messages (7) | |||
---|---|---|---|
msg63285 - (view) | Author: Brian White (bcwhite) | Date: 2008-03-05 16:24 | |
The attached diff adds a "-o" ("--one") option to the "unittest" module that causes the run to abort on the first error encountered. I name my tests so that the lowest-level tests get run first so stopping at the first error tends to prevent a lot of dependent errors and speed debugging. During development, I typically run the tests I'm writing with "-ov". During a full test run, I omit both those flags. |
|||
msg63313 - (view) | Author: Steve Purcell (purcell) | Date: 2008-03-06 08:27 | |
Hi Brian; The module is intended for test suites where the unit tests are written to be independent of each other, which is the "standard" way to do things. Note, for instance, that there is no convenient support for changing the order in which tests run. When tests are written like that, you can interrupt a bulk test run at any point, and you can run a single test to reproduce and then debug a failure. Given your test suite, I can see how this '--one' option is helpful to you, but I don't believe it should be made standard. (I've never seen it in any XP-inspired test framework or related IDE UI.) However, you can easily write a custom TestRunner that provides this "fast abort" behaviour that you want, and then hook it into unittest as follows: unittest.main(testRunner=MyCustomTestRunner()) (BTW, regarding the implementation, it's not ideal to pass the 'onlyOne' parameter down and through to the run() method; much better would be to initialise a TestResult subclass with the 'onlyOne' option, so that it could then abort when 'addError' or 'addFailure' is called. You can use that trick in any custom test runner you might write.) Best wishes, -Steve |
|||
msg63321 - (view) | Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * | Date: 2008-03-06 14:02 | |
Actually, py.test and nose both have the -x option for this purpose. I use it very often during development, mostly during a refactoring phase: failures are easy to correct, and I don't want to wait for the complete suite to complete and display tons of tracebacks. Even while developing on core python, this option would have helped me a couple of times. Please, reopen this item! |
|||
msg63322 - (view) | Author: Steve Purcell (purcell) | Date: 2008-03-06 14:15 | |
I guess I don't completely agree with the rationale, because I've never wanted this feature; when running tests en-masse after refactoring, I want an overview of what was broken. If the codebase is in good shape, the test failures will be few and close together, and then I can usually re-run one of the individual test cases to debug the error. However, this isn't a big issue for me, and if someone's willing to prepare a new patch with the different implementation I described previously, I'm happy to re-open this ticket and sign off on it. I'd suggest using the same argument names as nose and py.test in this case. |
|||
msg63333 - (view) | Author: Brian White (bcwhite) | Date: 2008-03-06 19:59 | |
Having tests run independently of each other is not the same as having tests be completely independent. I'd argue that the latter is impossible. You're never going to test the entire system in a single test case and thus the tests work together (i.e. not independently) to test everything. If one function under test calls another function that is also tested, then it makes sense to test the lower-level function first and display any problems as it will be easier/faster to find the root of the trouble than when the error causes unexpected results in the higher-level function. To make things easier, I simply name my tests such that lower-level functions are tested first. Each individual tests still runs independently, of course. The point of the "--one" option is just to have it stop when the first test fails, allowing me to fix the lowest level error. If that same error causes a dozen other tests to also fail and I just pick one failure randomly to start debugging, it's going to take me longer, perhaps a lot longer, to track down the problem. As for the method of implementation, I'm sure there are better ways to do it. Though I can write fully functional programs in Python, I by no means consider myself an expert in the language. I did it this way because the only other solution I saw was a global variable and figured that would be a poor way to do it. As such, I'd appreciate help on exactly how it should "properly" be done. :-) I'll let somebody else actually re-open this issue if it's a desired item since I'm not knowledgeable enough to see the solution you propose. Thanks! |
|||
msg63334 - (view) | Author: Steve Purcell (purcell) | Date: 2008-03-06 20:32 | |
Hi Brian - thanks for going into some details of your rationale! You might be surprised to hear that it's indeed possible to make all of your unit tests mutually independent; check out the area of 'mock objects'. It turns out to be possible, and indeed desirable once the "zen" of the technique clicks, to test every class in isolation without referring to other neighbouring classes. I was surprised by the enormous effectiveness of this somewhat hardcore technique when I was forced into it by working with one of the original Mock Object paper authors. Having already spent years coaching developers in XP techniques, I thought I was already a testing whiz. In most real-world cases, though, a class under test will be tested using its interaction with other separately-tested classes in the system, and the associated unit tests therefore bear some relation to one another. It's usually not helpful to divide those classes into layers that can be tested in order from the lowest layer to the highest, because classes tend to form clumps rather than layers. When a big suite of tests is run, failures therefore form clumps too, and often the underlying programming error is easier to see by looking at the clump rather than just the first failure. I think this explains why most people get by without an option like '-o'. Of course, it often makes sense to have separate test suites for different areas of the system under test so that they can be run in isolation. Rather than relying on test naming, you might consider explicitly building TestSuites that run your test cases in the desired order. As for preparing an updated patch, I'll get to it if I get a few minutes. All the best from this Brit in Germany. |
|||
msg63403 - (view) | Author: Brian White (bcwhite) | Date: 2008-03-08 15:21 | |
I am somewhat new to mock objects. I'm coding up my first one now (in D) to simulate a "stream" for other objects I want to write. Even within a single module, I typically have many tests for the methods within that module. And since a module's methods make use of each other, there is again a case for the tests of the lower-level functions to be executed first. Anyway, this is something that works for me, but I understand that not everybody operates this way. All the best from this Canadian in Switzerland. :-) |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:56:31 | admin | set | github: 46494 |
2008-03-08 15:21:53 | bcwhite | set | messages: + msg63403 |
2008-03-06 20:32:10 | purcell | set | messages: + msg63334 |
2008-03-06 19:59:04 | bcwhite | set | messages: + msg63333 |
2008-03-06 14:15:28 | purcell | set | messages: + msg63322 |
2008-03-06 14:03:00 | amaury.forgeotdarc | set | nosy:
+ amaury.forgeotdarc messages: + msg63321 |
2008-03-06 08:27:27 | purcell | set | status: open -> closed assignee: purcell resolution: rejected messages: + msg63313 |
2008-03-05 21:37:36 | benjamin.peterson | set | nosy: + purcell |
2008-03-05 16:24:30 | bcwhite | create |