classification
Title: move regrtest over to using more unittest infrastructure
Type: enhancement Stage: test needed
Components: Tests Versions: Python 3.3
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: berker.peksag, brett.cannon, chris.jerdonek, eric.araujo, ezio.melotti, r.david.murray, rhettinger, sandro.tosi, vstinner, zach.ware
Priority: low Keywords: gsoc

Created on 2011-01-20 23:45 by brett.cannon, last changed 2015-10-02 21:21 by vstinner. This issue is now closed.

Messages (19)
msg126665 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2011-01-20 23:45
test.regrtest is rather old and has not been updated to take advantage of all the latest features in unittest (e.g., test discovery). It might be a rather large undertaking with various bits requiring some changes (e.g., getting away from raising exceptions for skipped tests and instead using unittest.skipIf), but for maintainability it might be good to try to use as much unittest code in regrtest as possible.
msg126781 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-01-21 20:21
I think we have already been moving in this direction for quite some time.  Past policy is to only change things when we are working on that area of code anyway.  If someone wants to make some specific proposals to simplify regrtest by doing a wholesale move of test code away from a regrtest supported infrastructure to a unittest supported infrastructure, I think that's worth considering, but I think it should be done bit by bit with specific proposals to replace specific regrtest features.

Extending regrtest to support unittest test discovery directly is also a worthwhile specific proposal.

Do you intend this issue to be an "index issue" that links to issues with specific proposals?

(Note: see also the closely related issue 8273).
msg126784 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2011-01-21 20:39
Yes, I somewhat view this as an index issue. I don't expect a wholesale move but a more step-by-step move.
msg126818 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2011-01-22 07:37
Wouldn't the time be better spent factoring the test suite or improving coverage?  I'm not sure how simply rearranging the tests makes us better-off.
msg126844 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-01-22 17:35
I also would put increasing test coverage at a higher priority, but this sort of refactoring can be a good step in the development path of new contributors, and doing it does decrease the future maintenance burden.
msg140672 - (view) Author: √Čric Araujo (eric.araujo) * (Python committer) Date: 2011-07-19 14:57
For one thing, I think resources could be implemented in terms of skips, or even included into stock unittest.
msg170775 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-19 22:31
One important piece is that regrtest currently has no tests (e.g. there is no test_regrtest.py), so changing it must be done more carefully.  How do people feel about new (or newly modified) regrtest classes and functions going into a different fully-tested module with its own test_* file.  Currently, regrtest is a mixture of definitions and "running code," so I would be leery of adding test_regrtest.

This could also be done by making regrtest a package, so that regrtest-related modules would be in a single directory.
msg183229 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2013-02-28 19:40
> Extending regrtest to support unittest test discovery directly is also a worthwhile specific proposal.

Updating the tests to support discovery in all cases is discussed in (meta) issue 16748.  There are also many individual issues in the tracker (one per test module).
msg183314 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-02 12:05
> For one thing, I think resources could be implemented in terms of skips

While working on #17333 I noticed a related problem.  Most of the tests in test_imaplib are skipped if the 'network' resource is not specified.  The test contains the following lines to enable those tests when the file is executed directly:

if __name__ == "__main__":
    support.use_resources = ['network']
    unittest.main()

but this clearly doesn't work while using unittest discovery, so those tests are skipped.

Implementing resources as skips makes sense, but we might also need a way to specify what resources are needed that works with unittest discovery.  Running these tests only when the test file is executed directly or through `regrtest -u` is also an option.
msg183329 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-02 16:48
We have previously discussed adding resource-awareness to unittest, but that is a much bigger project (API design, scope considerations, etc etc).  In the meantime, what we could do is modify the current resource-skip logic slightly: make it so that tests are *not* skipped for resource reasons *unless* some flag in support is set indicating that resource enforcement is in effect *and* the specified resource is not in the list of currently-asserted resources.  That way tests are only skipped for resource reasons if run via regrtest.
msg183331 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-02 17:16
There are actually three separate issues here:
1) The resources don't use unittest skip (or at least not always), so while running the tests without regrtests there are no indication that some tests have been skipped.  This could be addressed by converting the current resources to proper skips (with something like @skip_unless_resource('network')) and shouldn't be too difficult.

2) The "network" resource could be enabled by default, given that network connectivity is nowadays fairly common (it would be even better if that was detected automatically at runtime).  This would enable more tests and shouldn't have negative side-effects (unless the network is not available and the tests have no way to realize that and fail).

3) unittest could have a "resource API", but that's not strictly necessary if we still provide a way to run the skipped tests through regrtest and/or while running the test file individually.
msg183334 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-02 17:35
No controversy on (1), we should just do that.

I don't see a strong reason to change (2).  I always run the test suite with -uall, but if I were running a restricted set I'd rather not have resources outside the local machine be accessed.

My proposal is a way to provide (3): running the regrtest-resource-skipped tests when not running them through regrtest.  If you *also* want a way to skip tests based on resource when running them outside regrtest, then unittest would have to grow a resource API.
msg183335 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-02 17:45
> I'd rather not have resources outside the local machine be accessed.

Fair enough.

> If you *also* want a way to skip tests based on resource when running 
> them outside regrtest, then unittest would have to grow a resource API.

This can already be done though -- adding a resource API to unittest would just provide an additional way to do it without going through regrtest.

I don't think we will ever get rid of regrtest.  It would be nice if most of the code could be removed and standard unittest features were used instead, but regrtest will always have more advanced/customized feature built on top of unittest (which is not to say that a resource API for unittest is useless, but that's not a blocker for this issue :).
msg183336 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-02 18:23
So you are agreeing with my proposal?

And no, I don't think the goal is to get rid of regrtest, it is just to make it as small as practical.  Or to put it another way, to add the features of regrtest that are generally useful to unittest.
msg183337 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-02 18:31
> So you are agreeing with my proposal?

I will have to see a concrete proposal first.  If a resource API is useful enough to be added to unittest, and we can come up with a decent API we can move it there, otherwise it can just stay in regrtest.  There's also the issue that the unittest API is already quite big, and adding even more things there might do more harm than good.

There are several other parts of regrtest that can take advantage of already existing unittest features first, and convert them might be enough to close this issue even if the existing resource infrastructure is kept in regrtest.
msg183338 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-02 18:34
My proposal is how to do this without adding anything to unittest.  It is about how to implement the skips so that all tests are run when run under unittest, but resource control still happens when the tests run under regrtest.
msg183341 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-02 18:45
Yes, that could be done as well, but even if they are not run by default under unittest, I'd be already happy to see them marked as skipped.  If I see that they are skipped I can always switch to regrtest to enable the necessary resource, or just execute the file directly if the resource is enabled just before unittest.main() (like the example in msg183314).
msg183344 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-02 19:58
Ah, but for me the point of running directly under unit test (and unittest discovery) it to be able to use the standard unittest facilities for running a single testcase or individual test.  Converting them to skips without making them run if regrtest isn't active might make things worse for that use case.  If we are going to convert them to skips, why not do it right to start with?  It won't any harder, since the logic will be baked into the skip decorators.
msg252162 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-10-02 21:21
This issue was opened 4 years ago and has no activity since 2 years :-/ Even if yes, it would be nice to enhance regrtest and unittest, this issue doesn't propose any concrete plan with a concrete patch. I understand that the current situation is not perfect but it's acceptable :-) I close the issue as out of date.

If someone is still motivated, please open a new issue with a concrete plan and maybe summarize this one.

In the meanwhile, the work on relying on unittest test discovery is still ongoing and made good progress last... years (see issue #16748).
History
Date User Action Args
2015-10-02 21:21:30vstinnersetstatus: open -> closed

nosy: + vstinner
messages: + msg252162

dependencies: - Make CPython test package discoverable
resolution: out of date
2015-03-12 14:28:45berker.peksagsetnosy: + berker.peksag
2014-08-06 23:53:55pitrouunlinkissue15974 dependencies
2013-03-04 14:37:33zach.waresetnosy: + zach.ware
2013-03-02 19:58:02r.david.murraysetmessages: + msg183344
2013-03-02 18:45:02ezio.melottisetmessages: + msg183341
2013-03-02 18:34:17r.david.murraysetmessages: + msg183338
2013-03-02 18:31:44ezio.melottisetmessages: + msg183337
2013-03-02 18:23:43r.david.murraysetmessages: + msg183336
2013-03-02 17:45:29ezio.melottisetmessages: + msg183335
2013-03-02 17:35:08r.david.murraysetmessages: + msg183334
2013-03-02 17:16:43ezio.melottisetmessages: + msg183331
2013-03-02 16:48:57r.david.murraysetmessages: + msg183329
2013-03-02 12:05:00ezio.melottisetmessages: + msg183314
2013-02-28 19:40:57chris.jerdoneksetdependencies: + Make CPython test package discoverable
messages: + msg183229
2012-09-19 22:31:24chris.jerdoneksetmessages: + msg170775
2012-09-19 22:09:06chris.jerdoneksetnosy: + chris.jerdonek
2012-09-19 19:25:24brett.cannonlinkissue15974 dependencies
2011-07-19 14:57:55eric.araujosetnosy: + eric.araujo
messages: + msg140672
2011-01-22 17:35:13r.david.murraysetnosy: brett.cannon, rhettinger, ezio.melotti, r.david.murray, sandro.tosi
messages: + msg126844
2011-01-22 07:37:20rhettingersetnosy: + rhettinger
messages: + msg126818
2011-01-22 03:42:40ezio.melottisetkeywords: + gsoc
nosy: + ezio.melotti
2011-01-21 20:39:39brett.cannonsetmessages: + msg126784
2011-01-21 20:21:50r.david.murraysetmessages: - msg126780
2011-01-21 20:21:43r.david.murraysetmessages: + msg126781
2011-01-21 20:20:17r.david.murraysetnosy: + r.david.murray
messages: + msg126780
2011-01-21 06:10:34sandro.tosisetnosy: + sandro.tosi
2011-01-20 23:45:44brett.cannoncreate