classification
Title: move generally useful test.support functions into the unittest package
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.4
process
Status: closed Resolution: rejected
Dependencies: 9754 Superseder:
Assigned To: michael.foord Nosy List: brett.cannon, eric.araujo, ezio.melotti, flox, michael.foord, r.david.murray, rhettinger, tarek, zach.ware
Priority: low Keywords:

Created on 2010-03-31 11:19 by tarek, last changed 2013-03-19 01:02 by michael.foord. This issue is now closed.

Messages (25)
msg101991 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2010-03-31 11:19
Let's move test_support in unittest !

Then maybe, let's expose some of test_support functions into a new class on the top of unittest.TestCase, so they can be used via methods.
 
The purpose is power up people when it comes to write test fixtures or work in a testing environment. These helpers were built to test Python itself, are quite unknown out there. I think it's a shame :)

http://docs.python.org/library/test.html#module-test.test_support
msg101992 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-03-31 11:23
Hmmm... I'm not sure moving everything as is is a good idea. Any parts in particular you want?
msg101993 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2010-03-31 11:27
Basically, all the APIs in test_support could live in unittest and the test package could only contain the test modules written for Python.

IOW, these tests helpers can help more people than the core devs
msg101994 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-03-31 11:31
Not all the APIs are appropriate though, for example TestFailed and TestSkipped are not needed. Some of the APIs only make sense for the test_support runner and unittest has its own runner now.

I think the proposal needs to be more specific about which APIs you are suggesting would be generally useful.
msg101995 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2010-03-31 11:32
Sure, no need to have the test runner parts.
msg101996 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-03-31 11:35
So which parts, can you list the ones you think are most important?

At the moment we don't have to maintain backwards compatibility with test.test_support as it isn't a 'public' API - so the cost in adding it to the standard library is the same as adding any code. Once we do this we are less free to modify / evolve the API.
msg101997 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-03-31 11:54
Actually, the list Terek posted a link to is a public doc page, and as was made clear in the recent issue about the check_warnings helper, we do have to maintain backward compatibility.  

There (should be) undocumented stuff in test_support, purposely left undocumented so that we *don't* have to maintain backward compatibility.  Myself I wish that doc page didn't even exist.  Putting stuff that users can use into unittest is a much better idea.

Maybe we could move the appropriate stuff and get rid of the test_support doc page altogether.  And when we move stuff to unittest maybe we could take the opportunity to clean up any warts in the interfaces.
msg101998 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-03-31 12:03
> Maybe we could move the appropriate stuff and get rid of the
> test_support doc page altogether.  And when we move stuff to unittest
> maybe we could take the opportunity to clean up any warts in the
> interfaces.

+1

I don't like the burden of backward compatibility concerns for this kind of test helpers.

The "Lib/test/" directory should be exclusively for internal Python tests.
It should be clearly stated in the documentation.

If some helpers are enough generic, they could be moved to unittest.support (or unittest.util).
msg101999 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-03-31 12:06
Right - that is exactly why I want a specific proposal as to which APIs are generally useful and should be considered for moving into unittest.
msg102001 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-03-31 12:11
Sorry, missed David's comment:

> Maybe we could move the appropriate stuff and get rid of the
> test_support doc page altogether.  And when we move stuff to unittest
> maybe we could take the opportunity to clean up any warts in the
> interfaces.

This I agree with. The question here is deciding what is "the appropriate stuff". 

If we have to maintain backwards compatibility in test_support we could just leave things in there... (possibly plus deprecation or deprecation via documentation.)
msg102002 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2010-03-31 12:55
Yes, let's make things explicit as David said:

- "test" package: tests for Python itself, with private stuff, undocumented. 

- "unittest" package: the test framework that is used by the test package and that is public

I'd add a support module in unittest, containing these (maybe renamed and consolidated) and deprecate them in test_support:

- is_jython
- TESTFN
- forget
- findfile
- check_warnings
- captured_stdout
- TransientResource
- transient_internet
- EnvironmentVarGuard
- WarningsRecorder
- cpython_only
- temp_cwd
- sortdict
msg102004 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-03-31 13:20
A 'support' sub-module in unittest is ok. I wonder if it is the best name?

- is_jython

This is too specific and ignores the other implementations. This information is also available from sys / platform modules. 

-1

- TESTFN

This is a global for setting the directory temporary files are created in? Don't think I like the global approach. Which functions is it used by?

- forget

This is the equivalent of sys.modules.pop(module_name, None) but it also deletes bytecode. I don't think a bytecode deleting API is needed in unittest.

-1


- findfile

I need to look at the code, don't understand it from the description in the docs.

- check_warnings

Could be useful.

- captured_stdout

Could be useful.

- TransientResource

I need to look at how it is used, but sounds useful.

- transient_internet

Not documented, I need to look at the code / how it is used.

- EnvironmentVarGuard

Could be useful.

- WarningsRecorder

Used by catch_warnings I guess.

- cpython_only

Not documented but could be useful.

- temp_cwd

Not documented but could be useful.

- sortdict

Not documented but could be useful.
msg102005 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2010-03-31 13:23
On 31/03/2010 14:20, Michael Foord wrote:

> - TESTFN
>
> This is a global for setting the directory temporary files are created in? Don't think I like the global approach. Which functions is it used by?

It's used *all over the place*. I started trying to rip it out
ages ago but it call got far too intrusive and I just backed
away. ISTR that more recent tests have used NamedTemporaryFile
or something similar, which makes much more sense.
msg102007 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2010-03-31 13:52
about : cpython_only / is_jython

I think the idea here is to mark some tests as being specific to some implementations.

I remember a discussion where we said that we could add in sys or platform the name of the implementation and make this somehow future-proof.

I can't recall where we said that...
msg102008 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-03-31 13:55
Something like this was discussed at the language summit. Now we have the skip decorators it is easy to skip tests on certain platforms or only run them on certain platforms.

The cpython_only decorator is fine - but I'd prefer a general solution rather than is_jython, is_pypy, is_ironpython, etc... Do we yet have a sys.implementation or equivalent? We need it. :-)
msg102026 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-03-31 18:22
I take the blame on documenting the handful of things in test.support; I thought it was a good idea to expose the handful of things that had an actual design to them. =)

But yes, we should probably shift to test.support to being private and simply exposing the stuff that we want to support to another module, although this stuff is not unittest-specific; its just testing utility functions. testlib or testutils would be a better place.

And we should make sure we are *very* comfortable supporting whatever we move over and that it has a good API as test.support has been a dumping ground for so long that most of that stuff should not get exposed (and honestly should get ripped out or cleaned up; TESTFN is just bad for instance).
msg102027 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-03-31 18:53
I think namespacing utility functions / classes inside unittest is fine (even good), but agree that there needs to be careful thinking about APIs that are made public in this way. To a certain extent this code has already proved its utility, but up until now it hasn't needed to be *generally* useful (and as brett says - some of the code in test_support would be better off dead).
msg102033 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-03-31 21:07
I agree that some things can be moved in unittest, however several of the function in test_support are designed specifically for being used for the python test suite. For example temp_cwd is something that could be really useful in unittest in my opinion, however we designed it looking at our use cases. We limited its utility to what we needed and we didn't document it so that we are free to change it if/when the use cases change.

On a side node, sortdict() is probably useless now that we have assertDictEqual().
msg103352 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-04-16 19:10
I'm writing unittests for a program I'm writing, and would really love to be able to use captured_stdout/stderr.  Of course, I have to support Python 2.5, and can't really justify installing unittest2, so I'll have to roll my own anyway, but I'm definitely +1 for those functions to move into unittest.  A bunch of the others, too, but that's the one I just looked at, and the API looks good to me.  (I'd rename captured_output and leave it be an implementation detail.)

How about 'tools' for the submodule name?
msg103353 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-04-16 19:26
-1 on moving the module into the mainstream without significant reworking and rethinking.  This module is a junk collection (I know because I've contributed some it).

The unittest module is important.  It needs a super clean, well thought out API.  The test_support module doesn't qualify.
msg103387 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-04-17 04:45
Not the module as it is, but just some of the APIs that it provides, possibly after a cleanup to remove things specific to our test suite and to make them more general and forward-compatible.
msg103388 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-04-17 04:51
If anything gets moved, it needs to be part of a very well thought-out plan with the same high threshold for approval that it would have had if it weren't in test_support.

The unittest module already brims with complexity and would suffer quite a bit if a hodge-podge of functions were added.
msg103612 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-04-19 16:07
Also note that the test_support module is for some reason documented here: http://docs.python.org/library/test.html#module-test.test_support
This means that there are probably developers out there that are already using these functions even if some of them are not intended/designed to be used outside the Python test suite.
This situation is sub-optimal for two reasons:
1) test_support can't evolve quickly because the documented functions have to be backward-compatible;
2) other developers are using these python-test-suite-specific functions in their tests;
It would be better to keep test_support for our test suite only and provide/move useful function in unittest.
msg115537 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-09-03 23:06
Issue #9754 proposes to implement assertWarns / assertWarnsRegexp
msg184576 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2013-03-19 01:02
If there are specific suggestions to move functionality from test support to unittest then they can be evaluated on a case-by-case basis. As a "general suggestion" I'm closing this issue.
History
Date User Action Args
2013-03-19 01:02:56michael.foordsetstatus: open -> closed
resolution: rejected
messages: + msg184576

stage: resolved
2013-03-04 14:42:35zach.waresetnosy: + zach.ware
2012-07-21 12:31:46floxsetversions: + Python 3.4, - Python 3.2, Python 3.3
2010-09-03 23:07:03floxsettitle: move generally useful test_support functions into the unittest package -> move generally useful test.support functions into the unittest package
2010-09-03 23:06:39floxsetdependencies: + assertWarns and assertWarnsRegexp
messages: + msg115537
2010-07-23 12:42:03eric.araujosetnosy: + eric.araujo
2010-04-19 17:17:56georg.brandlsetnosy: - georg.brandl
2010-04-19 16:10:35tim.goldensetnosy: - tim.golden
2010-04-19 16:07:30ezio.melottisetmessages: + msg103612
2010-04-17 04:51:59rhettingersetmessages: + msg103388
2010-04-17 04:45:08ezio.melottisetmessages: + msg103387
2010-04-16 19:27:55r.david.murraysettitle: move test_support into the unittest package -> move generally useful test_support functions into the unittest package
2010-04-16 19:26:22rhettingersetnosy: + rhettinger
messages: + msg103353
2010-04-16 19:10:16r.david.murraysetmessages: + msg103352
2010-03-31 21:07:53ezio.melottisetnosy: + ezio.melotti
messages: + msg102033
2010-03-31 18:53:35michael.foordsetmessages: + msg102027
2010-03-31 18:22:58brett.cannonsetnosy: + brett.cannon
messages: + msg102026
2010-03-31 13:55:12michael.foordsetmessages: + msg102008
2010-03-31 13:52:27tareksetmessages: + msg102007
2010-03-31 13:23:57tim.goldensetnosy: + tim.golden
messages: + msg102005
2010-03-31 13:20:26michael.foordsetmessages: + msg102004
2010-03-31 12:55:42tareksetmessages: + msg102002
2010-03-31 12:11:30michael.foordsetmessages: + msg102001
2010-03-31 12:06:32michael.foordsetmessages: + msg101999
2010-03-31 12:03:51floxsetnosy: + flox

messages: + msg101998
versions: + Python 3.2
2010-03-31 11:54:45r.david.murraysetpriority: low

nosy: + georg.brandl, r.david.murray
messages: + msg101997

type: enhancement
2010-03-31 11:35:03michael.foordsetmessages: + msg101996
2010-03-31 11:32:43tareksetmessages: + msg101995
2010-03-31 11:31:35michael.foordsetmessages: + msg101994
2010-03-31 11:27:30tareksetmessages: + msg101993
2010-03-31 11:23:25michael.foordsetmessages: + msg101992
2010-03-31 11:19:10tarekcreate