Let's move test_support in unittest !

Then maybe, let's expose some of test_support functions into a new class on the top of unittest.TestCase, so they can be used via methods.
 
The purpose is power up people when it comes to write test fixtures or work in a testing environment. These helpers were built to test Python itself, are quite unknown out there. I think it's a shame :)

http://docs.python.org/library/test.html#module-test.test_support

Hmmm... I'm not sure moving everything as is is a good idea. Any parts in particular you want?

Basically, all the APIs in test_support could live in unittest and the test package could only contain the test modules written for Python.

IOW, these tests helpers can help more people than the core devs

Not all the APIs are appropriate though, for example TestFailed and TestSkipped are not needed. Some of the APIs only make sense for the test_support runner and unittest has its own runner now.

I think the proposal needs to be more specific about which APIs you are suggesting would be generally useful.

Sure, no need to have the test runner parts.

So which parts, can you list the ones you think are most important?

At the moment we don't have to maintain backwards compatibility with test.test_support as it isn't a 'public' API - so the cost in adding it to the standard library is the same as adding any code. Once we do this we are less free to modify / evolve the API.

Actually, the list Terek posted a link to is a public doc page, and as was made clear in the recent issue about the check_warnings helper, we do have to maintain backward compatibility.  

There (should be) undocumented stuff in test_support, purposely left undocumented so that we *don't* have to maintain backward compatibility.  Myself I wish that doc page didn't even exist.  Putting stuff that users can use into unittest is a much better idea.

Maybe we could move the appropriate stuff and get rid of the test_support doc page altogether.  And when we move stuff to unittest maybe we could take the opportunity to clean up any warts in the interfaces.

> Maybe we could move the appropriate stuff and get rid of the
> test_support doc page altogether.  And when we move stuff to unittest
> maybe we could take the opportunity to clean up any warts in the
> interfaces.

+1

I don't like the burden of backward compatibility concerns for this kind of test helpers.

The "Lib/test/" directory should be exclusively for internal Python tests.
It should be clearly stated in the documentation.

If some helpers are enough generic, they could be moved to unittest.support (or unittest.util).

Right - that is exactly why I want a specific proposal as to which APIs are generally useful and should be considered for moving into unittest.

Sorry, missed David's comment:

> Maybe we could move the appropriate stuff and get rid of the
> test_support doc page altogether.  And when we move stuff to unittest
> maybe we could take the opportunity to clean up any warts in the
> interfaces.

This I agree with. The question here is deciding what is "the appropriate stuff". 

If we have to maintain backwards compatibility in test_support we could just leave things in there... (possibly plus deprecation or deprecation via documentation.)

Yes, let's make things explicit as David said:

- "test" package: tests for Python itself, with private stuff, undocumented. 

- "unittest" package: the test framework that is used by the test package and that is public

I'd add a support module in unittest, containing these (maybe renamed and consolidated) and deprecate them in test_support:

- is_jython
- TESTFN
- forget
- findfile
- check_warnings
- captured_stdout
- TransientResource
- transient_internet
- EnvironmentVarGuard
- WarningsRecorder
- cpython_only
- temp_cwd
- sortdict

A 'support' sub-module in unittest is ok. I wonder if it is the best name?

- is_jython

This is too specific and ignores the other implementations. This information is also available from sys / platform modules. 

-1

- TESTFN

This is a global for setting the directory temporary files are created in? Don't think I like the global approach. Which functions is it used by?

- forget

This is the equivalent of sys.modules.pop(module_name, None) but it also deletes bytecode. I don't think a bytecode deleting API is needed in unittest.

-1


- findfile

I need to look at the code, don't understand it from the description in the docs.

- check_warnings

Could be useful.

- captured_stdout

Could be useful.

- TransientResource

I need to look at how it is used, but sounds useful.

- transient_internet

Not documented, I need to look at the code / how it is used.

- EnvironmentVarGuard

Could be useful.

- WarningsRecorder

Used by catch_warnings I guess.

- cpython_only

Not documented but could be useful.

- temp_cwd

Not documented but could be useful.

- sortdict

Not documented but could be useful.

On 31/03/2010 14:20, Michael Foord wrote:

> - TESTFN
>
> This is a global for setting the directory temporary files are created in? Don't think I like the global approach. Which functions is it used by?

It's used *all over the place*. I started trying to rip it out
ages ago but it call got far too intrusive and I just backed
away. ISTR that more recent tests have used NamedTemporaryFile
or something similar, which makes much more sense.

about : cpython_only / is_jython

I think the idea here is to mark some tests as being specific to some implementations.

I remember a discussion where we said that we could add in sys or platform the name of the implementation and make this somehow future-proof.

I can't recall where we said that...

Something like this was discussed at the language summit. Now we have the skip decorators it is easy to skip tests on certain platforms or only run them on certain platforms.

The cpython_only decorator is fine - but I'd prefer a general solution rather than is_jython, is_pypy, is_ironpython, etc... Do we yet have a sys.implementation or equivalent? We need it. :-)

I take the blame on documenting the handful of things in test.support; I thought it was a good idea to expose the handful of things that had an actual design to them. =)

But yes, we should probably shift to test.support to being private and simply exposing the stuff that we want to support to another module, although this stuff is not unittest-specific; its just testing utility functions. testlib or testutils would be a better place.

And we should make sure we are *very* comfortable supporting whatever we move over and that it has a good API as test.support has been a dumping ground for so long that most of that stuff should not get exposed (and honestly should get ripped out or cleaned up; TESTFN is just bad for instance).

I think namespacing utility functions / classes inside unittest is fine (even good), but agree that there needs to be careful thinking about APIs that are made public in this way. To a certain extent this code has already proved its utility, but up until now it hasn't needed to be *generally* useful (and as brett says - some of the code in test_support would be better off dead).

I agree that some things can be moved in unittest, however several of the function in test_support are designed specifically for being used for the python test suite. For example temp_cwd is something that could be really useful in unittest in my opinion, however we designed it looking at our use cases. We limited its utility to what we needed and we didn't document it so that we are free to change it if/when the use cases change.

On a side node, sortdict() is probably useless now that we have assertDictEqual().

I'm writing unittests for a program I'm writing, and would really love to be able to use captured_stdout/stderr.  Of course, I have to support Python 2.5, and can't really justify installing unittest2, so I'll have to roll my own anyway, but I'm definitely +1 for those functions to move into unittest.  A bunch of the others, too, but that's the one I just looked at, and the API looks good to me.  (I'd rename captured_output and leave it be an implementation detail.)

How about 'tools' for the submodule name?

-1 on moving the module into the mainstream without significant reworking and rethinking.  This module is a junk collection (I know because I've contributed some it).

The unittest module is important.  It needs a super clean, well thought out API.  The test_support module doesn't qualify.

Not the module as it is, but just some of the APIs that it provides, possibly after a cleanup to remove things specific to our test suite and to make them more general and forward-compatible.

If anything gets moved, it needs to be part of a very well thought-out plan with the same high threshold for approval that it would have had if it weren't in test_support.

The unittest module already brims with complexity and would suffer quite a bit if a hodge-podge of functions were added.

Also note that the test_support module is for some reason documented here: http://docs.python.org/library/test.html#module-test.test_support
This means that there are probably developers out there that are already using these functions even if some of them are not intended/designed to be used outside the Python test suite.
This situation is sub-optimal for two reasons:
1) test_support can't evolve quickly because the documented functions have to be backward-compatible;
2) other developers are using these python-test-suite-specific functions in their tests;
It would be better to keep test_support for our test suite only and provide/move useful function in unittest.

Issue #9754 proposes to implement assertWarns / assertWarnsRegexp

If there are specific suggestions to move functionality from test support to unittest then they can be evaluated on a case-by-case basis. As a "general suggestion" I'm closing this issue.