In the discussion about adding support for parameterized tests (issue 7897), it seemed clear that parameterizing individual tests was a different issue from parameterizing TestCases. This, then, is a request to support parameterization of TestCases.

The fundamental idea is that one should be able to define a TestCase - fixtures, individual tests, etc. - and then be able to reuse that TestCase with different sets of parameters. This should all mesh cleanly with the rest of the unittest system, though it's not entirely clear what that entails.

As a motivation, consider a TestCase that tests the public API for a database abstraction system. The abstraction may work against any of a number of backends, but the public API remains the same for each. A parameterized TestCase would let you write one test for the public API and then run it for each of the backends.

Note that this is fairly simple to do now via subclassing, so any proposed API would need to show a clear benefit over that approach to be worth the extra complexity in the unittest code base.

> Note that this is fairly simple to do now via subclassing, so any
> proposed API would need to show a clear benefit over that approach to
> be worth the extra complexity in the unittest code base.

Agreed. Let's not add cruft to unittest.

Yeah, without some clear advantages and a clean api / implementation I'm -1.

David, is this the sort of thing you mean?

@skip # so do not run without backend
class AbstractDB2Testcase:
  backend = None
  <code>

class PostgressDB2Testcase(AbstractDB2Testcase):
  backend = postgress # well, enough info to fine it

...

If so, I think we should close this.

Yes, except that it would be:

   class PostgressDB2Testcase(AbstractDB2Testcase, unittest.TestCasse):

The fact that other test frameworks have found it worth implementing indicates there *might* be something worthwhile there, but unless someone makes the case, yes we should close this.

Yes, in some sense that's what I'm thinking of. But one problem with
this straightforward approach is that it doesn't scale well. If I've
got many TestCases, each if which I want to parameterize, I have to
create subclasses for each parameterization. If I add a new
parameterization, of course I have to add new subclasses for each.
Even worse, if my parameterizations are dynamically generated (e.g. if
my parameterizations are based on installed plugins or something),
then this naive approach just breaks down.

There are, as has been mentioned, various ways to implement quite
sophisticated parameterized test cases purely by subclassing the
existing loaders, runner, and TestCase functionality. On the other
hand (and recognizing that I may well be missing something), it's not
particularly straightforward to do so, and it feels like a bit more of
a hack than I would like. There are a lot of moving parts in unittest,
and when I have to start fiddling with one to get special new behavior
I always worry that I might subtly break something else...but maybe
I'm just being neurotic ;)

So, I'll put forward two arguments for official support for
parameterized test cases. The first is what I mentioned above:
implementing it extrinsic to the existing system seems non-obvious and
complex, and thus error prone. Putting official support in unittest
would mean nobody else ever had to worry about it. On the other hand,
assuming that parameterized test cases can be fully and cleanly
implemented without touching unittest, this also argues that someone
should just create an external package to do so, and leave unittest
alone.

The second argument is symmetry. If you implement parameterization of
individual tests, it seems like the logic for justifying them really
could be generalized for collections of tests, both "cases" and
"suites". I admit that this is just a gut feeling, but there you go.

Ultimately, I'm less concerned that any changes are made to unittest
than I am that the issue is given due consideration. Heck, a proper
"fix" might be to just add a section to the docs describing how one
might implement parameterized testcases.

On Sat, Jul 23, 2011 at 12:04 AM, Terry J. Reedy <report@bugs.python.org> wrote:
>
> Terry J. Reedy <tjreedy@udel.edu> added the comment:
>
> David, is this the sort of thing you mean?
>
> @skip # so do not run without backend
> class AbstractDB2Testcase:
>  backend = None
>  <code>
>
> class PostgressDB2Testcase(AbstractDB2Testcase):
>  backend = postgress # well, enough info to fine it
>
> ...
>
> If so, I think we should close this.
>
> ----------
> nosy: +terry.reedy
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue12600>
> _______________________________________
>

Having a "TestCase factory" would be pretty easy, and solve the scaling problems.

For example:

def make_testcase_classes():
    for backend in backends:
        yield type(
            '{}Test'.format(backend.name),
            (TheBaseClass, unittest.TestCase),
            {'backend': backend}
        )

You would use this in the load_tests function at the module level to generate all the test cases.

The lack of the ability to pass a parameter to a test case is a very
frustrating restriction with unittest. The frequent need if for a database
connection for testing database related classes.

Yes, there are lots of other ways to work around it, but they tend to involved need to understand and subclass several pieces of the unittest
framework.

An enthusiastic yes on this.

Drat, the tracker lost my post.  In summary, given a concrete use case (running a test case with a variety of different DB connections) and the improved readablility for the common case of just changing class constants in the 'parameterized' subclasses, I'd change to being a +0 on this.

Mark, would you like to work on a patch for this?

Why not create a simple TestCase factory in load_tests?

Before a patch can be produced a clean api that offers a clear benefit over the TestCase factory needs to be proposed.

Maybe we could add a recipe for doing this to the load_tests docs?

I don't think that load_tests is going to be more readable, though, since it doesn't allow you to put the parameterization next to the class you are parameterizing (unless you do some additional hackery).

But yes, if anything else is done a concrete API proposal is the first requirement.

> R. David Murray <rdmurray@bitdance.com> added the comment:
> 
> Meaning you want to run the same test suite with a variety of
> different DB connections?  That seems like a reasonable use
> case.

This is for external parameterization of a test to run in a
different environment.  Internal reuse of test code is usually
better done inside of the code using standard OOP approaches.

The different in database connections is due to wanting to run
the tests on different systems with different database users,
passwords and database names.  Normally we have an object that
reads this information from file specified on the command line,
the object is passed to code that creates the connections.

A similar problem exists when tests must be run against a server
and require a host name.  External parameterization to specify
the host name is required.

While it's certainly possible to come up with says to pass this
setting things in globals (including environment variable), this
does lead to confusing code.

Mark

That's not the kind of parameterization this ticket is about, though.  You are talking about passing data in to a test run from the command line (or other source), which is a different issue (though the implementations might share some common infrastructure).

Allowing loadTestsFromTestCase() to take either a testCaseClass
or a (testCaseClass, param) tuple, where the param is then past
to the __init__ function might do the trick.  One param is sufficient, since it can
be a container for any number of params. This allows more fields
to be added to the tuple for some future, unforeseen need.

An container object for class and parameters would be a bit more
structured than a tuple.

A factory function would be the most flexible, however it seems
to go against the class introspection for discovering tests.
Then again, I don't know the code very well.

R. David Murray <report@bugs.python.org> writes:
> 
> R. David Murray <rdmurray@bitdance.com> added the comment:
> 
> Maybe we could add a recipe for doing this to the load_tests docs?
> 
> I don't think that load_tests is going to be more readable, though, since it doesn't allow you to put the parameterization next to the class you are parameterizing (unless you do some additional hackery).
> 
> But yes, if anything else is done a concrete API proposal is the first requirement.
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue12600>
> _______________________________________

Mark, please stop discussing per-run parameters in this issue. Those are NOT the kind of parameters we're talking about (and are easily handled via a global settings module, anyway, the exact same way you can handle process global settings for *any* kind of application). If you want to continue discussing that topic, please create a new issue.

This issue is about parameterisation of a TestCase *within* a run, where the same set of tests is run against multiple sets of parameters, and the test framework provides good support for collating and presenting those results in a meaningful fashion.

Back on topic...

While I can see the advantage of parameterisation at the level of individual tests, I'm not at all clear on the benefits at the TestCase level.

For CPython's own test suite, if we want to share tests amongst multiple test cases, we just use ordinary inheritance. You get parameterisation pretty much for free with that approach:

class _BaseTest(object):
    # Tests go here
    # setUp and tearDown often go here, too

class FooTestCase(_BaseTest, TestCase):
    # Parameter settings go here

class BarTestCase(_BaseTest, TestCase):
    # Parameter settings go here

If you want to get data-driven about it, you can also do dynamic TestCase creation based on a sequence of parameter sets.

So, absent a compelling explanation for why the ordinary inheritance mechanisms aren't adequate, I'd be in favour of closing this one.

I'd still like to see a recipe for creating parameterized test cases via load_tests added to the docs.  It may be relatively obvious how to do it once you think of it, but it isn't obvious to a relative newcomer that you *can* do it, and it would make a great example of how load_tests can be used.  (The current example is very trivial since it just re-implements the default behavior, and while that's useful, it doesn't really demonstrate the power of load_tests).

I agree with David, so switching this over to a docs enhancement request.

BTW I'm very happy with testscenarios (on pypi) for this, modulo one small issue which is the use of __str__ by the stdlib [which isn't easily overridable - there is a separate issue on that]. I'd be delighted to have testscenarios be in the stdlib, if thats desirable.

Any volunteers to do this?  I'd do it myself but by the time somebody explains the detail, it'd probably be easier just to write a patch.  

@Robert just FTR do you know the issue number for "the use of __str__ by the stdlib [which isn't easily overridable - there is a separate issue on that]" which you mention in msg177328?

It remains to update the load_test doc along the lines of Michael's suggestion in msg140999.