classification
Title: Clean-up Unittest API
Type: Stage: resolved
Components: Tests Versions: Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ezio.melotti Nosy List: brett.cannon, eric.araujo, ezio.melotti, flox, gregory.p.smith, michael.foord, pitrou, r.david.murray, rhettinger
Priority: normal Keywords:

Created on 2010-11-01 00:13 by rhettinger, last changed 2010-12-10 01:46 by gregory.p.smith. This issue is now closed.

Messages (29)
msg120100 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-11-01 00:13
* Dedocument assertSetEqual, assertDictEqual, assertListEqual, and assertTupleEqual.  These are all automatically dispatched from assertEqual.  The user need not call any of these directly.  These methods should not have been exposed in the docs.

* Add new names and dedocument old names for assertLess, assertLessEqual, assertGreater, assertGreaterEqual.   The new names are modeled after the gt, le, lt convention used elsewhere in the language.  This avoids to problems remembering the current spelling quirks (dropping the Than in LessEqual, pluralization, camel casing, being too long, etc).  New names will be assertLE, assertLT, assertGE, and assertGT.

* Add news names and dedocument assertRegexpMatches and assertNotRegexpMatches.  These names haves have multiple issues (they actually do a re.search not a re.match, they are long, the pluralization is inconsistent with the assertEqual convention, they don't agree with the names used in other unittest implementations such as PHPunit and Junit).  The new names will be assertRegexp and assertNotRegexp.

* Remove the assertItemsEqual method (which is new in 3.2).  Its semantics are not obvious from its name (i.e. duplicates matter, order does not matter, expects elements not items, has O(n**2) behavior after an impending bug fix, uses on equality for matching not hashing or ordering).  In most common cases, it is more explicit and flexible to use assertEqual after casting to a set, a list, or sorted list.  Also note that other unittest implementations such as PHPunit and JUnit do not have this method.  See http://mail.python.org/pipermail/python-dev/2010-October/105073.html 

* Recombine the package into a single file.  See http://mail.python.org/pipermail/python-dev/2010-October/105025.html and http://mail.python.org/pipermail/python-dev/2010-October/104887.html

* We need to review the camel cased spelling on two methods that are new in 3.2.  Should it be assertIsinstance or assertIsInstance?
 assert
msg120101 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-11-01 00:18
I would prefer assertRegex to assertRegexp.
msg120106 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-11-01 03:32
In general *none* of this should be done until there is clear consensus on Python-dev and it isn't clear that this is the case.

* On the deocumenting: barry warsaw objects to public apis that aren't documented and gregory smith asserts (natch) that it *can* be useful to call these methods directly for the explicit type checking they do. Let's see how this discussion pans out.

* For the greater / less comparison methods (etc) they would have to be new names as Python 2.7, 3.1 and unittest2 have the old names. This makes the TestCase even *bigger*. Let's see if we can get any consensus on this on Python-dev. Like Antoine I prefer assertRegex to assertRegexp.

* Note that assertItemsEqual is not new in 3.2 it is new in 2.7. If we remove it we make porting code using Python 2.7 (or earlier with unittest2) unnecessarily hard. The functionality is still useful but I agree that making the slow path more efficient would be good.

* Recombining the package into a single file *will not* happen without a BDFL pronouncement. unittest is massively easier to maintain now.

* Review the camel casing on assertIsInstance - isn't this just bike shedding?
msg120127 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-11-01 14:01
* I would leave assert<type>Equal documented and specify that they should be called directly only if the type should be checked, otherwise (if the type doesn't matter or it's already tested elsewhere) using assertEqual is enough (see also the suggestions in msg120124). -0.5 about dedocumenting, +1 on better documentation, possibly in an "advanced" section of the doc (I also find useful to know how assertEqual works and that I can register new functions like assert<type>Equal).

* assertLT and friends wouldn't match with assertEqual and assertNotEqual (assertEQ, assertNE?). Also these names don't seem more obvious or readable than assertLess, and will just add confusion and duplication of names, so -1.

* FWIW I like 'Regex' more than 'Regexp' too, assuming that it's worth renaming these methods and deprecate the old names (they are already in 2.7). Having a 'Match' or 'Search' suffix might clarify what the method does -- if it does what it says. +0 on the renaming (and +1 to 'Regex' if the rename happens).

* as I said on IRC, assertItemsEqual provides an useful functionality, and even if the name is not 100% clear, I'd rather double-check the doc than having to reimplement the functionality myself (or having a clumsy long-but-clear name). -1 on the removal, +1 on better documentation/performance.

* I think assertIsInstance is fine, however I wonder why the opposite is not assertIsNotInstance. Anyway it's probably not worth creating more confusion and adding new names just to change the spelling (they are in 2.7 too).
msg120188 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-11-01 23:34
Just my 2 cents:

- I think dedocumenting the type-specific methods is fine (it doesn't mean removing them, though); or perhaps relegate them to some advanced section

- It's unfortunate that the renaming suggestion comes so late; I'm not sure it's a good idea to rename lots of things after they were introduced (although I really don't like assertRegexpMatches)

- assertIsInstance looks more natural to me than the alternative

- I don't think there's any point in undoing the package splitting, since it makes Michael's work easier and other people didn't object (not people who actively contribute to unittest, that is)
msg120194 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-11-02 00:28
[Antoine]
> I would prefer assertRegex to assertRegexp

That makes sense.  It is the name used in other unittest implementations, it matches what the re module used to be called in Python, and it avoids issues with camel casing (i.e. Regexp vs RegExp).
msg120201 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2010-11-02 02:39
Just sent an email to python-dev, but since this issue sparked it, I might as well comment here: unittest shouldn't be made back into a single module.

Ignoring the fact that the file structure has nothing to do with the public API and so is orthogonal to the main thrust of this issue, changing it back will break imports in code from Python 2.7 that uses the more modular structure that now underlies unittest. Considering how much whining we get about code breakage any time Michael tries to fix anything in the module I think this is the last thing anyone wants to have happen.
msg120220 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-11-02 13:11
FWIW in addition to assertRegexpMatches and assertNotRegexpMatches there are 2 more methods that use "Regexp": assertRaisesRegexp and the newly introduced assertWarnsRegexp.
msg120221 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-11-02 13:14
@Ezio

Good catch. Even though several of us (including myself) prefer assertRegex over assertRegexp it is probably better to have consistent APIs otherwise people will never remember which methods have the 'p' and which don't.
msg120294 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-11-03 00:33
After discussion on python-dev, it seems that API lock-in precludes any change to the package structure.  So, the main proposals left are the addition of new better aliases for some of the functions and changing to the docs so that the new name is clear:

   assertLE(self, other)
      old name:  AssertLessEqual
      . . .

Also, I we should still dedocument assert<sometype>Equal and its brethren and leave assertEqual as the primary interface.  

At some point, we could add an option to assertEqual to allow user control of how failing results are displayed.  This would let us separate content from presentation (i.e. a==b vs how a diff would get presented).
msg120298 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-11-03 01:31
I don't think assertLE is enough of an improvement over assertLessEqual to be worth adding yet more deprecated names to unittest. So I'm -0 on this change in general.  (I'd be -1 except that it would be kind of nice to have the names be shorter :)

If the change is made, then assertEqual should become assertEQ, and assertNotEqual should become assertNE.

Another option is to go the other way, and change assertLess to assertLessThan.  That is, make the spelling consistently be the spelled out version of the abbreviation used in the special methods and elsewhere in the docs.  This would involve far fewer name changes.
msg120301 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-11-03 02:29
Besides being shorter, the advantages of assertLE are consistent with the rich comparison method names, no worries about Than, no camel casing ambiguities, no pluralization or other nmemonic issues (LessThanEqual or LessThanEquals or LessThanOrEqual or LessThanOrEqualTo).  The goal is to make sure you don't have to look up the spelling every time you use them.  

That being said, assertEqual can't change.  It has been around forever and is clear about what it does.

What I am sure about is that assertLess and assertLessEqual are very non-standard spellings and that infrequent users of comparison assertions won't remember them.
msg120323 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-11-03 13:01
assertEquals existed forever, too, but we deprecated that :) (with no intent to remove it, as I understand it).

There is no more ambiguity in "assertLessThan" than there is in assertLT, one just has more letters.  True, you have to look it up the first time, but you have to do that anyway, so I don't see that as a big deal.  Once you've looked it up the first time you know the naming rule and you can figure out all the other names.  (Which is not true now, since instead of 'LessThan' we have "Less", which does *not* correspond to to the special method name by a simple rule).
msg120357 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-11-03 23:52
Renaming and aliasing methods has a cost. It confuses users of the old names (including future users - the current API is now baked into django 1.3 unless I can get an update done in time for them to change the version they're using). People who use autocomplete or introspection (dir) to explore the APIs will also still come across the old names. Given that unittest has just changed a lot a further radical change is 'unfortunate'.

However, for both the regexp methods and the Items method the names are actually misleading.

For assertItemsEqual Raymond suggests assertElementCountsEqual. See issue 10242.

For the regexp methods we should use assertRegexp - this is shorter, not misleading and consistent with assertRaisesRegexp.

We should leave the gt / le / lt methods as they are. There is less consensus that these methods should be changed anyway.

The documentation can be improved by moving *all* the deprecated methods (new and old) out of the main documentation and into a separate section at the end of the documentation.

The type specific asserts we should also move into their own section and out of the main documentation. This means users who discover don't have to go to the source (there are online references) but they are out of the main documentation.

Hopefully everyone is only equally unhappy with this resolution.
msg120360 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-11-04 00:04
> For assertItemsEqual Raymond suggests assertElementCountsEqual. See
> issue 10242.

Why replace a long awkward name with an even longer and more awkward
name?
msg120363 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-11-04 00:27
An alternative would be adding check_order and check_duplicates to assertSameElements and de-deprecate it.

Python 2.7 could be left unchanged because it's too late to add/rename/deprecate methods (it has assertItemsEqual but not assertSameElements). Py3.2 would also be unchanged (assertItemsEqual and the deprecation of assertSameElements are both new in 3.2, assertSameElements has been introduced in 3.1).
They would end up with two different methods though.

The assertSameElements name seems quite good, and providing extra args will make it more flexible (it will cover all the 4 cases listed in msg100685) and the behavior would be clear without clumsy method names.
msg120369 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-11-04 01:25
> For the regexp methods we should use assertRegexp 

assertRegex is better.  it matches the name used in other unittest implementations, there is not confusion with camel casing RegExp vs Regexp, and it matches the former name of Python's own regex module.

> We should leave the gt / le / lt methods as they are. 

The assertLessEqual name is non-standard, it will never be spelled right by someone who hasn't used it recently.


> The documentation can be improved by moving *all* the deprecated 
> methods (new and old) out of the main documentation and
> into a separate section at the end of the documentation

See issue 10242 for a suggestion on how to handle renaming (document the new and old name in the same entry, noting that one name is old).


>> For assertItemsEqual Raymond suggests assertElementCountsEqual. 
> Why replace a long awkward name with an even longer 
> and more awkward name?

If length is an issue, assertCountsEqual will also do fine.  The important virtue is clarity.  The problem with ItemsEqual is that it doesn't tell you anything about what it does (unordered comparison where duplicates matter).  Also, the term "items" in Python usually means key/value pairs.  That is why we use the term "elements" in sets, itertools, etc.  In contrast, assertCountsEqual tells you what it does, compares the element counts.  It is the same as Counter(a)==Counter(b) for hashable elements and an equivalent for unhashable elements.
msg120370 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-11-04 01:54
assertCountsEqual is IMO much clearer than assertItemsCountsEqual (or however you spell it).  I was unclear on what the latter did, but the former is fairly clear.  Ezio's suggestion is also clearer.

Raymond, since you said 'never' your statement about assertLessEqual is clearly false :).  I don't have any problem remembering it (it's just __le__ spelled out using the standard unittest camelcase).  assertLess, on the other hand, I often get wrong, since it is *not* __lt__ spelled out.
msg120372 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-11-04 02:23
> assertLess, on the other hand, I often get wrong, 
> since it is *not* __lt__ spelled out.

Right.  Even Michael gets that one wrong.

Meditate on why Guido created the special method __lt__ instead of __less_than__.

Regardless of why Guido made that choice, we do already have a standard way to spell less-than in Python, and it would be good to stick with it.
msg120393 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-11-04 11:52
It should be noted that, if we re-use assertSameElements, the default behavior should be preserved for compatibility with 3.1, and that is different (and possibly less useful) than the one of assertItemsEqual. Ambiguities could be solved easily specifying the args explicitly every time though.

Regarding assertLT I'm still -1. The current names are imho the best compromise between being short and descriptive.
They also match nicely assertEqual (assertLT <-> assertEQ; assertLessThan <-> assertEqualTo; but assertLess <-> assertEqual).
Also it might not be immediately obvious what assertGE does, if one is not familiar with the special methods or if taken out of context (it's easy to recognize it if it's together with assertLT/LE/GT, but alone might just look a specialized method with a bad name).
Moreover people who are used to the current spelling will have to notice the change, note that one name is now deprecated, update their code, remember if the correct name is assertLess or assertLT, wonder if assertEqual has been deprecated in favor of assertEQ too, remember the version where the name changed (e.g. if they try assertLT on 3.1 it won't work) and so on.
msg120597 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-11-06 08:18
> people who are used to the current spelling 
> will have to notice the change, note that one 
> name is now deprecated

I haven't proposed any deprecations.
Just add the new names as aliases.
Change the docs list the new names
as primary and mention the old name for reference.

We don't have to sticking with crummy naming.
msg122413 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-11-25 22:16
After discussion with Michael and Guido, am limiting this to:

* Fixing assertItemsEqual as described in issue10242

* Moving the docs for type specific equality methods inside the docs for assertEqual to emphasize that those get dispatched automatically and need not be called directly.

* Changing assertRegexpMatches to assertRegex
msg122416 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-11-25 22:56
> * Moving the docs for type specific equality methods inside the docs for assertEqual to emphasize that those get dispatched automatically and need not be called directly.

I already fixed this on py3k, adding a section where the type-specific methods are described[0] and added a reference in the assertEqual description[1].  I also clarified that usually there's no need to call them directly even thought I don't see why they shouldn't do it if they want to use a specific test.

> * Changing assertRegexpMatches to assertRegex

And assertNotRegexpMatches -> assertNotRegex,  assertRaisesRegexp -> assertRaisesRegex,  assertWarnsRegexp -> assertWarnRegex?
Will the *Regexp forms be deprecated too?

[0]: http://docs.python.org/dev/py3k/library/unittest.html#type-specific-methods
[1]: http://docs.python.org/dev/py3k/library/unittest.html#unittest.TestCase.assertEqual
msg122425 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-11-26 01:12
Yes, all the variants of RegexpMatches --> Regex 
No, on deprecations.  Just add a new alias and note in the docs that the oldname is obsolete.  Naming deprecations cause too much trouble for too little benefit.
msg122434 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-11-26 03:59
If I implement what I suggested in #10535, it will be possible to deprecate them without too much trouble. I offer to do it after #10535.
msg122758 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-11-29 02:39
Ezio, please do the regexp-->regex changes and move the tests under Lib/test.
msg122788 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-11-29 11:24
Raymond - I created a new issue for moving the tests: issue 10572

However, it seems that you are incorrect in saying that Python practise is to avoid putting tests inside standard library packages. In fact current Python practise seems to be that where tests themselves are a package they are *always* inside the standard library package and *not* inside Lib/test. For example:  email, distutils, ctypes, importlib, json, lib2to3, sqlite3, tkinter, 

I couldn't see any counter-examples. There are no test packages inside Lib/test (other than leakers that are obviously a category of tests and not tests for a particular package).
msg122971 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-12-01 02:35
s/regexp/regex/ done in r86910.
msg123713 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-12-10 01:46
fyi - since I didn't chime in earlier on this: I think you made the right choice with what was decided in msg122413 and implemented in the renaming done in r86910 and the work done in issue10242.
History
Date User Action Args
2010-12-10 01:46:44gregory.p.smithsetmessages: + msg123713
2010-12-01 02:35:37ezio.melottisetstatus: open -> closed
resolution: fixed
messages: + msg122971

stage: resolved
2010-11-29 11:24:13michael.foordsetmessages: + msg122788
2010-11-29 02:39:09rhettingersetassignee: rhettinger -> ezio.melotti
messages: + msg122758
2010-11-26 03:59:34ezio.melottisetmessages: + msg122434
2010-11-26 01:12:06rhettingersetmessages: + msg122425
2010-11-25 22:56:45ezio.melottisetmessages: + msg122416
2010-11-25 22:16:55rhettingersetmessages: + msg122413
2010-11-06 08:18:19rhettingersetmessages: + msg120597
2010-11-06 05:26:39eric.araujosetnosy: + eric.araujo
2010-11-04 14:50:07ezio.melottisetnosy: + gregory.p.smith, flox
2010-11-04 11:53:00ezio.melottisetmessages: + msg120393
2010-11-04 02:23:59rhettingersetmessages: + msg120372
2010-11-04 01:54:31r.david.murraysetmessages: + msg120370
2010-11-04 01:25:43rhettingersetmessages: + msg120369
2010-11-04 00:27:02ezio.melottisetmessages: + msg120363
2010-11-04 00:04:30pitrousetmessages: + msg120360
2010-11-03 23:52:33michael.foordsetmessages: + msg120357
2010-11-03 13:01:23r.david.murraysetmessages: + msg120323
2010-11-03 02:29:49rhettingersetmessages: + msg120301
2010-11-03 01:31:42r.david.murraysetnosy: + r.david.murray
messages: + msg120298
2010-11-03 00:33:52rhettingersetmessages: + msg120294
2010-11-02 13:14:53michael.foordsetmessages: + msg120221
2010-11-02 13:11:37ezio.melottisetnosy: brett.cannon, rhettinger, pitrou, ezio.melotti, michael.foord
messages: + msg120220
components: + Tests
2010-11-02 02:39:40brett.cannonsetnosy: + brett.cannon
messages: + msg120201
2010-11-02 00:29:17rhettingersetassignee: michael.foord -> rhettinger
2010-11-02 00:28:58rhettingersetmessages: + msg120194
2010-11-01 23:34:00pitrousetmessages: + msg120188
2010-11-01 14:01:19ezio.melottisetmessages: + msg120127
2010-11-01 03:32:33michael.foordsetassignee: rhettinger -> michael.foord
messages: + msg120106
2010-11-01 00:18:09pitrousetnosy: + pitrou
messages: + msg120101
2010-11-01 00:15:16ezio.melottisetnosy: + ezio.melotti, michael.foord
2010-11-01 00:13:41rhettingercreate