This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Documentation of assertItemsEqual in unittest is VERY misleading in 2.7
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, michael.foord, terry.reedy, vitaly
Priority: normal Keywords:

Created on 2016-05-19 20:05 by vitaly, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (9)
msg265889 - (view) Author: Vitaly (vitaly) Date: 2016-05-19 20:05
Python 2.7 documentation is VERY misleading about the functionality of assertItemsEqual. The implementation only compares item counts, but the documentation actually claims to compare not only the counts, but the actual sorted elements themselves. This documentation mislead my group to use this method for comparing the elements. See, which is what appears on

   .. method:: assertItemsEqual(actual, expected, msg=None)

      Test that sequence *expected* contains the same elements as *actual*,
      regardless of their order. When they don't, an error message listing the
      differences between the sequences will be generated.

      Duplicate elements are *not* ignored when comparing *actual* and
      *expected*. It verifies if each element has the same count in both
      sequences. It is the equivalent of ``assertEqual(sorted(expected),
      sorted(actual))`` but it works with sequences of unhashable objects as
msg265969 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-05-20 22:30
Note: 2.7 doc says that this is renamed assertCountEqual in 3.x.  It was added in 2.7 and 3.2.  The 3.x doc is essentially the same, but with this instead: "Equivalent to: assertEqual(Counter(list(first)), Counter(list(second))) ...".

If the method misfunctions, then it should be fixed, not have the bug documented.  But you need to provide specific evidence and example.  The unittest in 3.x has this test with same number, different elements.
        self.assertRaises(self.failureException, self.assertCountEqual,
                          [1, 2] + [3] * 100, [1] * 100 + [2, 3])

I ran following test with same number, different items.
FAIL: test_Count (__main__.Test)
Traceback (most recent call last):
  File "F:\Python\mypy\", line 4, in test_Count
    self.assertItemsEqual([1,2,3], [1,2,4])
AssertionError: Element counts were not equal:
First has 1, Second has 0:  3
First has 0, Second has 1:  4
This also fails in 3.5 with name change.
msg265970 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-05-20 22:31
Here is the specific test code so you can replace my lists with examples that you think malfunction.

import unittest as u
class Test(u.TestCase):
    def test_Count(self):
        self.assertItemsEqual([1,2,3], [1,2,4])
msg265975 - (view) Author: Vitaly (vitaly) Date: 2016-05-20 23:22
According to comment in #17866, the maintainers renamed `assertItemsEqual` to `assertCountEqual` somewhere in the 3.x development timeframe. They apparently latched on to the implementation of `assertItemsEqual` in 2.7.x, which compares counts only, rather than its documentation.
msg265977 - (view) Author: Vitaly (vitaly) Date: 2016-05-20 23:37
Thanks Terry, your assessment is correct. The maintainers appear to have drawn the wrong conclusion in #17866, which mislead me about the existing assertItemsEqual in 2.7, which appears to work correctly after all!

import unittest as u
class Test(u.TestCase):
    def test_equal_count_of_same_elements(self):
        self.assertItemsEqual([1,2,3], [1,2,3])

    def test_equal_count_of_different_elements(self):
        self.assertItemsEqual([1,2,3], [1,2,4])

    def test_different_count(self):
        self.assertItemsEqual([1,2,3], [1,2,3,4])

$ python --verbose
test_different_count (__main__.Test) ... FAIL
test_equal_count_of_different_elements (__main__.Test) ... FAIL
test_equal_count_of_same_elements (__main__.Test) ... ok

FAIL: test_different_count (__main__.Test)
Traceback (most recent call last):
  File "", line 12, in test_different_count
    self.assertItemsEqual([1,2,3], [1,2,3,4])
AssertionError: Element counts were not equal:
First has 0, Second has 1:  4

FAIL: test_equal_count_of_different_elements (__main__.Test)
Traceback (most recent call last):
  File "", line 8, in test_equal_count_of_different_elements
    self.assertItemsEqual([1,2,3], [1,2,4])
AssertionError: Element counts were not equal:
First has 1, Second has 0:  3
First has 0, Second has 1:  4

Ran 3 tests in 0.001s

FAILED (failures=2)
msg265978 - (view) Author: Vitaly (vitaly) Date: 2016-05-20 23:46
I think that we can close this as not a bug, as concerning 2.7. However, I think that 3.x made the wrong decision when it renamed assertItemsEqual with assertCountEqual, which now misleads users into thinking that it compares counts.
msg265981 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-05-20 23:55
It compares element by element counts, as indicated by "Equivalent to: assertEqual(Counter(list(first)), Counter(list(second))) ..." and the failure messages.  Since I understand Counter equality, I ignore the rest.
msg265982 - (view) Author: Vitaly (vitaly) Date: 2016-05-21 00:05
Yes, this caused me to examine Counter, which I had not used before, and now I understand it, too. However, the new name by itself does not sufficiently reflect the subtlety of element-by-element counts, which does a disservice to the readability of test code.
msg265983 - (view) Author: Vitaly (vitaly) Date: 2016-05-21 00:06
... and an unnecessary incompatibility between 2.7 and 3.x
Date User Action Args
2022-04-11 14:58:31adminsetgithub: 71247
2016-05-21 00:06:25vitalysetmessages: + msg265983
2016-05-21 00:05:51vitalysetmessages: + msg265982
2016-05-20 23:55:47terry.reedysetstatus: open -> closed
resolution: not a bug
messages: + msg265981

stage: test needed -> resolved
2016-05-20 23:46:14vitalysetmessages: + msg265978
2016-05-20 23:37:59vitalysetmessages: + msg265977
2016-05-20 23:22:17vitalysetmessages: + msg265975
2016-05-20 22:31:54terry.reedysetmessages: + msg265970
2016-05-20 22:30:14terry.reedysetnosy: + terry.reedy
messages: + msg265969

components: + Library (Lib), - Documentation
type: behavior
stage: test needed
2016-05-19 20:17:47georg.brandlsetnosy: + michael.foord
2016-05-19 20:05:37vitalysetassignee: docs@python

components: + Documentation
nosy: + docs@python
2016-05-19 20:05:23vitalysetversions: + Python 2.7
2016-05-19 20:05:14vitalycreate