Found as a comment on SO (https://stackoverflow.com/questions/12813633/how-to-assert-two-list-contain-the-same-elements-in-python#comment104082703_31832447):

In unittest, `self.assertCountEqual({1: [1, 2, 3]}, {1: [5, 6, 7]})` succeeds, even though the two are different.
In this simple case, using assertCountEqual is unnecessary, but there may be cases where a user wants to test for general equality regardless of order.

Note that `self.assertCountEqual([{1: [1, 2, 3]}], [{1: [5, 6, 7]}])` (where both are a list, with only a dict-element), does fail.
And comparing 2 dicts with different keys also fails as expected.

This is working as designed. assertCountEqual is documented here:

https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertCountEqual

It says: "Test that sequence *first* contains the same elements as *second*..." notice that it talks about *sequences*, not mappings. The (approximate) equivalent code is also given:

    assertEqual(Counter(list(first)), Counter(list(second)))

If the arguments are dicts, only the keys are compared. The example you give correctly passes, because it is equivalent to calling `assertCountEqual(first.keys(), second.keys())` and the keys are equal.

If you want to compare the items, you can call `assertCountEqual(first.items(), second.items())`.

The example comparing lists correctly fails because the list elements are different.