This is working as designed. assertCountEqual is documented here:

https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertCountEqual

It says: "Test that sequence *first* contains the same elements as *second*..." notice that it talks about *sequences*, not mappings. The (approximate) equivalent code is also given:

    assertEqual(Counter(list(first)), Counter(list(second)))

If the arguments are dicts, only the keys are compared. The example you give correctly passes, because it is equivalent to calling `assertCountEqual(first.keys(), second.keys())` and the keys are equal.

If you want to compare the items, you can call `assertCountEqual(first.items(), second.items())`.

The example comparing lists correctly fails because the list elements are different.