The attached script, named find_duplicate_test_names.py, prints
duplicate regression test method names in a given directory tree.
Running this script on the standard library test suite shows few
duplicates, see below.  It means that some of those tests are not
run while they are probably meant to be.

Here is find_duplicate_test_names.py output on the default branch:

$ ./python find_duplicate_test_names.py Lib/test
Duplicate test method names:
Lib/test/test_types.py: <module>.ClassCreationTests.test_new_class_exec_body
Lib/test/test_heapq.py: <module>.TestErrorHandling.test_get_only
Lib/test/test_complex.py: <module>.ComplexTest.test_truediv
Lib/test/test_webbrowser.py: <module>.OperaCommandTest.test_open_new
Lib/test/test_dis.py: <module>.DisTests.test_big_linenos
Lib/test/test_dis.py: <module>.DisTests.test_dis_object
Lib/test/test_import.py: <module>.ImportTests.test_import_name_binding
Lib/test/test_email/test_email.py: <module>.TestQuopri.test_encode_one_long_line
Lib/test/test_email/test_email.py: <module>.TestLongHeaders.test_splitter_split_on_punctuation_only_if_fws

Thanks a lot for the report.  Does the 2.7 branch have similar instances?

Running find_duplicate_test_names.py (after changing the print
statements) on the 2.7 branch, gives the following output:

$ ./python find_duplicate_test_names.py Lib/test/
Duplicate test method names:
Lib/test/test_unicode.py: <module>.UnicodeTest.test_capitalize
Lib/test/test_iter.py: <module>.TestCase.test_unicode_join_endcase

The duplicate in Lib/test/test_iter.py is defined on purpose.

The script could be actually be added somewhere (e.g. Tools/scripts) and run as part of `make patchcheck` on the test files that got changed.

The attached patch uses the infrastructure of patchcheck.py and
merges the script into patchcheck.py instead of adding a new script.

patchcheck output with the patch applied:

$ make patchcheck
./python ./Tools/scripts/patchcheck.py
Getting the list of files that have been added/changed ... 1 file
Fixing whitespace ... 0 files
Fixing C file whitespace ... 0 files
Fixing docs whitespace ... 0 files
Duplicate test names found in ... 1 file:
  Lib/test/test_heapq.py: test name 'TestErrorHandling.test_get_only'
Docs modified ... NO
Misc/ACKS updated ... NO
Misc/NEWS updated ... NO
configure regenerated ... not needed
pyconfig.h.in regenerated ... not needed

Did you run the test suite?

New changeset e99c0425da59 by Benjamin Peterson in branch 'default':
renmae test method to avoid conflict (#16056)
http://hg.python.org/cpython/rev/e99c0425da59

To simplify and keep the discussions more focused, etc, I would create a new issue for the patch to patchcheck (and mark it "enhancement").  Both issues can still reference each other.

> To simplify and keep the discussions more focused, etc, I would
> create a new issue for the patch to patchcheck

New issue 16079 has been created.

The proposed patch in the new issue 16079 is slightly improved to
produce a cleaner output by printing the number of duplicate tests
and not the number of files where there are duplicates.

New changeset 6d44906344f4 by Berker Peksag in branch '3.4':
Issue #16056: Rename test method in test_statistics to avoid conflict.
https://hg.python.org/cpython/rev/6d44906344f4

New changeset c49d7f4d1c04 by Berker Peksag in branch 'default':
Issue #16056: Rename test method in test_statistics to avoid conflict.
https://hg.python.org/cpython/rev/c49d7f4d1c04

- issue 16079 opened for "make patchcheck" integration
- issue 19119 opened for test_heapq
- issue 19113 opened for test_functions

And here's a patch for 2.7.

I think my patch is wrong.

test_weak_destroy_while_iterating and test_weak_destroy_and_mutate_while_iterating tests were committed as part of issue 7105 to 2.7 (see changeset https://hg.python.org/cpython/rev/03fcc12282fc). But, those same tests(they uses SomeClass instead of UserString) were also backported to 2.7 in https://hg.python.org/cpython/rev/04cc075bddf5

Here is a new patch.

New changeset fd786e4e331c by Berker Peksag in branch '2.7':
Issue #16056: Rename test methods to avoid conflict.
https://hg.python.org/cpython/rev/fd786e4e331c