A few tests fail in the coverage Travis target (see e.g. https://travis-ci.org/python/cpython/jobs/206480468): test_traceback and test_xml_etree.

I extracted the actual failures by running in verbose mode locally:

======================================================================
FAIL: test_recursive_traceback_cpython_internal (test.test_traceback.TracebackFormatTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jelle/cpython-dev/cpython-1/Lib/test/test_traceback.py", line 431, in test_recursive_traceback_cpython_internal
    self._check_recursive_traceback_display(render_exc)
  File "/home/jelle/cpython-dev/cpython-1/Lib/test/test_traceback.py", line 347, in _check_recursive_traceback_display
    self.assertEqual(actual[-1], expected[-1])
AssertionError: 'RecursionError: maximum recursion depth exceeded in comparison' != 'RecursionError: maximum recursion depth exceeded'
- RecursionError: maximum recursion depth exceeded in comparison
?                                                 --------------
+ RecursionError: maximum recursion depth exceeded


======================================================================
FAIL: test_recursive_traceback_python (test.test_traceback.TracebackFormatTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jelle/cpython-dev/cpython-1/Lib/test/test_traceback.py", line 423, in test_recursive_traceback_python
    self._check_recursive_traceback_display(traceback.print_exc)
  File "/home/jelle/cpython-dev/cpython-1/Lib/test/test_traceback.py", line 347, in _check_recursive_traceback_display
    self.assertEqual(actual[-1], expected[-1])
AssertionError: 'RecursionError: maximum recursion depth exceeded in comparison' != 'RecursionError: maximum recursion depth exceeded'
- RecursionError: maximum recursion depth exceeded in comparison
?                                                 --------------
+ RecursionError: maximum recursion depth exceeded



======================================================================
FAIL: test_bug_xmltoolkit63 (test.test_xml_etree.BugsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jelle/cpython-dev/cpython-1/Lib/test/test_xml_etree.py", line 1538, in test_bug_xmltoolkit63
    self.assertEqual(sys.getrefcount(None), count)
AssertionError: 505087 != 505084


Fixing this will improve the coverage check on PRs.

The test_traceback changes should be easy to tweak to use a regex or startwith() check so the tests pass again.

For the test_xml_etree failures I'm not sure how best to fix that other than taking it out. Maybe if sys.gettrace() returns something then skip the test?

Is this a duplicate of issue29048?

I don't think it's a duplicate because we had a passing coverage run when we initially made the migration so something got tweaked to lead to the failures.

BTW, just because I assigned this to me doesn't mean others can't write a PR to solve this. :) I will review any PR that fixes this issue, I'm just going to make sure it gets solved no matter what if no one else creates a PR.

I'll look into creating a PR when I have some time.

It would also be useful to tweak the Travis/coverage configuration so that it fails loudly if one of the tests doesn't pass in the coverage check.

The list of failing tests in issue29048 contains test_traceback and test_xml_etree. Thus this issue may be a part of issue29048. Some tests may be fixed since reporting issue29048.

So the reason we don't have the coverage run complain loudly is it takes at least 40 minutes to complete, so having to wait for that could potentially be annoying if you're e.g. trying to merge a cherry-pick and you just want to verify you didn't break anything.

And yes, there might be some connection to issue #29048. To keep things simple we can close this in favour of the other issue and track the work there.