classification
Title: Some tests fail in coverage Travis check
Type: Stage: resolved
Components: Tests Versions: Python 3.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Coverage influence tests, make some of them fail
View: 29048
Assigned To: brett.cannon Nosy List: Jelle Zijlstra, brett.cannon, eli.bendersky, ezio.melotti, michael.foord, scoder, serhiy.storchaka
Priority: normal Keywords:

Created on 2017-03-02 08:42 by Jelle Zijlstra, last changed 2017-03-07 17:36 by brett.cannon. This issue is now closed.

Messages (8)
msg288786 - (view) Author: Jelle Zijlstra (Jelle Zijlstra) * (Python triager) Date: 2017-03-02 08:42
A few tests fail in the coverage Travis target (see e.g. https://travis-ci.org/python/cpython/jobs/206480468): test_traceback and test_xml_etree.

I extracted the actual failures by running in verbose mode locally:

======================================================================
FAIL: test_recursive_traceback_cpython_internal (test.test_traceback.TracebackFormatTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jelle/cpython-dev/cpython-1/Lib/test/test_traceback.py", line 431, in test_recursive_traceback_cpython_internal
    self._check_recursive_traceback_display(render_exc)
  File "/home/jelle/cpython-dev/cpython-1/Lib/test/test_traceback.py", line 347, in _check_recursive_traceback_display
    self.assertEqual(actual[-1], expected[-1])
AssertionError: 'RecursionError: maximum recursion depth exceeded in comparison' != 'RecursionError: maximum recursion depth exceeded'
- RecursionError: maximum recursion depth exceeded in comparison
?                                                 --------------
+ RecursionError: maximum recursion depth exceeded


======================================================================
FAIL: test_recursive_traceback_python (test.test_traceback.TracebackFormatTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jelle/cpython-dev/cpython-1/Lib/test/test_traceback.py", line 423, in test_recursive_traceback_python
    self._check_recursive_traceback_display(traceback.print_exc)
  File "/home/jelle/cpython-dev/cpython-1/Lib/test/test_traceback.py", line 347, in _check_recursive_traceback_display
    self.assertEqual(actual[-1], expected[-1])
AssertionError: 'RecursionError: maximum recursion depth exceeded in comparison' != 'RecursionError: maximum recursion depth exceeded'
- RecursionError: maximum recursion depth exceeded in comparison
?                                                 --------------
+ RecursionError: maximum recursion depth exceeded



======================================================================
FAIL: test_bug_xmltoolkit63 (test.test_xml_etree.BugsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jelle/cpython-dev/cpython-1/Lib/test/test_xml_etree.py", line 1538, in test_bug_xmltoolkit63
    self.assertEqual(sys.getrefcount(None), count)
AssertionError: 505087 != 505084


Fixing this will improve the coverage check on PRs.
msg289124 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-03-06 19:15
The test_traceback changes should be easy to tweak to use a regex or startwith() check so the tests pass again.

For the test_xml_etree failures I'm not sure how best to fix that other than taking it out. Maybe if sys.gettrace() returns something then skip the test?
msg289125 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-06 19:29
Is this a duplicate of issue29048?
msg289126 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-03-06 19:37
I don't think it's a duplicate because we had a passing coverage run when we initially made the migration so something got tweaked to lead to the failures.
msg289127 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-03-06 19:38
BTW, just because I assigned this to me doesn't mean others can't write a PR to solve this. :) I will review any PR that fixes this issue, I'm just going to make sure it gets solved no matter what if no one else creates a PR.
msg289128 - (view) Author: Jelle Zijlstra (Jelle Zijlstra) * (Python triager) Date: 2017-03-06 19:40
I'll look into creating a PR when I have some time.

It would also be useful to tweak the Travis/coverage configuration so that it fails loudly if one of the tests doesn't pass in the coverage check.
msg289131 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-06 20:29
The list of failing tests in issue29048 contains test_traceback and test_xml_etree. Thus this issue may be a part of issue29048. Some tests may be fixed since reporting issue29048.
msg289178 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-03-07 17:36
So the reason we don't have the coverage run complain loudly is it takes at least 40 minutes to complete, so having to wait for that could potentially be annoying if you're e.g. trying to merge a cherry-pick and you just want to verify you didn't break anything.

And yes, there might be some connection to issue #29048. To keep things simple we can close this in favour of the other issue and track the work there.
History
Date User Action Args
2017-03-07 17:36:33brett.cannonsetstatus: open -> closed
resolution: duplicate
stage: resolved
2017-03-07 17:36:22brett.cannonsetsuperseder: Coverage influence tests, make some of them fail
messages: + msg289178
2017-03-06 20:29:50serhiy.storchakasetmessages: + msg289131
2017-03-06 19:40:35Jelle Zijlstrasetmessages: + msg289128
2017-03-06 19:38:51brett.cannonsetmessages: + msg289127
2017-03-06 19:37:59brett.cannonsetmessages: + msg289126
2017-03-06 19:29:29serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg289125
2017-03-06 19:15:39brett.cannonsetassignee: brett.cannon
2017-03-06 19:15:15brett.cannonsetnosy: + brett.cannon
messages: + msg289124
2017-03-02 08:42:36Jelle Zijlstracreate