classification
Title: Report more silently skipped tests as skipped
Type: enhancement Stage: resolved
Components: Tests Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: zach.ware Nosy List: ezio.melotti, haypo, michael.foord, ned.deily, pitrou, python-dev, serhiy.storchaka, terry.reedy, vajrasky, zach.ware
Priority: normal Keywords: patch

Created on 2013-11-13 17:18 by zach.ware, last changed 2013-12-12 16:33 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
skiptest_not_return_or_pass.diff zach.ware, 2013-11-13 17:18 review
skiptest_not_return_or_pass.v2.diff zach.ware, 2013-11-13 20:24 A couple additional skips review
skiptest_not_return_or_pass.v3.diff zach.ware, 2013-11-14 22:05 Version 3 review
skiptest_not_return_or_pass.v4-3.3.diff zach.ware, 2013-11-20 18:27 review
skiptest_not_return_or_pass.v4-2.7.diff zach.ware, 2013-11-21 19:41 review
skiptest_not_return_or_pass.v5-2.7.diff zach.ware, 2013-12-10 17:25 2.7, version 5 review
skiptest_not_return_or_pass.v6-2.7.diff zach.ware, 2013-12-10 17:46 Version 6, the real version 5! review
Messages (23)
msg202762 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-11-13 17:18
(Nosy list copied from #18702)

Grepping for "^\s+return$" and "^\s+pass$" in Lib/test turned up several more tests that are silently skipped (along with many legitimate uses of each).  The attached patch turns each into a skip in a few various ways, including splitting a test into two tests in a couple of cases.

I'll make a few comments on Rietveld myself to point out places where I would really like some extra scrutiny.
msg202769 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-11-13 19:05
See also issue19492 and issue19493.
msg202774 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-11-13 20:24
Starting a review of #19492, I realized my original regexs didn't take comments into account.  Grepping "^\s+return\s*(#.*)?$" instead turned up a couple more skips, including some in test_tempfile that said "return  # ugh, can't use SkipTest", relics of the days before unittest.SkipTest.
msg202793 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-11-13 22:35
I added some comments on Rietveld to the first patch. In general it LGTM. I will made a review for additional skips tomorrow.

I suggest extract importlib related changes in separated issue. Bratt Cannon should review them.
msg202795 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-11-13 22:44
> I will made a review for additional skips tomorrow.

Additional skips LGTM.
msg202906 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-11-14 22:05
Here's a new patch to address the reviews from Serhiy and Ezio (thanks to you both!).
msg203076 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-11-16 18:59
I get a failure on Linux with test_posix:
$ ./python -m test test_posix
[1/1] test_posix
test test_posix failed -- Traceback (most recent call last):
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 713, in test_getcwd_long_pathnames
    _create_and_do_getcwd(dirname)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 708, in _create_and_do_getcwd
    _create_and_do_getcwd(dirname, current_path_length + len(dirname) + 1)
  File "/home/wolf/dev/py/py3k/Lib/test/test_posix.py", line 706, in _create_and_do_getcwd
    os.getcwd()
OSError: [Errno 34] Numerical result out of range

1 test failed:
    test_posix
msg203080 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-11-16 20:38
Except pickletester and test_posix third patch LGTM. test_posix is worth separate issue. test_reduce and test_getinitargs in pickletester are always empty and can be just removed (if you don't want implement them).
msg203334 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-11-19 03:20
I'll open new issues for test_posix and pickletester and commit the rest of the patch as soon as I can get it backported.  Thanks for the reviews!
msg203494 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-11-20 15:42
pickletester issue opened at #19648.  The test_posix issue already has an open issue at #9246.
msg203526 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-11-20 19:48
skiptest_not_return_or_pass.v4-3.3.diff LGTM.
msg203647 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-11-21 15:53
The 2.7 patch has a lot of extra changes in it, extra review is probably in order for it.
msg205522 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-08 06:45
New changeset 3283fb24106d by Zachary Ware in branch '3.3':
Issue 19572: More silently skipped tests explicitly skipped.
http://hg.python.org/cpython/rev/3283fb24106d

New changeset 03afd2d7d395 by Zachary Ware in branch 'default':
Issue 19572: More silently skipped tests explicitly skipped.
http://hg.python.org/cpython/rev/03afd2d7d395
msg205523 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-12-08 06:46
Committed on 3.3 and default; I'd still like some extra eyes on the 2.7 patch before I commit it.
msg205533 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-12-08 09:21
I have added few comments on Rietveld.
msg205836 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-12-10 17:25
Here's a new 2.7 patch.  It addresses Serhiy's review comments and doesn't change test_xpickle, which I will be opening a new issue for.
msg205838 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-12-10 17:34
I missed the comments on test_bsddb; I'll either post a new patch here or open a new issue depending on how big that diff becomes.
msg205839 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-12-10 17:46
This patch includes Serhiy's suggestions.  Oops!
msg205844 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-12-10 18:55
$ ./python -3 -m test.regrtest test_builtin
test_builtin
test test_builtin crashed -- <type 'exceptions.AssertionError'>: filter ('.+ is renamed to imp.reload', DeprecationWarning) did not catch any warning
1 test failed:
    test_builtin

Rest of the skiptest_not_return_or_pass.v6-2.7.diff patch LGTM.
msg205849 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-10 20:09
New changeset 423e09aedf79 by Zachary Ware in branch '2.7':
Issue #19572: More silently skipped tests explicitly skipped.
http://hg.python.org/cpython/rev/423e09aedf79
msg205851 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2013-12-10 20:11
Fixed that last comment in the commit.  Thank you for all the reviews, Serhiy!
msg205863 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-10 22:06
New changeset ca9bca7aecda by Zachary Ware in branch '2.7':
Issue #19572: Replace a return that shouldn't have been removed from test_os.
http://hg.python.org/cpython/rev/ca9bca7aecda
msg205959 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-12 16:33
New changeset 1ad2ff119356 by Zachary Ware in branch '3.3':
Avoid UnicodeEncodeError by only printing ASCII.
http://hg.python.org/cpython/rev/1ad2ff119356
History
Date User Action Args
2013-12-12 16:33:38python-devsetmessages: + msg205959
2013-12-10 22:06:59python-devsetmessages: + msg205863
2013-12-10 20:11:00zach.waresetstatus: open -> closed
resolution: fixed
messages: + msg205851

stage: patch review -> resolved
2013-12-10 20:09:41python-devsetmessages: + msg205849
2013-12-10 18:55:48serhiy.storchakasetmessages: + msg205844
2013-12-10 17:46:31zach.waresetfiles: + skiptest_not_return_or_pass.v6-2.7.diff

messages: + msg205839
2013-12-10 17:34:41zach.waresetmessages: + msg205838
2013-12-10 17:25:20zach.waresetfiles: + skiptest_not_return_or_pass.v5-2.7.diff

messages: + msg205836
2013-12-08 09:21:55serhiy.storchakasetmessages: + msg205533
2013-12-08 06:46:37zach.waresetassignee: zach.ware
messages: + msg205523
versions: - Python 3.3, Python 3.4
2013-12-08 06:45:25python-devsetnosy: + python-dev
messages: + msg205522
2013-11-21 19:41:49zach.waresetfiles: + skiptest_not_return_or_pass.v4-2.7.diff
2013-11-21 19:37:40zach.waresetfiles: - skiptest_not_return_or_pass.v4-2.7.diff
2013-11-21 15:53:57zach.waresetfiles: + skiptest_not_return_or_pass.v4-2.7.diff

messages: + msg203647
2013-11-20 19:48:50serhiy.storchakasetmessages: + msg203526
2013-11-20 18:28:00zach.waresetfiles: + skiptest_not_return_or_pass.v4-3.3.diff
2013-11-20 15:42:18zach.waresetmessages: + msg203494
2013-11-19 03:20:30zach.waresetmessages: + msg203334
2013-11-16 20:38:35serhiy.storchakasetmessages: + msg203080
2013-11-16 18:59:28ezio.melottisetmessages: + msg203076
2013-11-14 22:05:55zach.waresetfiles: + skiptest_not_return_or_pass.v3.diff

messages: + msg202906
versions: + Python 2.7
2013-11-13 22:44:10serhiy.storchakasetmessages: + msg202795
2013-11-13 22:35:17serhiy.storchakasetmessages: + msg202793
2013-11-13 20:24:41zach.waresetfiles: + skiptest_not_return_or_pass.v2.diff

messages: + msg202774
2013-11-13 19:05:58serhiy.storchakasetmessages: + msg202769
2013-11-13 17:18:58zach.warecreate