This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Blacklist FunctionTestCase from test discovery
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.5
process
Status: closed Resolution: duplicate
Dependencies: 14534 Superseder:
Assigned To: Nosy List: Arfrever, Evan Hubinger, barry, ezio.melotti, martin.panter, michael.foord, pitrou, rbcollins
Priority: normal Keywords: easy, patch

Created on 2014-10-20 17:14 by pitrou, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
issue22680.patch Evan Hubinger, 2015-07-27 21:38 Patch to fix issue 22680. review
Messages (17)
msg229731 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-20 17:14
I just got the following traceback when trying discover without 3.5. It runs fine under 3.4...

$ ~/cpython/default/python -m unittest discover -v
Traceback (most recent call last):
  File "/home/antoine/cpython/default/Lib/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/antoine/cpython/default/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/antoine/cpython/default/Lib/unittest/__main__.py", line 18, in <module>
    main(module=None)
  File "/home/antoine/cpython/default/Lib/unittest/main.py", line 93, in __init__
    self.runTests()
  File "/home/antoine/cpython/default/Lib/unittest/main.py", line 244, in runTests
    self.result = testRunner.run(self.test)
  File "/home/antoine/cpython/default/Lib/unittest/runner.py", line 168, in run
    test(result)
  File "/home/antoine/cpython/default/Lib/unittest/suite.py", line 87, in __call__
    return self.run(*args, **kwds)
  File "/home/antoine/cpython/default/Lib/unittest/suite.py", line 125, in run
    test(result)
  File "/home/antoine/cpython/default/Lib/unittest/suite.py", line 87, in __call__
    return self.run(*args, **kwds)
  File "/home/antoine/cpython/default/Lib/unittest/suite.py", line 125, in run
    test(result)
  File "/home/antoine/cpython/default/Lib/unittest/suite.py", line 87, in __call__
    return self.run(*args, **kwds)
  File "/home/antoine/cpython/default/Lib/unittest/suite.py", line 125, in run
    test(result)
  File "/home/antoine/cpython/default/Lib/unittest/case.py", line 625, in __call__
    return self.run(*args, **kwds)
  File "/home/antoine/cpython/default/Lib/unittest/case.py", line 553, in run
    result.startTest(self)
  File "/home/antoine/cpython/default/Lib/unittest/runner.py", line 54, in startTest
    self.stream.write(self.getDescription(test))
  File "/home/antoine/cpython/default/Lib/unittest/runner.py", line 47, in getDescription
    return '\n'.join((str(test), doc_first_line))
  File "/home/antoine/cpython/default/Lib/unittest/case.py", line 1354, in __str__
    self._testFunc.__name__)
AttributeError: 'str' object has no attribute '__name__'
msg229732 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2014-10-20 17:16
I assume you mean you get the error *with* 3.5 (not "without"). Does this happen *every time* (i.e. is it trivially reproducible) - or can you provide a repro?

This is regression that I would *assume* (a totally lazy assumption) introduced by the new error handling code.
msg229733 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-20 17:19
> I assume you mean you get the error *with* 3.5 (not "without").

Yes, sorry :-)

> Does this happen *every time* (i.e. is it trivially reproducible)

It happens with the llvmlite repository: https://github.com/numba/llvmlite
Since using it requires some compiling, I could try to investigate myself if you tell me what to look for.
msg229734 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2014-10-20 17:22
As _testFunc is a string and shouldn't be, I'd be very interested to know *what* the string is. That may give us a clue as to where it has come from. (So a try...except...raise that also prints that value would be a good first step.)
msg229735 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-20 17:25
testFunc contains "runTest" (!).
msg229737 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-20 17:33
Ok, apparently it's because I have "from unittest import *" somewhere.
msg230367 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2014-10-31 18:35
Does removing "from unittest import *" from somewhere fix the issue?
Is this a bug in your code, or did that import reveal a bug in unittest?
msg230369 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-31 18:37
> Does removing "from unittest import *" from somewhere fix the issue?

Yes.

> Is this a bug in your code, or did that import reveal a bug in unittest?

Your choice :-)
msg230370 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-31 18:38
btw, in case that's not clear from the traceback, discover was mistaking FunctionTestCase as one of my test classes (while it was just there because of the "import *").
msg230372 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2014-10-31 18:40
Ah. Test discovery is discovering FunctionTestCase and attempting to instantiate it as a test. And failing.

Maybe discovery should special case that class and not treat it as a normal TestCase.
msg230373 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-31 18:41
Or perhaps everything that has as a __module__ something which has the attribute __unittest = True?

(the attribute is there for a reason, right?)
msg230374 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2014-10-31 18:43
Michael Foord opined:
---------------------
> Maybe discovery should special case that class and not treat it as a normal TestCase.

Not a bad idea.  On the other hand, I don't believe unittest is advertised as supporting 'from ... import *', which *is* usually advised to be a bad idea.
msg230629 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2014-11-04 15:07
This was reported as https://code.google.com/p/unittest-ext/issues/detail?id=71 a while back. I think blacklisting FunctionTestCase in TestLoader is entirely reasonable.
msg230672 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2014-11-05 09:49
I agree.
msg233275 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-01-01 06:44
Assuming that FunctionTestCase inherits from TestCase, a fix for Issue 14534 would be useful here. That bug is about avoiding TestCase subclasses being automatically run, which is useful for abstract base test classes.
msg247488 - (view) Author: Evan Hubinger (Evan Hubinger) * Date: 2015-07-27 21:38
I wrote a patch to blacklist FunctionTestCase in TestLoader, and a test to make sure FunctionTestCase doesn't show up in the TestSuite after loading a module that includes it. The test runs successfully. This is my first patch, so feedback would be appreciated if I did anything wrong.
msg248923 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-08-20 23:30
Thanks for this. I think that a better approach would be the other linked bug - we can kill many birds with one stone.
History
Date User Action Args
2022-04-11 14:58:09adminsetgithub: 66870
2015-08-25 07:37:04martin.pantersetdependencies: + Add means to mark unittest.TestCases as "do not load".
2015-08-20 23:30:49rbcollinssetstatus: open -> closed
resolution: duplicate
messages: + msg248923
2015-08-19 23:58:50rbcollinssetstage: needs patch -> patch review
2015-07-27 21:38:14Evan Hubingersetfiles: + issue22680.patch

nosy: + Evan Hubinger
messages: + msg247488

keywords: + patch
2015-07-21 07:40:27ethan.furmansetnosy: - ethan.furman
2015-01-01 06:44:32martin.pantersetmessages: + msg233275
2015-01-01 06:06:15martin.pantersetnosy: + martin.panter
2014-11-05 11:21:30ezio.melottisetkeywords: + easy
2014-11-05 09:49:13michael.foordsetmessages: + msg230672
title: unittest discovery is fragile -> Blacklist FunctionTestCase from test discovery
2014-11-04 15:07:53rbcollinssetmessages: + msg230629
2014-10-31 18:44:40Arfreversetnosy: + Arfrever
2014-10-31 18:43:57ethan.furmansetmessages: + msg230374
2014-10-31 18:41:53pitrousetmessages: + msg230373
2014-10-31 18:40:03michael.foordsetmessages: + msg230372
2014-10-31 18:38:49pitrousetmessages: + msg230370
2014-10-31 18:37:17pitrousetmessages: + msg230369
2014-10-31 18:35:23ezio.melottisetmessages: + msg230367
2014-10-20 17:53:32ethan.furmansetnosy: + ethan.furman
2014-10-20 17:45:58barrysetnosy: + barry
2014-10-20 17:33:17pitrousetmessages: + msg229737
2014-10-20 17:25:55pitrousetmessages: + msg229735
2014-10-20 17:22:26michael.foordsetmessages: + msg229734
2014-10-20 17:19:12pitrousetmessages: + msg229733
2014-10-20 17:16:55michael.foordsetmessages: + msg229732
2014-10-20 17:14:21pitroucreate