Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Special reporting of NotImplementedError in unittest #45807

Closed
erikandersen mannequin opened this issue Nov 19, 2007 · 9 comments
Closed

Special reporting of NotImplementedError in unittest #45807

erikandersen mannequin opened this issue Nov 19, 2007 · 9 comments
Labels
stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@erikandersen
Copy link
Mannequin

erikandersen mannequin commented Nov 19, 2007

BPO 1466
Nosy @gvanrossum
Files
  • unittest.diff
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2007-11-19.18:38:58.122>
created_at = <Date 2007-11-19.18:32:56.938>
labels = ['type-feature', 'library']
title = 'Special reporting of NotImplementedError in unittest'
updated_at = <Date 2007-11-19.20:09:26.518>
user = 'https://bugs.python.org/erikandersen'

    bugs.python.org fields:

    activity = <Date 2007-11-19.20:09:26.518>
actor = 'erik_andersen'
assignee = 'none'
closed = True
closed_date = <Date 2007-11-19.18:38:58.122>
closer = 'gvanrossum'
components = ['Library (Lib)']
creation = <Date 2007-11-19.18:32:56.938>
creator = 'erik_andersen'
dependencies = []
files = ['8780']
hgrepos = []
issue_num = 1466
keywords = []
message_count = 9.0
messages = ['57648', '57649', '57651', '57655', '57656', '57658', '57659', '57661', '57662']
nosy_count = 3.0
nosy_names = ['gvanrossum', 'erik_andersen', 'draghuram']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1466'
versions = []
    @erikandersen
    Copy link
    Mannequin Author

    erikandersen mannequin commented Nov 19, 2007

    When a unittest test case raises an Exception, that test case is
    considered a failure. However, raising NotImplementedError is not a
    failure. It is something completely normal during development and
    simply indicates that the functionality has not yet been implemented.
    Compare this to a test case that genuinely fails, which means that the
    implementation does something different from what is is supposed to do.

    I therefore think test cases raising NotImplementedError should not be
    treated as failures, but be reported separately as what they are --
    parts of the program that have not yet been implemented. Reporting such
    test cases as failures will give a lot of failure warnings that may
    detract attention from those cases that are genuinely errors. Also,
    SimpleTextRunner will report a lot of useless stack traces for these
    cases, thereby makeing it harder to find the stack traces for those
    tests that realy failed.

    I have made a patch to unittest that reports such cases as "not
    implemented" rather than as failures. This includes changes to the
    documentation and unit test case for unittest and doctest.
    I have included
    "added/changed in version 2.6" remarks in the documentation, so
    you have to changed these if the patch gets into some other version.

    Patch against 59056 attached

    @erikandersen erikandersen mannequin added stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Nov 19, 2007
    @gvanrossum
    Copy link
    Member

    No. If you're testing something that's not implemented, it is correct to
    see that as a failure. Talk to anyone doing TDD.

    @draghuram
    Copy link
    Mannequin

    draghuram mannequin commented Nov 19, 2007

    I don't think unittest automatically treats all exceptions as failures.
    Failures are those that are explicitly flagged with assert* and fail*
    methods. All other exceptions result in "errors". I think what you are
    asking for is to special case one such error but am not sure if it is
    worth it.

    @erikandersen
    Copy link
    Mannequin Author

    erikandersen mannequin commented Nov 19, 2007

    Raghuram Devarakonda skrev:

    Raghuram Devarakonda added the comment:

    I don't think unittest automatically treats all exceptions as failures.
    Failures are those that are explicitly flagged with assert* and fail*
    methods. All other exceptions result in "errors". I think what you are
    asking for is to special case one such error but am not sure if it is
    worth it.

    ----------
    nosy: +draghuram

    Tracker <report@bugs.python.org>
    <http://bugs.python.org/issue1466\>

    You are right, I used the wrong word. I mean that NotImplementedError
    should be treated as a special *error*.

    If you have ever written the test cases before you write the code, you
    will get a huge amount of uninteresting output from
    errors that come from tests to code you havn't written yet. This will
    drown the real errors and failures from the code
    you are actually testing and debugging. What I want is to have unittest
    report these differently so that I can see what are the
    real bugs the code, not
    to spam me with tracebacks I will never look at for test cases that
    refer to not implemented code.

    Erik

    @gvanrossum
    Copy link
    Member

    Sorry, just comment out those tests until you're ready to write the code.

    @erikandersen
    Copy link
    Mannequin Author

    erikandersen mannequin commented Nov 19, 2007

    Possible, but cumbersome. You have to analyze each case and see if it
    depends on code not written.
    When you implement a new piece of code, you have to go through all your
    tests to see if that one should
    be included.
    Also you don't get any reporting on the coverage of the code.
    If you accidentally forget to uncomment a test case, you may never
    realize that you have a bug --
    at least not until it happens in real life.

    Erik

    @draghuram
    Copy link
    Mannequin

    draghuram mannequin commented Nov 19, 2007

    Possible, but cumbersome. You have to analyze each case and see if it

    Have you tried to write a Test Runner that ignores
    NotImplementedError? You may have to override TestResult.addError.

    @gvanrossum
    Copy link
    Member

    Sorry Erik, but I don't think you should try to compensate for your
    flawed methodology by trying to change the unittest framework. End of
    discussion.

    @erikandersen
    Copy link
    Mannequin Author

    erikandersen mannequin commented Nov 19, 2007

    No, I definitely don't want to ignore them, I want them reported in a
    manner that is relevant to them. Just say that the test was not implemented.
    No tracebacks for debugging since there is nothing to debug.

    In addition to a TestRunner I need a new TestCase and a new TestResult,
    which is what is in the patch,
    although I changed the classes rather than inherit from them.

    Erik

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @gvanrossum