Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unittest swallows part of stack trace when raising AssertionError in a TestCase #69147

Closed
cjerdonek opened this issue Aug 29, 2015 · 11 comments
Closed
Labels
3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@cjerdonek
Copy link
Member

BPO 24959
Nosy @rbtcollins, @ezio-melotti, @voidspace, @cjerdonek, @miss-islington, @iritkatriel, @dseomn
PRs
  • bpo-24959: fix unittest.assertRaises bug where traceback entries are dropped from chained exceptions #23688
  • [3.10] bpo-24959: fix unittest.assertRaises bug where traceback entries are dropped from chained exceptions (GH-23688) #31775
  • [3.9] bpo-24959: fix unittest.assertRaises bug where traceback entries are dropped from chained exceptions (GH-23688) #31776
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2022-03-08.23:03:00.947>
    created_at = <Date 2015-08-29.16:15:14.217>
    labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
    title = 'unittest swallows part of stack trace when raising AssertionError in a TestCase'
    updated_at = <Date 2022-03-08.23:03:00.946>
    user = 'https://github.com/cjerdonek'

    bugs.python.org fields:

    activity = <Date 2022-03-08.23:03:00.946>
    actor = 'iritkatriel'
    assignee = 'none'
    closed = True
    closed_date = <Date 2022-03-08.23:03:00.947>
    closer = 'iritkatriel'
    components = ['Library (Lib)']
    creation = <Date 2015-08-29.16:15:14.217>
    creator = 'chris.jerdonek'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 24959
    keywords = ['patch']
    message_count = 11.0
    messages = ['249323', '249324', '261719', '261847', '261856', '261857', '269782', '382895', '414778', '414779', '414781']
    nosy_count = 8.0
    nosy_names = ['rbcollins', 'ezio.melotti', 'michael.foord', 'chris.jerdonek', 'Aaron Sokoloski', 'miss-islington', 'iritkatriel', 'dseomn']
    pr_nums = ['23688', '31775', '31776']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue24959'
    versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    @cjerdonek
    Copy link
    Member Author

    unittest swallows some lines of the stack trace when raising an AssertionError using the "raise from" syntax inside a TestCase. This marks it harder to pinpoint the source of test failures. It is also confusing to see a stack trace like this because the error doesn't originate where the stack trace says it originates.

    To reproduce:

        import unittest
    
        def foo():
            raise Exception("foo")
    
        class Test(unittest.TestCase):
    
            def test_not_okay(self):
                try:
                    foo()
                except Exception as exc:
                    raise AssertionError("bar") from exc
    
            def test_okay1(self):
                try:
                    foo()
                except Exception as exc:
                    raise ValueError("bar") from exc
    
            def test_okay2(self):
                try:
                    foo()
                except Exception as exc:
                    raise Exception("bar") from exc

    The result (observe how the display for "test_not_okay" differs from the other two):

    \======================================================================
    ERROR: test_okay1 (error.Test)
    \----------------------------------------------------------------------
    
        Traceback (most recent call last):
          File "/Users/chris/dev/error.py", line 17, in test_okay1
            foo()
          File "/Users/chris/dev/error.py", line 5, in foo
            raise Exception("foo")
        Exception: foo
    
        The above exception was the direct cause of the following exception:
    
        Traceback (most recent call last):
          File "/Users/chris/dev/error.py", line 19, in test_okay1
            raise ValueError("bar") from exc
        ValueError: bar
    \======================================================================
    ERROR: test_okay2 (error.Test)
    \----------------------------------------------------------------------
    
        Traceback (most recent call last):
          File "/Users/chris/dev/error.py", line 23, in test_okay2
            foo()
          File "/Users/chris/dev/error.py", line 5, in foo
            raise Exception("foo")
        Exception: foo
    
        The above exception was the direct cause of the following exception:
    
        Traceback (most recent call last):
          File "/Users/chris/dev/error.py", line 25, in test_okay2
            raise Exception("bar") from exc
        Exception: bar
    \======================================================================
    FAIL: test_not_okay (error.Test)
    \----------------------------------------------------------------------
    
        Traceback (most recent call last):
          File "/Users/chris/dev/error.py", line 11, in test_not_okay
            foo()
        Exception: foo
    
        The above exception was the direct cause of the following exception:
    
        Traceback (most recent call last):
          File "/Users/chris/dev/error.py", line 13, in test_not_okay
            raise AssertionError("bar") from exc
        AssertionError: bar
    \----------------------------------------------------------------------
    Ran 3 tests in 0.001s
    
    FAILED (failures=1, errors=2)
    

    @cjerdonek cjerdonek added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Aug 29, 2015
    @cjerdonek
    Copy link
    Member Author

    I guess this isn't limited just to the "raise from" syntax. It also occurs if "from exc" is removed from the example above.

    @cjerdonek cjerdonek changed the title unittest swallows part of stack trace using "raise from" with AssertionError unittest swallows part of stack trace when raising AssertionError in a TestCase Aug 29, 2015
    @rbtcollins
    Copy link
    Member

    Hmm, this is a little surprising, but - why are you raising AssertionError like that - thats what assertRaises is for.

    @cjerdonek
    Copy link
    Member Author

    This is simply a reduced test case to illustrate the issue more clearly. There was more to it in how I was using it in practice.

    @rbtcollins
    Copy link
    Member

    I'm fairly sure its eating the stack frames because the calling frames are annotated __unittest__ - its technically a feature :/.

    @rbtcollins rbtcollins changed the title unittest swallows part of stack trace when raising AssertionError in a TestCase unittest swallows par t of stack trace when raising AssertionError in a TestCase Mar 16, 2016
    @rbtcollins
    Copy link
    Member

    Yes, it is... ish.

    The frame skipping code occurs when we serialise exceptions, and we pass a limit in. The limit is calculated on the main exception only. If the cause has a longer exception than the limit we calculated, you'd see this behaviour.

    Probably need to make it possible to do per-exception processing of limit: I think via a callback or similar mechanism in traceback (because the cause might have thrown from some code that is also marked __unittest, so if we're honouring that, we should honour it within each exception.

    @rbtcollins rbtcollins changed the title unittest swallows par t of stack trace when raising AssertionError in a TestCase unittest swallows part of stack trace when raising AssertionError in a TestCase Mar 16, 2016
    @AaronSokoloski
    Copy link
    Mannequin

    AaronSokoloski mannequin commented Jul 4, 2016

    I've run into this bug too. Took a while to track down the cause :)

    @iritkatriel
    Copy link
    Member

    I think this is the same as bpo-42247, where I've recently attached a patch that removes __unittest frames from the traceback rather than count them and then set a limit for printing.

    I could add to that patch what Robert suggests here - to filter the __cause__ traceback as well.

    @iritkatriel iritkatriel added 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes labels Dec 12, 2020
    @iritkatriel
    Copy link
    Member

    New changeset 88b7d86 by Irit Katriel in branch 'main':
    bpo-24959: fix unittest.assertRaises bug where traceback entries are dropped from chained exceptions (GH-23688)
    88b7d86

    @miss-islington
    Copy link
    Contributor

    New changeset 26fa25a by Miss Islington (bot) in branch '3.10':
    bpo-24959: fix unittest.assertRaises bug where traceback entries are dropped from chained exceptions (GH-23688)
    26fa25a

    @iritkatriel
    Copy link
    Member

    New changeset f3ea249 by Irit Katriel in branch '3.9':
    bpo-24959: fix unittest.assertRaises bug where traceback entries are dropped from chained exceptions (GH-23688) (GH-31776)
    f3ea249

    @iritkatriel iritkatriel added 3.11 only security fixes and removed 3.8 only security fixes labels Mar 8, 2022
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants