Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Frame with -1 line number #88463

Closed
vstinner opened this issue Jun 3, 2021 · 16 comments
Closed

Frame with -1 line number #88463

vstinner opened this issue Jun 3, 2021 · 16 comments
Assignees
Labels
3.10 only security fixes 3.11 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs)

Comments

@vstinner
Copy link
Member

vstinner commented Jun 3, 2021

BPO 44297
Nosy @gvanrossum, @vstinner, @markshannon, @asottile, @pablogsal, @domdfcoding, @shreyanavigyan
PRs
  • bpo-44297: Fix missing line number in generator expressions #26801
  • [3.10] bpo-44297: Fix missing line number in generator expressions (Backport of GH-26801) #26821
  • bpo-44297: Make sure that line number is set correctly for call to __exit__ when handling exception in body of a with statement. #26890
  • [3.10] bpo-44297: Backport a regression test to 3.10 #26891
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/markshannon'
    closed_at = <Date 2021-08-13.08:27:24.665>
    created_at = <Date 2021-06-03.01:51:01.227>
    labels = ['interpreter-core', '3.10', '3.11']
    title = 'Frame with -1 line number'
    updated_at = <Date 2021-08-13.08:27:24.664>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2021-08-13.08:27:24.664>
    actor = 'Mark.Shannon'
    assignee = 'Mark.Shannon'
    closed = True
    closed_date = <Date 2021-08-13.08:27:24.665>
    closer = 'Mark.Shannon'
    components = ['Interpreter Core']
    creation = <Date 2021-06-03.01:51:01.227>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 44297
    keywords = ['patch']
    message_count = 16.0
    messages = ['394969', '395209', '396110', '396111', '396118', '396128', '396216', '396224', '396255', '396257', '396361', '396389', '396430', '396456', '396475', '399501']
    nosy_count = 7.0
    nosy_names = ['gvanrossum', 'vstinner', 'Mark.Shannon', 'Anthony Sottile', 'pablogsal', 'domdfcoding', 'shreyanavigyan']
    pr_nums = ['26801', '26821', '26890', '26891']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue44297'
    versions = ['Python 3.10', 'Python 3.11']

    @vstinner
    Copy link
    Member Author

    vstinner commented Jun 3, 2021

    While debugging https://bugs.python.org/issue43921 on Windows, I got a traceback with a single frame and the frame line number is -1.

    It looks like a Python 3.11 regression.

    Mark, Guido: can it be related to recent optimization work done in the main branch?

    See also bpo-44288 "unittest: _is_relevant_tb_level() fails because tb.tb_frame.f_globals=None".

    ======================================================================
    FAIL: test_pha_required_nocert (test.test_ssl.TestPostHandshakeAuth)
    ----------------------------------------------------------------------
    ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2522)

    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "C:\vstinner\python\main\lib\test\test_ssl.py", line -1, in test_pha_required_nocert
    AssertionError: "certificate required" does not match "EOF occurred in violation of protocol (_ssl.c:2522)"

    @vstinner vstinner added 3.11 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Jun 3, 2021
    @domdfcoding
    Copy link
    Mannequin

    domdfcoding mannequin commented Jun 6, 2021

    Is this a re-regression of https://bugs.python.org/issue43933?

    @asottile
    Copy link
    Mannequin

    asottile mannequin commented Jun 19, 2021

    this appears to have regressed in 3.10 as well according to some reports on pytest: pytest-dev/pytest#8227 (comment)

    @asottile asottile mannequin added the 3.10 only security fixes label Jun 19, 2021
    @asottile
    Copy link
    Mannequin

    asottile mannequin commented Jun 19, 2021

    here's the traceback pytest is trying to display and crashing:

    Traceback (most recent call last):
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/_pytest/runner.py", line 311, in from_call
        result: Optional[TResult] = func()
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/_pytest/runner.py", line 341, in <lambda>
        call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/_pytest/python.py", line 503, in collect
        self._inject_setup_module_fixture()
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/_pytest/python.py", line 516, in _inject_setup_module_fixture
        self.obj, ("setUpModule", "setup_module")
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/_pytest/python.py", line 291, in obj
        self._obj = obj = self._getobj()
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/_pytest/python.py", line 500, in _getobj
        return self._importtestmodule()
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/_pytest/python.py", line 578, in _importtestmodule
        mod = import_path(self.fspath, mode=importmode)
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/_pytest/pathlib.py", line 524, in import_path
        importlib.import_module(module_name)
      File "/home/asottile/workspace/cpython/prefix/lib/python3.10/importlib/__init__.py", line 126, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
      File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
        exec(co, module.__dict__)
      File "/tmp/rinohtype/tests/test_attribute.py", line 11, in <module>
        from rinoh.attribute import Attribute, Bool
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/rinoh/__init__.py", line 41, in <module>
        from . import resource
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/rinoh/resource.py", line 205, in <module>
        from .template import DocumentTemplate
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/rinoh/template.py", line 42, in <module>
        from .stylesheets import sphinx
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/rinoh/stylesheets/__init__.py", line 42, in <module>
        .format(stylesheet.description, stylesheet))
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/rinoh/style.py", line 670, in __str__
        for name, entry_point in self.installed_resources:
      File "/tmp/rinohtype/venv/lib/python3.10/site-packages/rinoh/resource.py", line 54, in installed_resources
        for entry_point in ilm.entry_points()[cls.entry_point_group]:
      File "/home/asottile/workspace/cpython/prefix/lib/python3.10/importlib/metadata/__init__.py", line 979, in entry_points
        return SelectableGroups.load(eps).select(**params)
      File "/home/asottile/workspace/cpython/prefix/lib/python3.10/importlib/metadata/__init__.py", line 437, in load
        ordered = sorted(eps, key=by_group)
      File "/home/asottile/workspace/cpython/prefix/lib/python3.10/importlib/metadata/__init__.py", line -1, in <genexpr>
      File "/home/asottile/workspace/cpython/prefix/lib/python3.10/importlib/metadata/_itertools.py", line 16, in unique_everseen
        k = key(element)
      File "/home/asottile/workspace/cpython/prefix/lib/python3.10/importlib/metadata/__init__.py", line 600, in _normalized_name
        return Prepared.normalize(self.name)
      File "/home/asottile/workspace/cpython/prefix/lib/python3.10/importlib/metadata/__init__.py", line 841, in normalize
        return re.sub(r"[-_.]+", "-", name).lower().replace('-', '_')
      File "/home/asottile/workspace/cpython/prefix/lib/python3.10/re.py", line 187, in sub
        return _compile(pattern, flags).sub(repl, string, count)
    TypeError: expected string or bytes-like object
    

    @asottile
    Copy link
    Mannequin

    asottile mannequin commented Jun 19, 2021

    here is a minimal reproduction:

    def iterboom():
        raise AssertionError
        yield 1
    
    next(1 for x in iterboom())

    python 3.9:

    $ python3.9 t.py
    Traceback (most recent call last):
      File "/tmp/rinohtype/t.py", line 5, in <module>
        next(1 for x in iterboom())
      File "/tmp/rinohtype/t.py", line 5, in <genexpr>
        next(1 for x in iterboom())
      File "/tmp/rinohtype/t.py", line 2, in iterboom
        raise AssertionError
    AssertionError
    
    $ python3.10 t.py
    Traceback (most recent call last):
      File "/tmp/rinohtype/t.py", line 5, in <module>
        next(1 for x in iterboom())
      File "/tmp/rinohtype/t.py", line -1, in <genexpr>
      File "/tmp/rinohtype/t.py", line 2, in iterboom
        raise AssertionError
    AssertionError
    

    @markshannon
    Copy link
    Member

    Thanks Anthony, that's a big help.

    @markshannon
    Copy link
    Member

    New changeset 82e5c28 by Mark Shannon in branch 'main':
    bpo-44297: Fix missing line number in generator expressions (GH-26801)
    82e5c28

    @markshannon
    Copy link
    Member

    New changeset 7674c83 by Mark Shannon in branch '3.10':
    bpo-44297: Fix missing line number in generator expressions (GH-26821)
    7674c83

    @vstinner
    Copy link
    Member Author

    Can the issue be closed?

    I'm not sure which assertion failed in test_ssl.test_pha_required_nocert(). I bet on this one:

                with self.assertRaisesRegex(
                    ssl.SSLError,
                    '(certificate required|EOF occurred)'
                ):
    

    I don't know if the commit 82e5c28 is supported is to fix issue or not.

    @vstinner
    Copy link
    Member Author

    Oh, I can still reproduce the issue on the main branch with this patch:

    diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
    index cee97a8302..3f66818ae1 100644
    --- a/Lib/test/test_ssl.py
    +++ b/Lib/test/test_ssl.py
    @@ -4499,7 +4499,7 @@ def msg_cb(conn, direction, version, content_type, msg_type, data):
                     # server aborts connection with an error.
                     with self.assertRaisesRegex(
                         ssl.SSLError,
    -                    '(certificate required|EOF occurred)'
    +                    'xxxxxxxxxxxxx'
                     ):
                         # receive CertificateRequest
                         data = s.recv(1024)
    $ ./python -m test test_ssl -m test_pha_required_nocert -v
    (...)
    FAIL: test_pha_required_nocert (test.test_ssl.TestPostHandshakeAuth)

    (...)
    Traceback (most recent call last):
      File "/home/vstinner/python/main/Lib/test/test_ssl.py", line -1, in test_pha_required_nocert
    AssertionError: (...)
    ...

    => "line -1"

    @shreyanavigyan
    Copy link
    Mannequin

    shreyanavigyan mannequin commented Jun 22, 2021

    Here presumably the error is occurring somewhere near unittest. I've tested this and wrote a minimal reproducible example.

    import unittest
    
    class TestingError(unittest.TestCase):
        def test_negative_one(self):
            with self.assertRaisesRegex(AssertionError, "xxxxx"):
                self.assertEqual(1, 2)
    

    Running this with unittest discover or in any other command-line way (like -c) results in frame with -1 while running it from script (with unittest.main probably) does not.

    Hope this helps.

    @shreyanavigyan
    Copy link
    Mannequin

    shreyanavigyan mannequin commented Jun 23, 2021

    Oh it is also occurring when running from script (I ran it from IDLE and it always results in correct lineno). I wrote example similar to how unittests work and the frame with -1 lineno is also occurring there. Here is the code -

    class A:
        def __enter__(self):
            return self
        def __exit__(self, *args, **kwargs):
            raise Exception("Frame is -1 if program is run from command line")
    
    with A():
        raise Exception("Normal Error")
    

    Also, unsurprisingly, pdb fails with a "TypeError: '>=' not supported between instances of 'NoneType' and 'int'". Full pdb log -

    Traceback (most recent call last):
      File "C:\Users\shrey\Desktop\line_negative_one.py", line -1, in <module>
      File "C:\github\cpython\lib\bdb.py", line 96, in trace_dispatch
        return self.dispatch_exception(frame, arg)
      File "C:\github\cpython\lib\bdb.py", line 169, in dispatch_exception
        if self.stop_here(frame):
      File "C:\github\cpython\lib\bdb.py", line 212, in stop_here
        return frame.f_lineno >= self.stoplineno
    TypeError: '>=' not supported between instances of 'NoneType' and 'int'

    @markshannon
    Copy link
    Member

    Thanks for the reproducer.

    @markshannon markshannon self-assigned this Jun 23, 2021
    @shreyanavigyan
    Copy link
    Mannequin

    shreyanavigyan mannequin commented Jun 24, 2021

    Found the culprit -

    c->u->u_lineno = -1;

    Same goes for async with -

    c->u->u_lineno = -1;

    (Didn't test async with so I can be wrong there.)

    Is this intentional? I removed these lines and the line number is coming correct. Can changing this affect other use cases?

    Without the change:

    Traceback (most recent call last):
      File "C:\Users\shrey\Desktop\line_negative_one.py", line 8, in <module>
        raise Exception("Normal Error")
    Exception: Normal Error
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "C:\Users\shrey\Desktop\line_negative_one.py", line -1, in <module>
      File "C:\Users\shrey\Desktop\line_negative_one.py", line 5, in __exit__
        raise Exception("Frame is -1 if program is run from command line")
    Exception: Frame is -1 if program is run from command line

    With the change:

    Traceback (most recent call last):
      File "C:\Users\shrey\Desktop\line_negative_one.py", line 8, in <module>
        raise Exception("Normal Error")
    Exception: Normal Error
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "C:\Users\shrey\Desktop\line_negative_one.py", line 7, in <module>
        with A():
      File "C:\Users\shrey\Desktop\line_negative_one.py", line 5, in __exit__
        raise Exception("Frame is -1 if program is run from command line")
    Exception: Frame is not -1 anymore if program is run from command line

    @pablogsal
    Copy link
    Member

    New changeset 0b6b286 by Mark Shannon in branch '3.10':
    bpo-44297: Add a regression test for line numbers in with statements (GH-26891)
    0b6b286

    @markshannon
    Copy link
    Member

    I believe this to be fixed.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.10 only security fixes 3.11 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs)
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants