Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tokenize the source to manage Pdb breakpoints #59118

Open
xdegaye mannequin opened this issue May 25, 2012 · 4 comments
Open

tokenize the source to manage Pdb breakpoints #59118

xdegaye mannequin opened this issue May 25, 2012 · 4 comments
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@xdegaye
Copy link
Mannequin

xdegaye mannequin commented May 25, 2012

BPO 14913
Nosy @birkenfeld, @asvetlov, @xdegaye
Files
  • pdb_default.patch
  • pdb_default_2.patch
  • pdb_lnotab.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2012-05-25.11:49:13.311>
    labels = ['type-bug', 'library']
    title = 'tokenize the source to manage Pdb breakpoints'
    updated_at = <Date 2012-12-05.11:07:36.486>
    user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2012-12-05.11:07:36.486>
    actor = 'asvetlov'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)']
    creation = <Date 2012-05-25.11:49:13.311>
    creator = 'xdegaye'
    dependencies = []
    files = ['25704', '26050', '27565']
    hgrepos = []
    issue_num = 14913
    keywords = ['patch']
    message_count = 4.0
    messages = ['161570', '163182', '172877', '176279']
    nosy_count = 3.0
    nosy_names = ['georg.brandl', 'asvetlov', 'xdegaye']
    pr_nums = []
    priority = 'normal'
    resolution = None
    stage = None
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue14913'
    versions = ['Python 3.4']

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 25, 2012

    Pdb behavior is not consistent with GNU gdb behavior when setting a
    breakpoint on an empty line, a comment or a multi-line statement (the
    breakpoint is ignored by pdb on a non-first line of a multi-line
    statement and rejected on empty lines or comment lines). It is also
    confusing that, when a breakpoint is set on a function definition, pdb
    also stops at the function definition execution when the module is
    being loaded.

    Pdb performance:
    ----------------
    When a breakpoint is set in a module, pdb sets the trace function on
    all the frames whose function is defined in this module (and the trace
    function is run for each line in the function), regardless of the fact
    that the function may not have any breakpoint.

    Pdb breakpoint management problems:
    -----------------------------------
    See bpo-14789, bpo-14792, bpo-14795, bpo-14808 and bpo-14912.

    Pdb rejects setting a breakpoint in a function or method defined in a
    non imported submodule of a package, but accepts to set it when the
    non imported module is not in a package. For example with
    'asyncore.close_all' and 'logging.info':

        $ python
        Python 3.2.2 (default, Dec 27 2011, 17:35:55) 
        [GCC 4.3.2] on linux2
        Type "help", "copyright", "credits" or "license" for more
        information.
        >>> import pdb; pdb.run('def foo(): pass')
        > <string>(1)<module>()
        (Pdb) break asyncore.close_all
        Breakpoint 1 at /usr/local/lib/python3.2/asyncore.py:580
        (Pdb) break logging.info
        *** The specified object 'logging.info' is not a function or was
        not found along sys.path.

    When a breakpoint is set at the line number of a function definition,
    the algorithm used to find the first statement of the function (the
    actual breakpoint) is not robust, so that when another breakpoint is
    also set at this fuction using the function name instead of the line
    number, then this last breakpoint hides the first one.

    Use tokenize:
    -------------
    One solution is to parse the module source to build a description of
    each function and method defined in the source:
    The description is the set of line_set of a function. A line_set
    is either the line(s) of a function definition statement, or a
    group of consecutive statement lines, or a group of the physical
    lines of a logical line.

    The attached patch implements this solution with std lib tokenize:

    When a breakpoint is set at an empty line or a comment in a
    function, pdb stops at the next statement. When a breakpoint is
    set at a multi-line statement (but not a function definition
    statement), pdb stops at the first line of the multi-line
    statement.  When a breakpoint is set at a line in a function
    definition or at a function name, pdb stops at the first statement
    in this function.  Attempting to set a breakpoint on a line
    outside a function definition is rejected with an error message.
    Pdb does not stop anymore at the function definition execution
    when the module is loaded.
    
    The trace function is only set on frames whose function has one
    breakpoint or more.
    
    The patch fixes bpo-14789, bpo-14792, bpo-14795, bpo-14808
    and bpo-14912.  The patch fixes the problem of setting a
    breakpoint in a function or method defined in package submodule
    when the submodule is not imported.  The patch fixes the problem
    described above of having a breakpoint set by function name hide a
    breakpoint set by line number.  The patch includes test cases for
    all those issues.
    
    Parser performance: a file is only parsed when a breakpoint is set
    in this file (and parsed only once of course).  Some performance
    results of the parser on a laptop:
    
        * rpdb2.py from the rpdb2 project: 14500 lines parsed in 2.0
          seconds
        * a more reasonable (but still large) file size with
          turtle.py from the std library: 4100 lines parsed in 0.6
          second
    

    @xdegaye xdegaye mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels May 25, 2012
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Jun 19, 2012

    Uploaded pdb_default_2.patch.
    This new patch fixes the previous patch that fails to stop at
    breakpoints set in nested functions, and extends the previous patch in
    allowing breakpoints outside function and method definitions.

    When a breakpoint is set at the line number of a function definition,
    the algorithm used to find the first statement of the function (the
    actual breakpoint) is not robust, so that when another breakpoint is
    also set at this fuction using the function name instead of the line
    number, then this last breakpoint hides the first one.

    This is not correct. One should read instead:

    When a breakpoint is set at the first statement of a function and
    another breakpoint is set using the function name, then the first
    breakpoint hides the other one.
    

    This is fixed as well in both patches.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Oct 14, 2012

    Attached patch pdb_lnotab.patch uses lnotabs (see
    Objects/lnotab_notes.txt) to find the actual breakpoint line number and
    parses the module source with tokenize to find the set of function and
    fully qualified method names in a module.

    The patch fixes issues 6322, 14789, 14792, 14795, 14808.

    The local trace function is only set on functions where a breakpoint is
    set, this provides a significant performance improvement.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Nov 24, 2012

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    0 participants