Title: inspect.getsourcelines finds wrong lines when lambda used argument to decorator
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.5
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Claudiu.Popa, akaptur, ballingt, meador.inge, ncoghlan, pitrou, python-dev, terry.reedy, yselivanov
Priority: normal Keywords: patch

Created on 2014-04-14 17:31 by ballingt, last changed 2015-07-24 04:00 by python-dev. This issue is now closed.

File name Uploaded Description Edit
issue21217.patch akaptur, 2014-04-14 22:49 review
issue21217-v2.patch akaptur, 2014-04-15 17:40 review
issue21217-v3.patch akaptur, 2014-04-15 18:18 review
issue21217-v4.patch ballingt, 2015-04-13 20:31
issue21217-v5.patch ballingt, 2015-04-13 21:00 review
issue21217-v6.patch ballingt, 2015-04-13 22:25 review
Messages (22)
msg216127 - (view) Author: Thomas Ballinger (ballingt) * Date: 2014-04-14 17:31

inspect.getsourcelines incorrectly guesses what lines correspond
to the function foo
see getblock in
once it finds a lambda, def or class it finishes it then stops
so get getsourcelines returns only the first two noop decorator
lines of bar, while normal behavior is to return all decorators
as it does for foo
import inspect
from pprint import pprint
def noop(arg):
    def inner(func):
        return func
    return inner
def foo():
    return 1
@noop(lambda: None)
def bar():
    return 1
msg216128 - (view) Author: Thomas Ballinger (ballingt) * Date: 2014-04-14 17:36
"The code object's co_lnotab is how inspect should be getting the sourcelines of the code, instead of looking for the first codeblock."

I'm looking at this now, thanks to Yhg1s for the above.
msg216245 - (view) Author: A Kaptur (akaptur) * Date: 2014-04-14 22:49
This patch adds tests demonstrating broken behavior inspect.getsource and inspect.getsourcelines of decorators containing lambda functions, and modifies inspect.getsourcelines to behave correctly.

We use co_lnotab to extract line numbers on all objects with a code object. inspect.getsourcelines can also take a class, which cannot use co_lnotab as there is no associated code object.

@ballingt and I paired on this patch.

Some open questions about inspect.getsource not created or addressed by this patch:
- Is this a bug that should be patched in previous versions as well?
- the docs for say it can take a traceback. What is the correct behavior here?  There aren't any tests at the moment. We suggest the line of code that caused the traceback, i.e. the line at tb.tb_lineno
-  We added tests of decorated classes. The source of decorated classes does not include the decorators, which is different than the usual behavior of decorated functions. What is the correct behavior here?
- inspect.getblock and inspect.BlockFinder use the term "block" in a way that is inconsistent with its typical use in the interpreter (that is, in ceval.c). Should this be renamed? If so, to what? ("chunk"?)
msg216344 - (view) Author: A Kaptur (akaptur) * Date: 2014-04-15 17:40
v2 of the patch incorporating the comments at
msg216345 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2014-04-15 17:45
Apart from one nit, the patch is looking good.

Also, could you please sign the contributor agreement, as described here:
msg216351 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2014-04-15 18:01
"-  We added tests of decorated classes. The source of decorated classes does not include the decorators, which is different than the usual behavior of decorated functions. What is the correct behavior here?"

There is an open issue for this, It has a patch which uses inspect.unwrap in order to unwrap the decorated functions.
msg216352 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2014-04-15 18:02
Claudiu: I'll take a look at your patch, thanks!
msg216357 - (view) Author: A Kaptur (akaptur) * Date: 2014-04-15 18:18
v3 of patch, including misc/news update, docstring for function, and removing class decorator tests, since it sounds like those are better handled in
msg240738 - (view) Author: Thomas Ballinger (ballingt) * Date: 2015-04-13 20:31
v4 of patch, with tests updated for changed lines in inspect fodder file
msg240749 - (view) Author: Thomas Ballinger (ballingt) * Date: 2015-04-13 21:00
Patch reformatted to be non-git style, NEWS item removed
msg240787 - (view) Author: Thomas Ballinger (ballingt) * Date: 2015-04-13 22:25
Use dis.findlinestarts() to find lines of function instead of grubbing with co_lnotab manually, making dis module dependency non-optional.
msg240791 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-04-13 22:35
It sounds like at least a somewhat functional dis module is a pragmatic requirement for a Python implementation to support introspection, so +1 for reverting to the mandatory dependency on dis rather than duplicating its logic.
msg241050 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-14 22:42
New changeset ac86e5b2d45b by Antoine Pitrou in branch 'default':
Issue #21217: inspect.getsourcelines() now tries to compute the start and
msg241051 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-04-14 22:42
I've committed the latest patch. Thank you, Thomas!
msg241077 - (view) Author: Thomas Ballinger (ballingt) * Date: 2015-04-15 03:48
Thanks Antoine! Could you add Allison Kaptur to NEWS and ACKS? This was an update to her original patch, and we paired on the whole thing.
msg241079 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-15 04:00
New changeset 582e8e71f635 by Benjamin Peterson in branch 'default':
add Allison Kaptur (#21217)
msg245868 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-06-26 20:46
I strongly suspect that ac86e5b2d45b is the cause of the regression reported in #24485. 

def outer():
    def inner():
from inspect import getsource

omits the body of inner.  Ditto if outer is a method.  All is okay if outer is a method and the source of the class is requested.

Could the authors, Allison and Thomas, take a look and try to fix _line_number_helper to pass the added test (and possibly incorporate it into findsource)?  Since there is a new issue already, this one can be left closed and further work posted to #24485.
msg245872 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2015-06-26 22:22
Here's an update on #24485 regression.

Looks like getsource() is now using code objects instead of tokenizer to determine blocks first/last lines.

The problem with this particular case is that "inner" function's code object is completely independent from "outer"'s.

So, for an outer() function below:

def outer():
    def inner():

the code object contains the following opcodes:

 71           0 LOAD_CONST               1 (<code object inner ...>)
              3 LOAD_CONST               2 ('outer1.<locals>.inner')
              6 MAKE_FUNCTION            0
              9 STORE_FAST               0 (inner)
             12 LOAD_CONST               0 (None)
             15 RETURN_VALUE

The correct solution is to use co_lnotab along with co_firstlineno to iterate through opcodes recursively accounting for MAKE_FUNCTION's code objects.

*However*, I don't think we can solve this for classes, i.e.

def outer_with_class():
   class Foo:
      b = 1
      a = 2

there is no way (as far as I know) to get information about the Foo class start/end lineno.

I think that the only way we can solve this is to revert the patch for this issue.
msg245876 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2015-06-27 02:17
> I think that the only way we can solve this is to revert the patch for this issue.

I agree with this.  It seems like doing this analysis at the bytecode level is the
wrong approach.  Perhaps the syntactical analysis being used before should be beefed
up to handle things like the lambda case.
msg245915 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2015-06-28 19:17
FYI, I posted a patch to handle this case and the regression
noted in issue24485 on issue24485.
msg247202 - (view) Author: Roundup Robot (python-dev) Date: 2015-07-23 14:10
New changeset 4e42a62d5648 by Yury Selivanov in branch '3.5':
Issue #24485: Revert backwards compatibility breaking changes of #21217.

New changeset 98a2bbf2cce2 by Yury Selivanov in branch 'default':
Merge 3.5 (issues #21217, #24485).
msg247245 - (view) Author: Roundup Robot (python-dev) Date: 2015-07-24 04:00
New changeset 5400e21e92a7 by Meador Inge in branch '3.5':
Issue #24485: Function source inspection fails on closures.

New changeset 0e7d64595223 by Meador Inge in branch 'default':
Issue #24485: Function source inspection fails on closures.
Date User Action Args
2015-07-24 04:00:15python-devsetmessages: + msg247245
2015-07-23 14:10:52python-devsetmessages: + msg247202
2015-06-28 19:17:56meador.ingesetmessages: + msg245915
2015-06-27 02:17:30meador.ingesetnosy: + meador.inge
messages: + msg245876
2015-06-26 22:22:48yselivanovsetmessages: + msg245872
2015-06-26 20:46:46terry.reedysetnosy: + terry.reedy
messages: + msg245868
2015-04-15 04:00:51python-devsetmessages: + msg241079
2015-04-15 03:48:03ballingtsetmessages: + msg241077
2015-04-14 22:42:37pitrousetstatus: open -> closed
resolution: fixed
messages: + msg241051

stage: patch review -> resolved
2015-04-14 22:42:10python-devsetnosy: + python-dev
messages: + msg241050
2015-04-13 22:35:08ncoghlansetnosy: + ncoghlan
messages: + msg240791
2015-04-13 22:25:21ballingtsetfiles: + issue21217-v6.patch

messages: + msg240787
2015-04-13 21:11:47pitrousetnosy: + pitrou
2015-04-13 21:00:44ballingtsetfiles: + issue21217-v5.patch

messages: + msg240749
2015-04-13 20:31:14ballingtsetfiles: + issue21217-v4.patch

messages: + msg240738
2014-04-15 18:18:05akaptursetfiles: + issue21217-v3.patch

messages: + msg216357
2014-04-15 18:02:42yselivanovsetmessages: + msg216352
2014-04-15 18:01:12Claudiu.Popasetnosy: + Claudiu.Popa
messages: + msg216351
2014-04-15 17:45:39yselivanovsetmessages: + msg216345
2014-04-15 17:40:09akaptursetfiles: + issue21217-v2.patch

messages: + msg216344
2014-04-14 22:49:51akaptursetfiles: + issue21217.patch

versions: - Python 3.1, Python 2.7, Python 3.2, Python 3.3, Python 3.4
keywords: + patch
nosy: + akaptur

messages: + msg216245
stage: patch review
2014-04-14 20:31:46yselivanovsetnosy: + yselivanov
2014-04-14 17:36:00ballingtsetmessages: + msg216128
2014-04-14 17:31:00ballingtcreate