Title: inspect.getsourcelines ignores last line in module
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.1, Python 3.2, Python 2.7
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: belopolsky Nosy List: RafeSacks, belopolsky, ggenellina, loewis, zsolt
Priority: normal Keywords: easy, patch

Created on 2008-11-17 08:09 by RafeSacks, last changed 2010-12-02 00:17 by belopolsky. This issue is now closed.

File name Uploaded Description Edit belopolsky, 2008-12-08 21:30
inspect.patch belopolsky, 2008-12-09 02:14 review
issue4335.patch belopolsky, 2008-12-09 16:43
Messages (12)
msg75953 - (view) Author: Rafe Sacks (RafeSacks) Date: 2008-11-17 08:09

If the last line of the function is also the last line of the module, it
is ignored. To repro:

1) Create module 'repro'
def test():
    module_line_2 = None
    module_line_3 = None
    module_line_4 = None
    module_line_5 = None
<end module>

2) Run test:
>>> import repro
>>> import inspect
>>> inspect.getsourcelines(tmp.test)
(['def test():\n', '    module_line_2 = None\n', '    module_line_3 =
None\n', '    module_line_4 = None\n'], 1)

notice no module_line_5.


- Rafe
msg77346 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-12-08 21:30
In order to reproduce the reported problem I had to conjure the attached which is missing '\n' on the last line.

I need to double check the language definition, but I feel this is not a 
bug because a file that ends with an unterminated line is not a well-
formed python program.
msg77347 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-12-08 21:37
According to my understanding of

a newline character is required at the end of the python file.
msg77380 - (view) Author: Rafe Sacks (RafeSacks) Date: 2008-12-09 00:52
"I feel this is not a bug because a file that ends with an unterminated
line is not a well-formed python program."

I used Komodo to make the file. Nothing special. I just made a new
python document and used it. If there is a Komodo Bug that is fine, but
if this issue can happen (and it has), then the module should be made
more robust to protect against it happening, shouldn't it?
msg77387 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-12-09 02:14

Can you confirm that behavior on missing end-of-line is indeed your 
issue?  Note that at first I could not even reproduce the problem based 
on your report.

I am attaching a proof-of-concept patch that would arguably improve the 
inspect module by making it more robust in your case.

Note that it is unlikely that a fix along the lines of my patch will be 
considered for bug-fix releases of 2.5.x and 2.6.x series.  If you want 
an immediate solution, you have a better chance convincing Komodo 
developers to fix their text editor (emacs' python mode auto-terminates 
the last line in .py files.)
msg77392 - (view) Author: Zsolt Haraszti (zsolt) Date: 2008-12-09 07:58
We are also using inspect.getsourceline() in a large system at my
company and we observed some unreliable behavior in the past which we
never had time to analyze till today. Interesting coincidence, but today
we finally took the effort and traced the issue, and we found that:

getsourceline() does not report the last line of a class definition body
  (1) that line is the last line of the file, _AND_
  (2) that line is not terminated with a newline (\n)

This also answers Alexandre's question: yes, the problem happens only if
the line is unterminated.

I understand that an unterminated line may be non-conforming per Python
specs, but my WingIDE happily saved the file, so I suspect other editors
may just do the same, and I do agree with Rafe that it would be nice if
inspect would be resilient against the missing \n. I assume there would
be no harm, and it could eliminate some angst (it is really not obvious
first why it fails).

I would really love to see this fixed in a patch release on the 2.5+
lines. Any chance?
msg77422 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-12-09 15:10
Just to add to the list of getsource quirks: with the following in,

f = lambda: 0 \

>>> import inspect, x; inspect.getsource(x.f)
Traceback (most recent call last):
tokenize.TokenError: ('EOF in multi-line statement', (2, 0))

Same with

def f():
    0 \
msg77433 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-12-09 16:43
You convinced me.  After all, if python interpreter does not complain 
about missing end-of-line and processes the unterminated line, getsource 
should report it.

Here is a patch that passes existing tests and adds a "NoEOF" test case.

I am adding 2.5.3 to the list of versions, but someone will have to 
lobby python-dev to make it happen.

Note that I've modernized the code a little bit by switching from a 
callback tokenizer to the newer token generator.  Hopefully this will 
make it easier to review the code, but I can easily produce an old-style  
patch as well.
msg77514 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-12-10 09:01
The patch has not been accepted for the trunk, so backporting it to
2.5.3 is out of scope at this point.
msg110696 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-18 22:05
Ok so we missed the opportunity to backport this to 2.5.3, could we please get it into modern versions of Python, or what?
msg110697 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-07-18 22:29

Was there a reason for not accepting this patch for the trunk?
msg123026 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-12-02 00:17
3.2 and 2.7 don't exhibit this issue.  I added a test in r86922.
Date User Action Args
2010-12-02 00:17:58belopolskysetstatus: open -> closed

nosy: - BreamoreBoy
messages: + msg123026

resolution: out of date
stage: patch review -> resolved
2010-07-18 22:29:38belopolskysetkeywords: + easy
assignee: belopolsky
messages: + msg110697
2010-07-18 22:05:00BreamoreBoysetnosy: + BreamoreBoy

messages: + msg110696
versions: + Python 3.2, - Python 2.6
2009-04-27 01:47:52ajaksu2setpriority: normal
stage: patch review
type: behavior
versions: + Python 3.1
2008-12-10 09:01:44loewissetnosy: + loewis
messages: + msg77514
versions: - Python 2.5.3
2008-12-09 16:43:41belopolskysetfiles: + issue4335.patch
messages: + msg77433
versions: + Python 2.6, Python 2.7, Python 2.5.3, - Python 2.5
2008-12-09 15:10:20belopolskysetmessages: + msg77422
2008-12-09 07:58:05zsoltsetnosy: + zsolt
messages: + msg77392
2008-12-09 02:14:38belopolskysetfiles: + inspect.patch
keywords: + patch
messages: + msg77387
2008-12-09 00:52:16RafeSackssetmessages: + msg77380
2008-12-08 21:37:24belopolskysetmessages: + msg77347
2008-12-08 21:30:30belopolskysetfiles: +
messages: + msg77346
2008-12-06 00:41:36ggenellinasetnosy: + ggenellina
2008-12-03 04:40:20belopolskysetnosy: + belopolsky
2008-11-17 08:09:10RafeSackscreate