classification
Title: Python 2.7.12: pydoc.help(lambda (a,): lambda x: a) raises IndexError
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Elias Vanderstuyft, emilyemorehouse, gvanrossum, python-dev, serhiy.storchaka, yselivanov
Priority: normal Keywords: patch

Created on 2017-01-23 21:50 by Elias Vanderstuyft, last changed 2017-02-01 22:03 by yselivanov. This issue is now closed.

Files
File name Uploaded Description Edit
inspect_argtuple_unpack_cellvars.patch serhiy.storchaka, 2017-01-24 07:44 review
Messages (10)
msg286119 - (view) Author: Elias Vanderstuyft (Elias Vanderstuyft) Date: 2017-01-23 21:50
The following might be either a bug in the pydoc module or in the inspect module:

$ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pydoc
>>> pydoc.help(lambda (a,): lambda x: a)    # this fails
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pydoc.py", line 1797, in __call__
    self.help(request)
  File "/usr/lib/python2.7/pydoc.py", line 1844, in help
    else: doc(request, 'Help on %s:')
  File "/usr/lib/python2.7/pydoc.py", line 1581, in doc
    pager(render_doc(thing, title, forceload))
  File "/usr/lib/python2.7/pydoc.py", line 1576, in render_doc
    return title % desc + '\n\n' + text.document(object, name)
  File "/usr/lib/python2.7/pydoc.py", line 363, in document
    if inspect.isroutine(object): return self.docroutine(*args)
  File "/usr/lib/python2.7/pydoc.py", line 1321, in docroutine
    args, varargs, varkw, defaults = inspect.getargspec(object)
  File "/usr/lib/python2.7/inspect.py", line 817, in getargspec
    args, varargs, varkw = getargs(func.func_code)
  File "/usr/lib/python2.7/inspect.py", line 791, in getargs
    args[i] = stack[0]
IndexError: list index out of range
>>> pydoc.help(lambda (a,): lambda x: x)    # this succeeds

>>>
msg286129 - (view) Author: Emily Morehouse (emilyemorehouse) * (Python committer) Date: 2017-01-24 03:45
Aspiring contributor here!

I believe I have narrowed this error down to the inspect module, specifically in the for loop beginning on line 759. 

There's a bit of "acrobatics" (to quote the code comments) that I'm trying to sift through, I've been using the getargspec method from the inspect module as well as print statements in the inspect module to narrow down the error. It definitely doesn't seem like the previously mentioned for loop is functioning properly, as the first item in the stack is being referenced when it is empty. Additionally, I'm utilizing the dis module to help make sense of the full code object to compare against the behavior in the for loop.

Yury, I've respectfully included you here as you're listed as the expert for the inspect module. I would really like to submit a patch for this, but may need a nudge in the right direction. I will update here with my progress.
msg286131 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-01-24 04:38
Hi Emily!  Python 2.7 is in a maintenance-only mode at this point and mostly receives security-related patches.  The problem with accepting other kind of patches is that we want 2.7 to be super stable.

You can continue working on this bug anyways, once the patch is ready we'll have to ask Benjamin Peterson (Python 2.7 release manager) to review it.

I suspect the bug is that inspect.getargspec doesn't fully support tuple arguments unpacking (it's a rather obscure feature that we removed from Python 3).
msg286144 - (view) Author: Emily Morehouse (emilyemorehouse) * (Python committer) Date: 2017-01-24 06:34
Yury, thanks for the encouragement to continue on this patch. I think it will be a good exercise to dive a bit deeper into Python's bytecode and put some knowledge to use.

I believe that tuple argument unpacking is handled appropriately, but there is an even further edge case where a closure is introduced. In the following, inspect.getargspec works for the first bit of code, but the second fails, as 'a' is referenced outside of scope.

>>> dis.dis(lambda (a,): lambda x: x)
  1           0 LOAD_FAST                0 (.0)
              3 UNPACK_SEQUENCE          1
              6 STORE_FAST               1 (a)
              9 LOAD_CONST               1 (<code object <lambda> at 0x10087a130, file "<stdin>", line 1>)
             12 MAKE_FUNCTION            0
             15 RETURN_VALUE
>>> dis.dis(lambda (a,): lambda x: a)
  1           0 LOAD_FAST                0 (.0)
              3 UNPACK_SEQUENCE          1
              6 STORE_DEREF              0 (a)
              9 LOAD_CLOSURE             0 (a)
             12 BUILD_TUPLE              1
             15 LOAD_CONST               1 (<code object <lambda> at 0x10087a930, file "<stdin>", line 1>)
             18 MAKE_CLOSURE             0
             21 RETURN_VALUE

I'll keep poking at this and see where I get.

-- EM
msg286148 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-01-24 07:44
Proposed patch fixes the issue.
msg286692 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-01 20:54
New changeset a889c5524520 by Serhiy Storchaka in branch '2.7':
Issue #29354: Fixed inspect.getargs() for parameters which are cell
https://hg.python.org/cpython/rev/a889c5524520
msg286696 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-02-01 21:07
You were on right way Emily. Sorry for crossing your road, but I already had a written patch.
msg286703 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-02-01 21:30
Emily, if you have time and want to help us with the inspect module, we have many other unresolved issues.  I'd be happy to help you with code reviews etc.
msg286706 - (view) Author: Emily Morehouse (emilyemorehouse) * (Python committer) Date: 2017-02-01 21:40
Serhiy, thank you for the update. I looked over your patch to ensure that I understood your solution. I also appreciate your tests, as I was able to see other edge cases that I perhaps would not have thought of.

Yury, I'm more than happy to help! I've spent a decent amount of time learning about code objects, byte code, etc. and I'd love to put that knowledge to use. Let me know the best way to find these issues on the tracker or otherwise.
msg286713 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-02-01 22:03
> Yury, I'm more than happy to help! I've spent a decent amount of time learning about code objects, byte code, etc. and I'd love to put that knowledge to use. Let me know the best way to find these issues on the tracker or otherwise.

Certainly!

There seem to be a lot of issues with inspect.findsource and inspect.getsource functions. Quick search results: #28121 and #26890 (and I know there are few more open issues).

For PEP 525 we need to add two new functions to the inspect module (I overlooked them and didn't add them to 3.6): inspect.getasyncgeneratorstate and inspect.getasyncgeneratorlocals (mirroring inspect.getgeneratorstate and inspect.getgeneratorlocals).

There are some issues that nobody even replied to, like #26577.  For those the protocol is usually to reproduce the problem first, and then make a patch/tests.  You can use advanced search on this tracker to find issues like that one.

If you are into async world, you can work on adding contextlib.asyncontextmanager decorator.  Although this one should first be discussed on the python-ideas mailing list.

Also, if you are into C programming, Victor Stinner, INADA Naoki and I are trying to optimize Python ceval loop from time to time (you knowledge of dis/opcodes can be helpful), so you can look for those kind of open issues too.
History
Date User Action Args
2017-02-01 22:03:28yselivanovsetmessages: + msg286713
2017-02-01 21:40:23emilyemorehousesetmessages: + msg286706
2017-02-01 21:30:22yselivanovsetmessages: + msg286703
2017-02-01 21:14:27gvanrossumsetnosy: + gvanrossum
2017-02-01 21:07:25serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg286696

stage: patch review -> resolved
2017-02-01 20:54:53python-devsetnosy: + python-dev
messages: + msg286692
2017-01-24 07:44:14serhiy.storchakasetfiles: + inspect_argtuple_unpack_cellvars.patch

nosy: + serhiy.storchaka
messages: + msg286148

keywords: + patch
stage: patch review
2017-01-24 06:34:09emilyemorehousesetmessages: + msg286144
2017-01-24 04:38:37yselivanovsetmessages: + msg286131
2017-01-24 03:45:07emilyemorehousesetnosy: + yselivanov, emilyemorehouse
messages: + msg286129
2017-01-23 21:50:38Elias Vanderstuyftcreate