classification
Title: Generator return value ignored in lambda function
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Rosuav, benjamin.peterson, bru, ezio.melotti, gvanrossum, python-dev, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2015-01-08 14:42 by Rosuav, last changed 2015-03-11 16:25 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
0001-lambda-generators-don-t-throw-away-stack-top.patch bru, 2015-03-02 11:16 Fix lambda generator throwing away sent value
0002-lambda-generator-fix-try-to-add-a-dis-test.patch bru, 2015-03-02 11:16 Failed test tentative
0001-Add-tests-for-issue-23192.patch bru, 2015-03-11 12:00
Messages (8)
msg233662 - (view) Author: Chris Angelico (Rosuav) * Date: 2015-01-08 14:42
As yield is an expression, it's legal in a lambda function, which then
means you have a generator function. But it's not quite the same as
the equivalent function made with def:

$ python3
Python 3.5.0a0 (default:1c51f1650c42+, Dec 29 2014, 02:29:06)
[GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> f=lambda: (yield 5)
>>> x=f()
>>> next(x)
5
>>> x.send(123)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>> def f(): return (yield 5)
...
>>> x=f()
>>> next(x)
5
>>> x.send(123)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration: 123
>>> x = (lambda: print((yield 1)) or 2)()
>>> next(x)
1
>>> x.send(3)
3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

The last example demonstrates that send() is working, but the return value is not getting propagated. Disassembly shows this:

>>> dis.dis(lambda: (yield 5))
  1           0 LOAD_CONST               1 (5)
              3 YIELD_VALUE
              4 POP_TOP
              5 LOAD_CONST               0 (None)
              8 RETURN_VALUE
>>> def f(): return (yield 5)
... 
>>> dis.dis(f)
  1           0 LOAD_CONST               1 (5)
              3 YIELD_VALUE
              4 RETURN_VALUE

I'm sure this is a bug that will affect very approximately zero people, but it's still a peculiar inconsistency!

Verified with 3.5 and 3.4.
msg233668 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-01-08 15:56
Hm, looks like nobody bothered to update the lambda code generation to use the value from yield. I almost feel like there is some unnecessary check "if we are in a lambda" in the code generation for yield. Have you looked through the code generation yet?
msg233673 - (view) Author: Chris Angelico (Rosuav) * Date: 2015-01-08 17:27
I'm not sure what to look for in the code generation. In compile.c lines 3456 and following, there's a LOAD_CONST None coming through, in the else branch of "if (e->v.Yield.value)", but nothing talking about lambda functions. There are constants COMPILER_SCOPE_LAMBDA and COMPILER_SCOPE_FUNCTION, but the only place where they're used is compiler_set_qualname() and I can't see anything obvious there. Hopefully someone more familiar with the code internals will be able to figure this out!
msg237041 - (view) Author: Bruno Cauet (bru) * Date: 2015-03-02 11:16
Here are the operations being emitted (line, macro used and eventual argument):

>>> f = lambda: (yield 5)
3487: ADDOP_O LOAD_CONST e->v.Num.n
3472: ADDOP YIELD_VALUE
1907: ADDOP_IN_SCOPE POP_TOP
4349: ADDOP_O LOAD_CONST Py_None
4350: ADDOP RETURN_VALUE
1457: ADDOP_O LOAD_CONST (PyObject*)co
1458: ADDOP_O LOAD_CONST qualname
1459: ADDOP_I MAKE_FUNCTION args
4349: ADDOP_O LOAD_CONST Py_None
4350: ADDOP RETURN_VALUE
>>> def g(): return (yield 5)
... 
3487: ADDOP_O LOAD_CONST e->v.Num.n
3472: ADDOP YIELD_VALUE
2533: ADDOP RETURN_VALUE
1457: ADDOP_O LOAD_CONST (PyObject*)co
1458: ADDOP_O LOAD_CONST qualname
1459: ADDOP_I MAKE_FUNCTION args
4349: ADDOP_O LOAD_CONST Py_None
4350: ADDOP RETURN_VALUE

So there's an extra POP_TOP + LOAD_CONST Py_NONE for the lambda version that throws away the "123" (in the exemple).

The attached patch (0001-...) fixes it. However please note that I'm not knowledgable about that part of the code and devised the patch empirically.
Moreover a test should probably added but I did not know where (test_dis? tried and failed... see patch 0002-...).
msg237804 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-10 19:38
Could you please add a test based on Chris's example? And it would be good to add a test for a lambda with "yield from".
msg237861 - (view) Author: Bruno Cauet (bru) * Date: 2015-03-11 12:00
Here is a working test, testing yield by lambda & function as well as lambda and function yielding from those.
msg237883 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-11 16:23
New changeset 2b4a04c3681b by Serhiy Storchaka in branch '3.4':
Issue #23192: Fixed generator lambdas.  Patch by Bruno Cauet.
https://hg.python.org/cpython/rev/2b4a04c3681b

New changeset a3b889e9d3f3 by Serhiy Storchaka in branch 'default':
Issue #23192: Fixed generator lambdas.  Patch by Bruno Cauet.
https://hg.python.org/cpython/rev/a3b889e9d3f3
msg237884 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-11 16:25
Committed with non-dis test. Thank you for your contribution Bruno.
History
Date User Action Args
2015-03-11 16:25:22serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg237884

stage: test needed -> resolved
2015-03-11 16:23:18python-devsetnosy: + python-dev
messages: + msg237883
2015-03-11 12:00:52brusetfiles: + 0001-Add-tests-for-issue-23192.patch

messages: + msg237861
2015-03-10 19:38:55serhiy.storchakasetassignee: serhiy.storchaka
messages: + msg237804
stage: test needed
2015-03-02 11:16:40brusetfiles: + 0002-lambda-generator-fix-try-to-add-a-dis-test.patch
2015-03-02 11:16:23brusetfiles: + 0001-lambda-generators-don-t-throw-away-stack-top.patch

nosy: + bru
messages: + msg237041

keywords: + patch
2015-03-02 08:29:57ezio.melottisetnosy: + ezio.melotti
2015-01-08 17:27:16Rosuavsetmessages: + msg233673
2015-01-08 16:02:25vstinnersetnosy: + vstinner
2015-01-08 15:56:52gvanrossumsetmessages: + msg233668
2015-01-08 14:52:37serhiy.storchakasetnosy: + gvanrossum, benjamin.peterson, serhiy.storchaka
2015-01-08 14:42:18Rosuavsettype: behavior
2015-01-08 14:42:05Rosuavcreate