classification
Title: Uncaught comprehension SyntaxError eats up all memory
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder: yield expression inside generator expression does nothing
View: 10544
Assigned To: docs@python Nosy List: BreamoreBoy, docs@python, georg.brandl, ncoghlan, scoder, serhiy.storchaka, terry.reedy
Priority: normal Keywords:

Created on 2009-08-09 14:09 by scoder, last changed 2018-05-09 08:40 by serhiy.storchaka. This issue is now closed.

Messages (11)
msg91428 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2009-08-09 14:09
Here's a simple coroutine that works perfectly in Python 2.6 but seems
to let Py3.1 enter an infinite loop that ends up eating all memory.

-----------------
def printing_sink():
    "A simple sink that prints the received values."
    while True:
        print( (yield) )

def chunker(chunk_size, target):
    """Receives single items and forwards chunks of a fixed size.

    Usage example:
    >>> sink = printing_sink()
    >>> next(sink)
    >>> cr = chunker(4, sink)
    >>> next(cr)

    >>> for i in range(8):
    ...    cr.send(i)
    [0, 1, 2, 3]
    [4, 5, 6, 7]
    >>> cr.close()
    """
    while True:
        target.send([ (yield) for i in range(chunk_size) ])

if __name__ == '__main__':
    import doctest
    doctest.testmod()
-----------------

Fails on:
Python 3.1 (r31:73572, Jun 28 2009, 21:07:35)
[GCC 4.3.2] on linux2

Works on:
Python 2.6.2 (r262:71600, Apr 17 2009, 11:29:30)
[GCC 4.3.2] on linux2

The problem seems to be the list comprehension. When I replace it with a
normal for-loop, it works perfectly.
msg91432 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-08-09 21:22
Try list(genexp) instead of [listcomp] in 2.x and see what happens...
msg91435 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2009-08-09 22:06
Hmm, ok, so this is actually an anticipated bug? And I assume this has
been discussed before and was decided to get solved by doing... what?

Is it documented somewhere why this happens and what one must avoid to
not run into this kind of pitfall?
msg91436 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-08-09 22:09
No idea, actually. I just wanted to point out that it is nothing
specific to Python 3.
msg91488 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-08-11 21:58
Not a bug.

The list comprehension in your chunker:

    while True:
        target.send([ (yield) for i in range(chunk_size) ])

is equivalent to the following generator in Python 3:

    while True:
        def g():
            for i in range(chunk_size):
                yield (yield)
        target.send(list(g()))

This clearly needs not what you want. So, just rewrite your code using
for-loop:

    while True:
        result = []
        for i in range(chunk_size):
            result.append((yield))
        target.send(result)
msg91582 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-08-14 23:53
Reopening - this should be rejected by the compiler as a SyntaxError,
since it is the comprehension equivalent of writing "return Value"
inside a generator function.

Specifically, in 3.x, the equivalent written out code is:

    while True:
        def _listcomp():
            result = []
            for i in range(chunk_size):
                result.append((yield))
            return result # Would trigger SyntaxError!
        target.send(_listcomp())

As noted in the comment, the compiler would disallow that expansion with
a SyntaxError on the marked line, so the comprehension equivalent should
be rejected as well. This also applies to dict and set comprehensions.

Generator expressions are slightly less clear, since their expanded
equivalent would produce legal code:

>>> g = ((yield)*(yield) for i in range(1))
>>> next(g)
>>> g.send(2)
>>> g.send(3)
6
>>> next(g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

That first line is roughly equivalent to:
        def _genexp():
            for i in range(1):
                yield (yield)*(yield)
        g = _genexp()

So the compiler should probably be keeping track of whether it is inside
a comprehension or not to decide whether or not to allow yield expressions.
msg91656 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2009-08-17 06:32
Very good argumentation, thanks Nick!

I think this is worth being fixed in the 3.1 series.
msg139227 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-06-26 21:32
With 32bit winxp Windows Task Manager open to Processes, sorted by Men Usage, I verified expanding memory usage in 3.2.0 (and stopped at 200 mb versus normal <20Mb). While we do not expect compiler to catch while True: pass, it would be nice if this was, with a reasonable error message.

No one marked for 2.7 because behavior only occurs with genexp and Nick says not so clearly a SyntaxError.
msg139234 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-06-27 01:47
The acceptance of PEP 380 actually muddies the waters on this one - the expansion I cited as a syntax error now has a defined meaning. Specifically, list comprehensions containing yield expressions would become generators that return a list (set and dict comprehensions would become generators that return a set or dict, respectively).

Once it is confirmed those semantics are an acceptable outcome of PEP 380's acceptance, this behaviour would be documented in the appropriate locations (likely under the yield expression, with pointers from other parts of the documentation).
msg221003 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-06-19 16:10
Assuming that documentation changes are needed, who's best placed to do them?
msg316315 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-05-09 08:40
"yield" in comprehensions is deprecated in 3.7:

../issue6673.py:22: DeprecationWarning: 'yield' inside list comprehension
  target.send([ (yield) for i in range(chunk_size) ])

and an error in 3.8:

  File "../issue6673.py", line 22
    target.send([ (yield) for i in range(chunk_size) ])
                 ^
SyntaxError: 'yield' inside list comprehension
History
Date User Action Args
2018-05-09 08:40:47serhiy.storchakasetstatus: open -> closed

superseder: yield expression inside generator expression does nothing

nosy: + serhiy.storchaka
messages: + msg316315
resolution: duplicate
stage: needs patch -> resolved
2014-06-19 16:10:56BreamoreBoysetnosy: + BreamoreBoy
messages: + msg221003
2013-12-07 09:16:42alexandre.vassalottisetnosy: - alexandre.vassalotti
2013-11-30 19:52:03alexandre.vassalottisetassignee: docs@python
components: + Documentation, - Interpreter Core
versions: + Python 3.4, Python 3.5, - Python 3.2, Python 3.3
keywords: - 64bit
nosy: + docs@python

type: resource usage -> enhancement
stage: needs patch
2011-06-27 01:47:23ncoghlansettype: crash -> resource usage
messages: + msg139234
2011-06-26 21:32:59terry.reedysetnosy: + terry.reedy
title: Py3.1 hangs in coroutine and eats up all memory -> Uncaught comprehension SyntaxError eats up all memory
messages: + msg139227

versions: + Python 3.2, Python 3.3, - Python 3.1
2009-08-17 06:32:14scodersetmessages: + msg91656
2009-08-17 06:29:07scodersetstatus: closed -> open
2009-08-14 23:53:10ncoghlansetnosy: + ncoghlan
messages: + msg91582

keywords: + 64bit
resolution: not a bug -> (no value)
2009-08-11 21:58:23alexandre.vassalottisetstatus: open -> closed

nosy: + alexandre.vassalotti
messages: + msg91488

resolution: not a bug
2009-08-09 22:09:23georg.brandlsetmessages: + msg91436
2009-08-09 22:06:26scodersetmessages: + msg91435
2009-08-09 21:22:06georg.brandlsetnosy: + georg.brandl
messages: + msg91432
2009-08-09 14:09:44scodercreate