classification
Title: Python 2.6 - Python 3.4 allows unparenthesized generator with *args, **kw, forbidden in 3.5
Type: behavior Stage: resolved
Components: Documentation, Interpreter Core Versions: Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, python-dev, r.david.murray, skrah, vstinner, yselivanov, ztane
Priority: normal Keywords:

Created on 2015-09-11 13:03 by ztane, last changed 2015-09-12 21:52 by yselivanov. This issue is now closed.

Messages (10)
msg250468 - (view) Author: Antti Haapala (ztane) * Date: 2015-09-11 13:03
User DeTeReR asked a question (http://stackoverflow.com/questions/32521140/generator-as-function-argument) on Stack Overflow about a special case of code that seemed to work in Python 3.4:

    f(1 for x in [1], *[2])

and

    f(*[2], 1 for x in [1])

I found out that when Python 2.6 introduced the "keyword arguments after *args", the checks in ast.c did not follow:

    for (i = 0; i < NCH(n); i++) {
        node *ch = CHILD(n, i);
        if (TYPE(ch) == argument) {
            if (NCH(ch) == 1)
                nargs++;
            else if (TYPE(CHILD(ch, 1)) == gen_for)
                ngens++;
            else
                nkeywords++;
        }
    }
    if (ngens > 1 || (ngens && (nargs || nkeywords))) {
        ast_error(n, "Generator expression must be parenthesized "
                  "if not sole argument");
        return NULL;
    }

the *args, **kwargs were not considered to be of type "argument" by the Grammar, and thus the error was not generated in this case.

Further down, the error "non-keyword arg after keyword arg" was not triggered in the case of sole unparenthesized generator expression.

Now, the parsing changes in 3.5 have disallowed all of these constructs:

    f(1 for i in [42], **kw)
    f(1 for i in [42], *args)
    f(*args, 1 for i in [42])

which were (erroneously) allowed in previous versions.

I believe at least 3.5 release notes should mention this change.
msg250469 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-09-11 13:15
I guess that it's a consequence of the PEP 448 "additional unpacking generalizations".
msg250470 - (view) Author: Antti Haapala (ztane) * Date: 2015-09-11 13:25
@haypo yes.

I must add that I found out that Python 2.5 also allows 

    f(1 for x in [1], *a)

and 

    f(1 for x in [1], **kw)

but not

    f(*a, 1 for x in [1])

So I do not know if the first and second cases were intentional or not.
Also, in Python 2.6 - 3.4, f(*a, 1 for x in [1]) provides the generator as the *first* positional argument, in 3.5 it'd be the last one.
msg250471 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2015-09-11 13:46
According to the human readable grammar in the docs

   https://docs.python.org/3/reference/expressions.html#calls

I'd say this was a bug that is now fixed in 3.5. The call should
either take a single unparenthesized generator expression or an
argument_list (which may contain parenthesized generator expressions).
msg250473 - (view) Author: Antti Haapala (ztane) * Date: 2015-09-11 13:50
Yeah, it is a bug in 2.5 too; https://docs.python.org/2.5/ref/calls.html

    call ::= primary "(" [argument_list [","]
		         | expression genexpr_for] ")"
msg250500 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-09-11 21:40
If we have decided that this is a "fixed bug", it should indeed be mentioned in what's new for 3.5 since it is a behavior change.
msg250511 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2015-09-12 03:36
> If we have decided that this is a "fixed bug", [..]

I'd call this a bug fix.  The old syntax was completely unreadable.  

Should I close this issue?

> [..] it should indeed be mentioned in what's new for 3.5 since it is a behavior change.

Sure, I'll document it.
msg250519 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2015-09-12 12:10
I think the issue can be closed (after updating "what's new").

Even if we wanted to keep backwards compatibility, it's too
late now.
msg250535 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-09-12 21:51
New changeset 10a9d4acd9cb by Yury Selivanov in branch '3.5':
whatsnew/3.5 More edits
https://hg.python.org/cpython/rev/10a9d4acd9cb
msg250536 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2015-09-12 21:52
What's new is now updated with description of this issue.  Closing it now.
History
Date User Action Args
2015-09-12 21:52:41yselivanovsetstatus: open -> closed
resolution: wont fix
messages: + msg250536

stage: resolved
2015-09-12 21:51:23python-devsetnosy: + python-dev
messages: + msg250535
2015-09-12 12:10:46skrahsetmessages: + msg250519
2015-09-12 03:36:39yselivanovsetmessages: + msg250511
2015-09-12 03:28:12terry.reedysetversions: - Python 3.2, Python 3.3
2015-09-11 21:40:29r.david.murraysetnosy: + r.david.murray
messages: + msg250500
2015-09-11 13:50:47ztanesetnosy: + docs@python
messages: + msg250473

assignee: docs@python
components: + Documentation
2015-09-11 13:46:51skrahsetnosy: + skrah
messages: + msg250471
2015-09-11 13:25:48ztanesetmessages: + msg250470
2015-09-11 13:15:52vstinnersetnosy: + vstinner, yselivanov
messages: + msg250469
2015-09-11 13:03:44ztanecreate