Creating an issue to track efforts to forward port the PEP-380 reference implementation to Python 3.3.

(“Create patch” does not work for some reason, I’ll report that to the metatracker in a few days if nobody does it first)

It's because it isn't a true Mercurial clone-and-update - it's a patch queue, which the review generation code doesn't know how to interpret.

As nick said, the repo only host a patch queue.
the patch itself is visible here:
https://bitbucket.org/rndblnch/cpython-pep380/qseries?apply=t&qs_apply=pep380

Or it can be download here in raw text:
https://bitbucket.org/rndblnch/cpython-pep380/raw/tip/pep380

I have documented how to apply the patch to a cpython clone here:
https://bitbucket.org/rndblnch/cpython-pep380/wiki

OK, this isn't going to be as simple as I thought it was. The big problem is that the test suite from [1] needs to be refactored into a format that is suitable for use in regrtest. That means porting them to either unittest or doctest, as "golden output" tests of the style that Greg used are no longer accepted as part of the regression test suite.

[1] http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yield_from.html

Renaud has updated the patch on bitbucket to incorporate Greg's tests (still in golden output form, but in a unittest/regrtest compatible way). That's a good enough starting point for me - they can be refactored into proper assert based tests later.

The only tweak needed is to explicitly save and restore stdout and stderr, since restoring the OS level ones often does the wrong thing when running the regression test suite.

I've just updated the PEP-380-test patch to make it slightly simpler (still in golden output form though)

https://bitbucket.org/rndblnch/cpython-pep380/src/317eadf5e3e8/pep380-tests

Once again got close to committing this, but then realised it is missing all the necessary documentation updates for a core language change.

I have uploaded a patch that includes all the changes from Bitbucket as well as the subsequent fixes to avoid modifying sys.stdout when the tests are run and to comply with the whitespace rules set up in the source control hooks (the ACKS, NEWS and a placeholder in whatsnew are also included).

What's missing are updates to at least:

http://docs.python.org/dev/tutorial/classes.html#generators
(a simple example showing delegation should suffice there)

http://docs.python.org/dev/reference/simple_stmts.html#the-yield-statement
http://docs.python.org/dev/reference/expressions.html#grammar-token-yield_expression
http://docs.python.org/dev/reference/simple_stmts.html#the-return-statement
(the language reference must be updated for a post PEP-380 world)

There are likely other places that should also be updated, but these are the critical ones needed before the patch can be included.

I can not comment on http://bugs.python.org/review/11682/, so a quick comment here: Doc/whatsnew/3.3.rst patch gives me credit together with Greg Ewing for the implementation, but I've only upgraded his patches to 3.3.
So, the following line:
+(Implementation by Greg Ewing and Renauld Blanch)
should really be:
+(Implementation by Greg Ewing)

And, as I'm on it, the proper spelling for my surname is Renaud (not Renauld as it also appears in the commit message :)

I moved my personal sandbox from hg.python.org to bitbucket, so it should be easier for folks to build off the work in progress. (And fixed the typo in Renaud's name - I at least had it right in ACKS. I also reworded the draft attribution text in What's New)

I skimmed Benjamin's comments and basically agree with them. I've marked bpo-11816 as a dependency as some of the new tests will be easier to refactor once dis.get_opinfo() is available.

The PEP-380 branch in my bitbucket repo has been updated with the refactored tests that Ryan Kelly put together at the PyconAU sprints (as well as being brought up to speed with 3.x tip).

The update depends on the new dis.get_opinfo() API added by issue bpo-11816 (which is currently with Raymond for review).

The AST validator still needs to be updated to take into account the new syntax and the patch attribution needs to be updated to account for Ryan's contribution, but this is getting fairly close to being ready.

Uploaded a patch that should be complete. Note that my PEP-380 branch is based on my get_opinfo branch (see bpo-11816), so if you're applying patches manually rather than updating directly from my sandbox with hg, you'll need to apply the latest patch from bpo-11816 before applying this one.

Raymond: assigning to you, since you said you wanted to review the dis.get_opinfo changes before they went in, and those changes are a dependency for the updated PEP-380 test suite.

Actually, not assigning to Raymond for review yet, after all - I just noticed there are some of Benjamin's review comments relating to cosmetic details rather than functionality that I still need to address.

I'll kick it in Raymond's direction once I've dealt with those (probably next weekend, but possibly some time this week)

I've created some documentation... The patches are the bitbucket repo.

Nothing is added to the tutorial, because I think that this isn't material for a newcomer to python. The tutorial doesn't mention generator.throw() and send() either, just talks a little about writing simple generator functions.

I have updated the bitbucket repo with changes to address most of Benjamin's review comments.

A few points of note:

• I agree in principle with the idea of splitting Yield and YieldFrom into distinct AST nodes, but I'd prefer to focus on getting the current implementation into the main repo first
• in cleaning up the handling of missing send/close/throw attributes on subiterators I discovered that the PyObject_CallMethod* APIs were discarding exception information and coercing everything to a terse AttributeError. The branch now changes them to allow the error reported by the underlying call to PyObject_GetAttr to pass through unmodified.
• the generator close() method treats an AttributeError as expected when looking for a close() method on the subiterator, but uses WriteUnraisable to deal with anything else.
• I share Benjamin's suspicion that some of the INCREF/DECREF pairs in genobject.c aren't actually necessary, but I don't think it's worth messing with them at this stage.

I haven't looked at Zbyszek's proposed doc changes at this point - the pull request has a lot of irrelevant line wrapping changes in it, so it makes it hard to review.

Here are the un-reflowed documentation changes, as a patch this time. I've split the changes in two:
0001 is the real documentation change
0002 is the link fixes [optional].

Doc patch 002 is not strictly related to PEP-380 but pertains more to bpo-10289.

A little bit of input on this issue. Considering that exceptions are now getting keyword arguments for things like import errors and other things for attributes I would find it much better if StopIteration would follow that as well (StopIteration(value=42) instead of StopIteration(42)).

Bitbucket repo and attached patch updated relative to latest get_opinfo branch (which in turn was updated to apply cleanly against current CPython tip).

(I still need to incorporate the doc updates and look into adding keyword argument support)

It looks like there is a memory leak bug (on StopIteration exception instances).

Attached is the test to expose it.

It seems that adding 'Py_DECREF(e);' after 'PyErr_SetObject(PyExc_StopIteration, e);' in 'genobject.c' fixes the leak.

In reviewing Zbyszek's doc updates and comparing them against the Grammar, I discovered a gratuitous change in the implementation: it allows a bare (i.e. no parentheses) 'yield from' as an argument to a function, even when there's multiple arguments. By contrast, an ordinary yield expression requires parentheses everywhere other than when it's being used as a statement, or as the sole expression on the RHS of an assignment.

I'll add a new test to ensure "yield from x" requires parentheses whenever "yield x" requires them (and fix the Grammar file on the implementation branch accordingly).

I've also pushed a fix to the branch for the refleak Yury pointed out (once I actually ran regrtest in refleak hunting mode it also pointed out that there was a problem, so I know his suggested change definitely fixed it).

OK, the bitbucket repo now has a more sane version of the new Grammar ('yield from' now requires parentheses wherever 'yield' does).

The updated test_grammar does a more thorough check of the expected acceptance and rejection of yield expressions in various situations and the overly sensitive test in test_parser (that checked for an exact expected parse tree output) is gone.

Nick Coghlan wrote:

I don't want to completely rearrange the yield related sections of the
language reference as part of incorporating this PEP. If you're happy
to submit a new pull request with a minimalist change just documenting
the new features, that would be great, otherwise I'll eventually
figure out my own set of updates.

OK, I should have it ready before Monday.

The cross-linking fixes are independent of this PEP, and should be
handled as a separate tracker issue rather than being rolled into the
PEP update.
Will do so.

Zbyszek

Updated doc patch 0001-2.diff: following ncoghlan's request, the bulk of yield documentation is kept in expressions.rst, and simple_stmts.rst mostly refers to the other one. (In previous version it was the other way around).

After doing this change, I still think that it is better to have most of the documentation in the _statement_ part, and keep the _expression_ part relatively simple. The reason is that yield does more than just return a value, but also throws exceptions, suspends execution, etc. This is detached from the fact that is can be used in an expression.

Unfortunately yield as an expression is the only case where a statement can be used in an expression. Therefore there aren't any other cases which could be used as a guide.

*Any* expression can be used as a standalone statement and (since PEP-352 was implemented) that now applies to 'yield' as well.

PEP-352 fundamentally changed the way yield was conceptualised within the language - thinking of it as "a statement that can be used as an expression" is just plain *wrong*. It's now just an expression that happens to have a couple of special cases in the grammar to allow the parentheses to be skipped when they're distracting rather than helpful.

That is, the only reason the yield statement still exists as a separate entity is to allow us to drop the otherwise mandatory parentheses:

(yield val) # yield expression as statement
yield val # yield statement

There is a test for 'yield from' as a function argument without the extra parentheses.

      f(yield from x)

You do need them in the case of a regular yield.

  f((yield))   or f((yield value))

Shouldn't the same rule apply in both cases?

• I'm trying to do a version of the patch with 'yield_from' as a separate item from 'yield' in the grammar, but it insists on the parentheses due to it being a yield_expr component.

>> f((yield))

This requirement seems unnecessary. And surprising, because f(<generator-expression>) or f('a' if 'a' else 'b') doesn't require parenthes. There's no room for confusion if parentheses were omitted in the single-argument case. (Following the rule given in documentation for generator expressions: "The parentheses can be omitted on calls with only one argument.").

OK, I removed all the old files and repo links (they're still available in the history below if needed). (well, the link to Renaud's hg patch queue is gone, but that's also present in one of the early comments). (Ron: your comment makes me think you were looking at an old version of the PEP-380 updates. The most up to date published version is the tip of my BitBucket branch)

Also, don't spend a lot of time messing with functional aspects of the patch folks, especially the Grammar. 'yield from' is going to be a variant of the existing yield expressions to avoid parser ambiguity, and it's going to inherit all of the associated restrictions, including the requirement for parentheses when it's the sole argument to a function.

While that restriction isn't *necessary*, it's also irrelevant to this PEP (hence why I reverted Greg's changes that allowed it for 'yield from', but not for other yield expressions).

As far as *why* yield expressions aren't special cased the way generator expressions are - just an oversight when yield expressions were added, no real grand master plan. It's not a big deal in practice, so nobody has ever cared enough to argue for fixing it.

Conditional expressions simply don't include mandatory parentheses as part of the syntax in the first place.

Thanks for the updated links Nick.

There is a comment in the docs that recommends putting parentheses around any yield expression that returns a value. So it is in agreement with that in the function argument case.

The grammar I used does keep it as a variant.

yield_expr 'yield' [testlist | yield_from]
yield_from 'from' test

The purpose of doing that is so I can do ...

yield_expr 'yield' [testlist | yield_from | yield_raise]
yield_from 'from' test
yield_raise 'raise' [test ['from' test]]

The yield_raise part is only an interesting exercise to help me understand cpythons internals better. I'm getting there and hope to be able to contribute to more bug fixes, and patches. :-)

Nice milestone this evening: by incorporating doc updates based on Zbysek's efforts and dropping the explicit bytecode generation tests, I now have something that appears ready to commit *without* a dependency on the proposed dis module updates.

So, assuming nobody spots any major problems between now and then, the PEP-380 implementation should be landing in the main repo some time this weekend :)

Some minor comments in http://bugs.python.org/review/11682/show.

Committed for 3.3: http://hg.python.org/cpython/rev/d64ac9ab4cd0

Thanks to Greg for the initial effort on the PEP and reference implementation and to all involved in updating the original patch for 3.3 and getting the tests and documentation to an acceptable state.

Any issues discovered after this can be given a new tracker entry :)

Kudos!

Just noting that the PEP-380 integration branch is also available in the hg.python.org clone of my sandbox repo.