Yury, are you planning to do more for this for 3.6.0?  This is marked as a release blocker but, since it's just a doc change, I'm not going to hold 3.6.0 for it.

Hi, I'm a new contributor trying to get started with documentation and testing.

I have been following and testing PEP525 and PEP530 and feel familiar enough to try to work on adding this to the Python Language Reference if it would not be redundant with work already being completed. I think this makes sense to tackle along with issue 28090 as it appears that documentation of asynchronous generator expressions would necessarily refer to terminology from asynchronous generator objects. 

Specifically, for PEP525:

async generator functions would be added first in section 3.2 after coroutines.

Most of the work would be in expanding 6.2.9 (yield expressions) to discuss the asynchronous case, probably adding 6.2.9.3(4) to document the methods for asynchronous generator objects.

In section 8.8.1 I would need to remove the statement “it is a syntax error to use yield expressions in async def coroutines”, add statement that “yield from” is a syntax error in this context.

Finally, the new concepts should be added to the Glossary.

Then for PEP530:

Section 6.2.4 (Displays for Lists,...) needs to be updated for the grammar change that adds the optional ASYNC, i.e.:

comp_for: [ASYNC] 'for' exprlist 'in' or_test [comp_iter]

Then both sections 6.2.4 and 6.2.8  (Generator expressions) need to be expanded to include the async case.

If writing this up is not already underway, and if I’m generally on the right track for what needs to be done, I think I can have a reasonable patch for this by the end of the week. Also, please feel free to contact me if there is any other documentation/testing/test coverage work that I could help with. The new native coroutines are an area of particular interest for me.

Hi Eric, please feel free to go ahead and submit a patch, I'll review it!

Eric, are you working on a patch?  If not, I'll be focusing on this myself in the next couple of days.

Hi Yury - Yes, I am excited about doing this and getting to work on it. Today was pretty busy but I have some time blocked out tomorrow to really dig in. I was planning to give myself a deadline of Friday afternoon if that works for the review/release schedule. If needed I can try to get it up earlier.

Friday is good, take your time!

I think this needs considerable checking and polishing, but I am submitting this "Work In Progress" patch for the PEP525 and PEP530 documentation so that I can get some general feedback since I am new to this, and to ensure that if this is not generally what was needed or expected I don't delay the effort.

I tried to start where possible by using the wording in corresponding generator sections, so this may err on the side of redundant phrasing, which I can change.

I had some difficulty with the following points, and am not sure if they merit opening other tickets:

1. In PEP525 the documentation for aclose() is a bit terse and unclear to me. It appeared to suggest that you could catch GeneratorExit and yield, but I found this to result in a RuntimeError like a normal generator. I tried to document this as it actually behaves.

2. One thing that I noticed documented about normal generators is that they raise a ValueError if you try to run send() while another send() call is currently running. I verified this using threads. I looked into corresponding behavior for asynchronous generators, calling asend(), running the awaitable halfway through, and then calling asend() again to get a second awaitable before the first one finished. Asyncio seems to prevent more than one awaitable from a single async generator running at the same time, but I couldn't figure out how. Running some coroutines "by hand" calling asend() and send(), I was permitted to run multiple awaitables concurrently which produced odd results.

I did not attempt to document the behavior in (2) in the submitted patch.

> 1. In PEP525 the documentation for aclose() is a bit terse and unclear to me. It appeared to suggest that you could catch GeneratorExit and yield, but I found this to result in a RuntimeError like a normal generator. I tried to document this as it actually behaves.

Yes, it should result in a RuntimeError. What PEP 525 is trying to explain is that it's OK to do this (although 'finally' is better):

   async def gen():
      try:
         yield
      except GeneratorExit:
         await smth()
         # using 'yield' here will trigger a RuntimeError
         raise

> 2. One thing that I noticed documented about normal generators is that they raise a ValueError if you try to run send() while another send() call is currently running. I verified this using threads. I looked into corresponding behavior for asynchronous generators, calling asend(), running the awaitable halfway through, and then calling asend() again to get a second awaitable before the first one finished. Asyncio seems to prevent more than one awaitable from a single async generator running at the same time, but I couldn't figure out how. Running some coroutines "by hand" calling asend() and send(), I was permitted to run multiple awaitables concurrently which produced odd results.

Interesting.  This is something that has to be fixed (in 3.6.1)

Eric, I've left you some review comments.

Guido, Nick, I have a question I hope you have an opinion on: for synchronous generators we define the following (sometimes confusing) terminology:

- "generator" = generator function
- "generator iterator" = generator object

I'm thinking if we can define asynchronous generators slightly differently:

- "asynchronous generator function"
- "asynchronous generator"

It sounds particularly confusing that an "[XXX] generator" is the function
in one case and the object in the other case.

There's of course also Nathaniel Smith's opinion that we should switch away
from the term coroutine and call those "async def". Is that relevant?

> It sounds particularly confusing that an "[XXX] generator" is the function
in one case and the object in the other case.

Right, this is the biggest argument against having slightly different terminology between async generators and sync generators.

Maybe we should update docs for synchronous generators and replace "generator" with "generator function" and "generator iterable" with "generator"?

Most of the existing documentation refers to generator objects when the term "generator" is used anyways.

> There's of course also Nathaniel Smith's opinion that we should switch away
from the term coroutine and call those "async def". Is that relevant?

IIRC he's advocating to refer to "coroutine functions" as "async functions" and to "coroutines" as "async something". I personally don't buy his arguments and think that the current terminology is good and people already seem to understand/like it.

I think that most documentation uses "generator" to refer to both functions
and objects. So whenever you want to be clear you have to write the long
form.

I agree with Guido that the in-practice terminology for synchronous generators is:

 "generator" = infer from context (and/or the ambiguity doesn't matter)
 "generator function" = generator iterator factory defined with "def"
 "generator iterator" = result of calling a generator function

Hence the caveat on https://docs.python.org/3/glossary.html#term-generator: "Usually refers to a generator function, but may refer to a generator iterator in some contexts. In cases where the intended meaning isn’t clear, using the full terms avoids ambiguity."

For async I believe you may be better off avoiding the term "generator" entirely, and instead going with "native async iterator":

  "native async iterator" = infer from context
  "native async iterator function" = async iterator factory defined with "async def"
  "native async iterator object" = result of calling a native async iterator function

Yes, technically there's generator machinery under the hood, but I think we're better off letting the discovery of "Oh, coroutines, native async iterators, and generators are all basically the same thing at runtime" be delayed until new users are further along in the process of using Python productively.

Guido, Nick,  

OK, let's keep using the current terminology.

Eric,

Will you have time to make a new version of the patch (addressing the review comments)?

Yes - I'll work on the patch tonight.

I believe that I addressed all the comments in the previous review (although its always possible I missed something), and have a new patch with the improvements and fixes.

I also noticed that in asyncio, loop.shutdown_asyncgens() is a coroutinemethod and fixed the markup to reflect that as I previously labeled it a method. The example I added here is as suggested, but it differs from the example in PEP525 in that shutdown_asyncgens() is run in a finally clause, which makes sense to me.

When applying the comments to sys.set_asyncgen_hooks() I verified that the keywords are optional but it is apparently not keyword only. I couldn't find many good examples of referencing source, so you may want to closely check how I linked to the shutdown_asyncgens() implementation.

> I believe that I addressed all the comments in the previous review (although its always possible I missed something), and have a new patch with the improvements and fixes.

Thanks Eric!  I've left a few more comments in the review.  Almost there!

> I also noticed that in asyncio, loop.shutdown_asyncgens() is a coroutinemethod and fixed the markup to reflect that as I previously labeled it a method. The example I added here is as suggested, but it differs from the example in PEP525 in that shutdown_asyncgens() is run in a finally clause, which makes sense to me.

Right, I'll fix the PEP. Thanks!

Thanks for the explanations and example code in the review! Working on this issue is really helping to improve my understanding of a number of tricky things like finalization.

I *think* that I have all the comments implemented in the attached patch.

I also removed a spurious 2-line fragment at the top of the Asynchronous generator-iterator methods section that I had neglected to delete and escaped review.

New changeset 78c8f450b84c by Yury Selivanov in branch '3.6':
Issue #28091: Document PEP 525 & PEP 530.
https://hg.python.org/cpython/rev/78c8f450b84c

New changeset 6bf84e661e69 by Yury Selivanov in branch 'default':
Merge 3.6 (issue #28091)
https://hg.python.org/cpython/rev/6bf84e661e69

Pushed.  Thanks a lot, Eric, very good job!  (BTW, please enable trailing whitespace trimming in your editor)

Ned, maybe you want to cherry-pick this commit to 3.6.0, but I guess most people just use docs.python.org, so it's not that important.

New changeset 5a11263bb612 by Yury Selivanov in branch '3.6':
Issue #28091: Document PEP 525 & PEP 530.
https://hg.python.org/cpython/rev/5a11263bb612

[cherrypicked for 3.6.0rc2]