classification
Title: Document PEP 525 & 530
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: yselivanov Nosy List: Eric Appelt, gvanrossum, ncoghlan, ned.deily, python-dev, yselivanov
Priority: Keywords: patch

Created on 2016-09-12 01:47 by yselivanov, last changed 2016-12-16 07:58 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
pep525_pep530_docs_WIP.patch Eric Appelt, 2016-12-09 04:54 review
pep525_pep530_docs_v1.patch Eric Appelt, 2016-12-13 04:24 review
pep525_pep530_docs_v2.patch Eric Appelt, 2016-12-15 03:39 review
Messages (22)
msg282461 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-12-05 20:19
Yury, are you planning to do more for this for 3.6.0?  This is marked as a release blocker but, since it's just a doc change, I'm not going to hold 3.6.0 for it.
msg282494 - (view) Author: Eric Appelt (Eric Appelt) * Date: 2016-12-06 03:26
Hi, I'm a new contributor trying to get started with documentation and testing.

I have been following and testing PEP525 and PEP530 and feel familiar enough to try to work on adding this to the Python Language Reference if it would not be redundant with work already being completed. I think this makes sense to tackle along with issue 28090 as it appears that documentation of asynchronous generator expressions would necessarily refer to terminology from asynchronous generator objects. 

Specifically, for PEP525:

async generator functions would be added first in section 3.2 after coroutines.

Most of the work would be in expanding 6.2.9 (yield expressions) to discuss the asynchronous case, probably adding 6.2.9.3(4) to document the methods for asynchronous generator objects.

In section 8.8.1 I would need to remove the statement “it is a syntax error to use yield expressions in async def coroutines”, add statement that “yield from” is a syntax error in this context.

Finally, the new concepts should be added to the Glossary.

Then for PEP530:

Section 6.2.4 (Displays for Lists,...) needs to be updated for the grammar change that adds the optional ASYNC, i.e.:

comp_for: [ASYNC] 'for' exprlist 'in' or_test [comp_iter]

Then both sections 6.2.4 and 6.2.8  (Generator expressions) need to be expanded to include the async case.

If writing this up is not already underway, and if I’m generally on the right track for what needs to be done, I think I can have a reasonable patch for this by the end of the week. Also, please feel free to contact me if there is any other documentation/testing/test coverage work that I could help with. The new native coroutines are an area of particular interest for me.
msg282499 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-12-06 04:45
Hi Eric, please feel free to go ahead and submit a patch, I'll review it!
msg282590 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-12-07 03:54
Eric, are you working on a patch?  If not, I'll be focusing on this myself in the next couple of days.
msg282593 - (view) Author: Eric Appelt (Eric Appelt) * Date: 2016-12-07 04:10
Hi Yury - Yes, I am excited about doing this and getting to work on it. Today was pretty busy but I have some time blocked out tomorrow to really dig in. I was planning to give myself a deadline of Friday afternoon if that works for the review/release schedule. If needed I can try to get it up earlier.
msg282594 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-12-07 04:11
Friday is good, take your time!
msg282760 - (view) Author: Eric Appelt (Eric Appelt) * Date: 2016-12-09 04:54
I think this needs considerable checking and polishing, but I am submitting this "Work In Progress" patch for the PEP525 and PEP530 documentation so that I can get some general feedback since I am new to this, and to ensure that if this is not generally what was needed or expected I don't delay the effort.

I tried to start where possible by using the wording in corresponding generator sections, so this may err on the side of redundant phrasing, which I can change.

I had some difficulty with the following points, and am not sure if they merit opening other tickets:

1. In PEP525 the documentation for aclose() is a bit terse and unclear to me. It appeared to suggest that you could catch GeneratorExit and yield, but I found this to result in a RuntimeError like a normal generator. I tried to document this as it actually behaves.

2. One thing that I noticed documented about normal generators is that they raise a ValueError if you try to run send() while another send() call is currently running. I verified this using threads. I looked into corresponding behavior for asynchronous generators, calling asend(), running the awaitable halfway through, and then calling asend() again to get a second awaitable before the first one finished. Asyncio seems to prevent more than one awaitable from a single async generator running at the same time, but I couldn't figure out how. Running some coroutines "by hand" calling asend() and send(), I was permitted to run multiple awaitables concurrently which produced odd results.

I did not attempt to document the behavior in (2) in the submitted patch.
msg282802 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-12-09 20:30
> 1. In PEP525 the documentation for aclose() is a bit terse and unclear to me. It appeared to suggest that you could catch GeneratorExit and yield, but I found this to result in a RuntimeError like a normal generator. I tried to document this as it actually behaves.

Yes, it should result in a RuntimeError. What PEP 525 is trying to explain is that it's OK to do this (although 'finally' is better):

   async def gen():
      try:
         yield
      except GeneratorExit:
         await smth()
         # using 'yield' here will trigger a RuntimeError
         raise

> 2. One thing that I noticed documented about normal generators is that they raise a ValueError if you try to run send() while another send() call is currently running. I verified this using threads. I looked into corresponding behavior for asynchronous generators, calling asend(), running the awaitable halfway through, and then calling asend() again to get a second awaitable before the first one finished. Asyncio seems to prevent more than one awaitable from a single async generator running at the same time, but I couldn't figure out how. Running some coroutines "by hand" calling asend() and send(), I was permitted to run multiple awaitables concurrently which produced odd results.

Interesting.  This is something that has to be fixed (in 3.6.1)
msg282805 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-12-09 21:00
Eric, I've left you some review comments.

Guido, Nick, I have a question I hope you have an opinion on: for synchronous generators we define the following (sometimes confusing) terminology:

- "generator" = generator function
- "generator iterator" = generator object

I'm thinking if we can define asynchronous generators slightly differently:

- "asynchronous generator function"
- "asynchronous generator"
msg282807 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-12-09 21:49
It sounds particularly confusing that an "[XXX] generator" is the function
in one case and the object in the other case.

There's of course also Nathaniel Smith's opinion that we should switch away
from the term coroutine and call those "async def". Is that relevant?
msg282808 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-12-09 22:04
> It sounds particularly confusing that an "[XXX] generator" is the function
in one case and the object in the other case.

Right, this is the biggest argument against having slightly different terminology between async generators and sync generators.

Maybe we should update docs for synchronous generators and replace "generator" with "generator function" and "generator iterable" with "generator"?

Most of the existing documentation refers to generator objects when the term "generator" is used anyways.

> There's of course also Nathaniel Smith's opinion that we should switch away
from the term coroutine and call those "async def". Is that relevant?

IIRC he's advocating to refer to "coroutine functions" as "async functions" and to "coroutines" as "async something". I personally don't buy his arguments and think that the current terminology is good and people already seem to understand/like it.
msg282809 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-12-09 23:01
I think that most documentation uses "generator" to refer to both functions
and objects. So whenever you want to be clear you have to write the long
form.
msg282823 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-12-10 04:17
I agree with Guido that the in-practice terminology for synchronous generators is:

 "generator" = infer from context (and/or the ambiguity doesn't matter)
 "generator function" = generator iterator factory defined with "def"
 "generator iterator" = result of calling a generator function

Hence the caveat on https://docs.python.org/3/glossary.html#term-generator: "Usually refers to a generator function, but may refer to a generator iterator in some contexts. In cases where the intended meaning isn’t clear, using the full terms avoids ambiguity."

For async I believe you may be better off avoiding the term "generator" entirely, and instead going with "native async iterator":

  "native async iterator" = infer from context
  "native async iterator function" = async iterator factory defined with "async def"
  "native async iterator object" = result of calling a native async iterator function

Yes, technically there's generator machinery under the hood, but I think we're better off letting the discovery of "Oh, coroutines, native async iterators, and generators are all basically the same thing at runtime" be delayed until new users are further along in the process of using Python productively.
msg283016 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-12-12 15:54
Guido, Nick,  

OK, let's keep using the current terminology.

Eric,

Will you have time to make a new version of the patch (addressing the review comments)?
msg283059 - (view) Author: Eric Appelt (Eric Appelt) * Date: 2016-12-12 22:24
Yes - I'll work on the patch tonight.
msg283072 - (view) Author: Eric Appelt (Eric Appelt) * Date: 2016-12-13 04:24
I believe that I addressed all the comments in the previous review (although its always possible I missed something), and have a new patch with the improvements and fixes.

I also noticed that in asyncio, loop.shutdown_asyncgens() is a coroutinemethod and fixed the markup to reflect that as I previously labeled it a method. The example I added here is as suggested, but it differs from the example in PEP525 in that shutdown_asyncgens() is run in a finally clause, which makes sense to me.

When applying the comments to sys.set_asyncgen_hooks() I verified that the keywords are optional but it is apparently not keyword only. I couldn't find many good examples of referencing source, so you may want to closely check how I linked to the shutdown_asyncgens() implementation.
msg283151 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-12-14 01:05
> I believe that I addressed all the comments in the previous review (although its always possible I missed something), and have a new patch with the improvements and fixes.

Thanks Eric!  I've left a few more comments in the review.  Almost there!

> I also noticed that in asyncio, loop.shutdown_asyncgens() is a coroutinemethod and fixed the markup to reflect that as I previously labeled it a method. The example I added here is as suggested, but it differs from the example in PEP525 in that shutdown_asyncgens() is run in a finally clause, which makes sense to me.

Right, I'll fix the PEP. Thanks!
msg283234 - (view) Author: Eric Appelt (Eric Appelt) * Date: 2016-12-15 03:39
Thanks for the explanations and example code in the review! Working on this issue is really helping to improve my understanding of a number of tricky things like finalization.

I *think* that I have all the comments implemented in the attached patch.

I also removed a spurious 2-line fragment at the top of the Asynchronous generator-iterator methods section that I had neglected to delete and escaped review.
msg283348 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-12-15 22:36
New changeset 78c8f450b84c by Yury Selivanov in branch '3.6':
Issue #28091: Document PEP 525 & PEP 530.
https://hg.python.org/cpython/rev/78c8f450b84c

New changeset 6bf84e661e69 by Yury Selivanov in branch 'default':
Merge 3.6 (issue #28091)
https://hg.python.org/cpython/rev/6bf84e661e69
msg283349 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-12-15 22:38
Pushed.  Thanks a lot, Eric, very good job!  (BTW, please enable trailing whitespace trimming in your editor)

Ned, maybe you want to cherry-pick this commit to 3.6.0, but I guess most people just use docs.python.org, so it's not that important.
msg283375 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-12-16 07:44
New changeset 5a11263bb612 by Yury Selivanov in branch '3.6':
Issue #28091: Document PEP 525 & PEP 530.
https://hg.python.org/cpython/rev/5a11263bb612
msg283387 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-12-16 07:58
[cherrypicked for 3.6.0rc2]
History
Date User Action Args
2016-12-16 07:58:06ned.deilysetpriority: deferred blocker ->

messages: + msg283387
2016-12-16 07:44:45python-devsetmessages: + msg283375
2016-12-15 22:59:36yselivanovsettitle: Document PEP 525 -> Document PEP 525 & 530
2016-12-15 22:39:13yselivanovlinkissue28090 superseder
2016-12-15 22:38:25yselivanovsetstatus: open -> closed
resolution: fixed
messages: + msg283349

stage: needs patch -> resolved
2016-12-15 22:36:46python-devsetnosy: + python-dev
messages: + msg283348
2016-12-15 03:39:20Eric Appeltsetfiles: + pep525_pep530_docs_v2.patch

messages: + msg283234
2016-12-14 01:05:32yselivanovsetmessages: + msg283151
2016-12-13 04:24:44Eric Appeltsetfiles: + pep525_pep530_docs_v1.patch

messages: + msg283072
2016-12-12 22:24:43Eric Appeltsetmessages: + msg283059
2016-12-12 15:54:18yselivanovsetmessages: + msg283016
2016-12-10 04:17:25ncoghlansetmessages: + msg282823
2016-12-09 23:01:32gvanrossumsetmessages: + msg282809
2016-12-09 22:04:12yselivanovsetmessages: + msg282808
2016-12-09 21:49:17gvanrossumsetmessages: + msg282807
2016-12-09 21:00:29yselivanovsetnosy: + gvanrossum, ncoghlan
messages: + msg282805
2016-12-09 20:30:30yselivanovsetmessages: + msg282802
2016-12-09 04:54:48Eric Appeltsetfiles: + pep525_pep530_docs_WIP.patch
keywords: + patch
messages: + msg282760
2016-12-07 04:11:22yselivanovsetmessages: + msg282594
2016-12-07 04:10:32Eric Appeltsetmessages: + msg282593
2016-12-07 03:54:21yselivanovsetmessages: + msg282590
2016-12-06 04:45:21yselivanovsetmessages: + msg282499
2016-12-06 03:26:25Eric Appeltsetnosy: + Eric Appelt
messages: + msg282494
2016-12-05 20:21:36ned.deilysetversions: + Python 3.7
2016-12-05 20:19:53ned.deilysetmessages: - msg282460
2016-12-05 20:19:46ned.deilysetmessages: + msg282461
2016-12-05 20:18:46ned.deilysetpriority: release blocker -> deferred blocker

messages: + msg282460
2016-09-12 01:47:13yselivanovcreate