diff -r 4b37328efa08 Doc/glossary.rst --- a/Doc/glossary.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/glossary.rst Thu Dec 08 23:42:53 2016 -0500 @@ -74,6 +74,29 @@ :keyword:`async with` statement by defining :meth:`__aenter__` and :meth:`__aexit__` methods. Introduced by :pep:`492`. + asynchronous generator + A function which returns an :term:`asynchronous generator iterator`. It + looks like a coroutine function defined with :keyword:`async def` except that it + contains :keyword:`yield` expressions for producing a series of values usable + in a :keyword:`async for` loop. + + Usually refers to a asynchronous generator function, but may refer to a + *asynchronous generator iterator* in some contexts. In cases where the intended + meaning isn't clear, using the full terms avoids ambiguity. + + asynchronous generator iterator + An object created by a :term:`asynchronous generator` function. + + This is an :term:`asynchronous iterator` which when called using the + :meth:`__anext__` method returns an awaitable object which will execute + that the body of the asynchronous generator function until the + next :keyword:`yield` expression. + Each :keyword:`yield` temporarily suspends processing, remembering the + location execution state (including local variables and pending + try-statements). When the *asynchronous generator iterator* effectively + resumes with another awaitable returned by :meth:`__anext__`, it picks-up where + it left-off. + asynchronous iterable An object, that can be used in an :keyword:`async for` statement. Must return an :term:`asynchronous iterator` from its diff -r 4b37328efa08 Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/library/asyncio-eventloop.rst Thu Dec 08 23:42:53 2016 -0500 @@ -88,6 +88,17 @@ This is idempotent and irreversible. No other methods should be called after this one. + +.. method:: AbstractEventLoop.shutdown_asyncgens() + + Schedule all currently open :term:`asynchronous generator` objects to close with + an aclose() call. After calling this method method, the event loop will + issue a warning whenever a new asynchronous generator is iterated for + the first time. + + .. versionadded:: 3.6 + + .. _asyncio-pass-keywords: Calls diff -r 4b37328efa08 Doc/library/inspect.rst --- a/Doc/library/inspect.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/library/inspect.rst Thu Dec 08 23:42:53 2016 -0500 @@ -318,6 +318,22 @@ .. versionadded:: 3.5 +.. function:: isasyncgenfunction(object) + + Return true if the object is an :term:`asynchronous generator` function + (a function defined with an :keyword:`async def` syntax that contains a + :keyword:`yield` expression). + + .. versionadded:: 3.6 + + +.. function:: isasyncgen(object) + + Return true if the object is a :term:`awaitable` created by an + :term:`asynchronous generator` function. + + .. versionadded:: 3.6 + .. function:: istraceback(object) Return true if the object is a traceback. diff -r 4b37328efa08 Doc/library/sys.rst --- a/Doc/library/sys.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/library/sys.rst Thu Dec 08 23:42:53 2016 -0500 @@ -603,6 +603,18 @@ .. versionchanged:: 3.6 Added *platform_version* + +.. function:: get_asyncgen_hooks() + + Returns an *asyncgen_hooks* object, which is a kind of 2-tuple containing + *firstiter* and *finalizer* functions, which take an + :term:`asynchronous generator iterator` as an argument, and are used to + schedule finalization of an asychronous generator by an event loop. + + .. versionadded:: 3.6 + See :pep:`525` for more details. + + .. function:: get_coroutine_wrapper() Returns ``None``, or a wrapper set by :func:`set_coroutine_wrapper`. @@ -1107,6 +1119,17 @@ implementation platform, rather than part of the language definition, and thus may not be available in all Python implementations. +.. function:: set_asyncgen_hooks(firstiter, finalizer) + + Accepts two arguments which are callables that accept an + :term:`asynchronous generator iterator` as an argument. The *firstiter* callable + will be called when an asynchronous generator is iterated for the first time. + The *finalizer* will be called when an asynchronous generator is about to be + garbage collected. + + .. versionadded:: 3.6 + See :pep:`525` for more details. + .. function:: set_coroutine_wrapper(wrapper) diff -r 4b37328efa08 Doc/library/types.rst --- a/Doc/library/types.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/library/types.rst Thu Dec 08 23:42:53 2016 -0500 @@ -104,6 +104,14 @@ .. versionadded:: 3.5 +.. data:: AsyncGeneratorType + + The type of :term:`asynchronous generator`-iterator objects, created by + asynchronous generator functions. + + .. versionadded:: 3.6 + + .. data:: CodeType .. index:: builtin: compile diff -r 4b37328efa08 Doc/reference/compound_stmts.rst --- a/Doc/reference/compound_stmts.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/reference/compound_stmts.rst Thu Dec 08 23:42:53 2016 -0500 @@ -697,7 +697,7 @@ Functions defined with ``async def`` syntax are always coroutine functions, even if they do not contain ``await`` or ``async`` keywords. -It is a :exc:`SyntaxError` to use :keyword:`yield` expressions in +It is a :exc:`SyntaxError` to use ``yield from`` expressions in ``async def`` coroutines. An example of a coroutine function:: diff -r 4b37328efa08 Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/reference/datamodel.rst Thu Dec 08 23:42:53 2016 -0500 @@ -627,6 +627,24 @@ as well as :keyword:`async with` and :keyword:`async for` statements. See also the :ref:`coroutine-objects` section. + Asynchronous generator functions + .. index:: + single: asynchronous generator; function + single: asynchronous generator; asynchronous iterator + + A function or method which is defined using :keyword:`async def` and which + uses the :keyword:`yield` statement is called a + :dfn:`asynchronous generator function`. Such a function, when called, returns + an asynchronous iterator object which can be used in an :keyword:`async for` + statement to execute the body of the function. Calling the asynchronous + iterator's :meth:`aiterator.__anext__` method will return an + :term:`awaitable` which when awaited + will execute until it provides a value using the :keyword:`yield` + statement. When the function executes an empty :keyword:`return` statement + or falls off the end, a :exc:`StopAsyncIteration` exception is raised + and the asynchronous iterator will have reached the end of the set of + values to be returned. + Built-in functions .. index:: object: built-in function diff -r 4b37328efa08 Doc/reference/expressions.rst --- a/Doc/reference/expressions.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/reference/expressions.rst Thu Dec 08 23:42:53 2016 -0500 @@ -172,7 +172,7 @@ .. productionlist:: comprehension: `expression` `comp_for` - comp_for: "for" `target_list` "in" `or_test` [`comp_iter`] + comp_for: [ASYNC] "for" `target_list` "in" `or_test` [`comp_iter`] comp_iter: `comp_for` | `comp_if` comp_if: "if" `expression_nocond` [`comp_iter`] @@ -186,6 +186,12 @@ Note that the comprehension is executed in a separate scope, so names assigned to in the target list don't "leak" into the enclosing scope. +If the comprehension appears in an :keyword:`async def` function, any :keyword:`for` +clause may be replaced with an :keyword:`async for` clause. Additionally, any +expression in the display may be an :keyword:`await` expression. If a display +contains either :keyword:`async for` clauses or :keyword:`await` expressions it is +called an :dfn:`asynchronous comprehension`. An asynchronous comprehension may +suspend the execution of the coroutine function in which it appears. .. _lists: @@ -315,6 +321,13 @@ The parentheses can be omitted on calls with only one argument. See section :ref:`calls` for details. +If the generator appears in an :keyword:`async def` function, any :keyword:`for` +clause may be replaced with an :keyword:`async for` clause. Additionally, any +expression in the display may be an :keyword:`await` expression. If a generator +expression contains either :keyword:`async for` clauses or :keyword:`await` +expressions it is called an :dfn:`asynchronous generator expression`. +An asynchronous generator expression yields a new asynchronous generator object, +which is an asynchronous iterator (see :ref:`async-iterators`). .. _yieldexpr: @@ -330,9 +343,12 @@ yield_atom: "(" `yield_expression` ")" yield_expression: "yield" [`expression_list` | "from" `expression`] -The yield expression is only used when defining a :term:`generator` function and +The yield expression is used when defining a :term:`generator` function and thus can only be used in the body of a function definition. Using a yield -expression in a function's body causes that function to be a generator. +expression in a function's body causes that function to be a generator. The only +other valid use of the yield expression is in defining a +:term:`asynchronous generator` function, which is described separately in +section :ref:`asynchronous-generator-functions`. When a generator function is called, it returns an iterator known as a generator. That generator then controls the execution of the generator function. @@ -496,6 +512,131 @@ For examples using ``yield from``, see :ref:`pep-380` in "What's New in Python." +.. _asynchronous-generator-functions: + +Asynchronous generator functions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The presence of a yield expression in a function or method defined using +:keyword:`async def` further defines the function as a +:term:`asynchronous generator` function. + +When an asynchronous generator function is called, it returns an +asynchronous iterator known as an asynchronous generator object. +That object then controls the execution of the generator function. +An asynchronous generator object is typically used in an +:keyword:`async for` statement in a coroutine function analogously to +how a generator object would be used in a :keyword:`for` statement. + +Calling one of the asynchronous generator's methods returns an +:term:`awaitable` object, and the execution starts when this object +is awaited on. At that time, the execution proceeds to the first yield +expression, where it is suspended again, returning the value of +:token:`expression_list` to the awaiting coroutine. As with a generator, +suspension means that all local state is retained, including the +current bindings of local variables, the instruction pointer, the internal +evaluation stack, and the state of any exception handling. When the execution +is resumed by awaiting on the next object returned by the asynchronous +generator's methods, the function can proceed exactly as if the yield expression +were just another external call. The value of the yield expression after +resuming depends on the method which resumed the execution. If +:meth:`~agen.__anext__` is used then the result is :const:`None`. Otherwise, if +:meth:`~agen.asend` is used, then the result will be the value passed in to +that method. + +In an asynchronous generator function, yield expressions are allowed anywhere +in a :keyword:`try` construct, except for a :keyword:`finally` clause. + +If an asynchronous generator is not resumed before it is +finalized (by reaching a zero reference count or by being garbage collected), +it is the responsibility of the event loop or scheduler running the +asynchronous generator to call the asynchronous +generator-iterator's :meth:`~agen.aclose` method and run the resulting +coroutine object, thus allowing any pending :keyword:`finally` clauses to execute. +To take care of finalization, an event loop should define a *finalizer* function +which takes an asynchronous generator-iterator and presumably calls +:meth:`~agen.aclose` and executes the coroutine. This *finalizer* may +be registered by calling :func:`sys.set_asyncgen_hooks`. When first iterated over, +an asynchronous generator-iterator will store the registered *finalizer* +to be called upon finalization. + +The expression ``yield from `` is a syntax error when used in an +asynchronous generator function. + +.. index:: object: asynchronous-generator +.. _asynchronous-generator-methods: + +Asynchronous generator-iterator methods +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This subsection describes the methods of an asynchronous generator iterator, +which are used to control the execution of a generator function. + + +Note that unlike a generator-iterator, calling any of the asynchronous generator +methods below when will not raise an exception. + + +.. index:: exception: StopAsyncIteration + +.. method:: agen.__anext__() + + Returns an awaitable which when run starts to execute the asynchronous generator + or resumes it at the last executed yield expression. When an asynchronous + generator function is resumed with a :meth:`~agen.__anext__` method, the + current yield expression always evaluates to :const:`None` in the returned + awaitable, which when run will continue to the next yield expression. The + value of the :token:`expression_list` of the yield expression is the value + of the :exc:`StopIteration` exception raised by the completing coroutine. + If the asynchronous generator exits without yielding another value, the + awaitable instead raises an :exc:`StopAsyncIteration` exception, signalling + that the asynchronous iteration has completed. + + This method is normally called implicitly by a :keyword:`async for` loop. + + +.. method:: agen.asend(value) + + Returns an awaitable which when run resumes the execution of the asynchronous + generator. As with the :meth:`~generator.send()` method for a generator, + this "sends" a value into the asynchronous generator function, and the + *value* argument becomes the result of the current yield expression. The + awaitable returned by the :meth:`asend` method will return the next value yielded + by the generator as the value of the raised :exc:`StopIteration`, or + raises :exc:`StopAsyncIteration` if the asynchronous generator exits without + yielding another value. When :meth:`asend` is called to start the asynchronous + generator, it must be called with :const:`None` as the argument, + because there is no yield expression that could receive the value. + + +.. method:: agen.athrow(type[, value[, traceback]]) + + Returns an awaitable that raises an exception of type ``type`` at the point + where the asynchronous generator was paused, and returns the next value yielded + by the generator function as the value of the raised :exc:`StopIteration` + exception. If the asynchronous generator exits without yielding another value, + an :exc:`StopAsyncIteration` exception is raised by the awaitable. + If the generator function does not catch the passed-in exception, or + raises a different exception, then when the awaitalbe is run that exception + propagates to the caller of the awaitable. + +.. index:: exception: GeneratorExit + + +.. method:: agen.aclose() + + Returns an awaitable that when run will throw a :exc:`GeneratorExit` into the + asynchronous generator function at the point where it was paused. + If the asynchronous generator function then exits gracefully, is already closed, + or raises :exc:`GeneratorExit` (by not catching the exception), then the returned + awaitable will raise a :exc:`StopIteration` exception. Any further awaitables + returned by subsequent calls to the asynchronous generator will raise a + :exc:`StopAsyncIteration` exception. If the asynchronous generator yields a + value, a :exc:`RuntimeError` is raised by the awaitable. If the asynchronous + generator raises any other exception, it is propagated to the caller of + the awaitable. If the asynchronous generator has already exited due to an + exception or normal exit, then further calls to :meth:`aclose` will return an + awaitable that does nothing. .. _primaries: diff -r 4b37328efa08 Doc/reference/simple_stmts.rst --- a/Doc/reference/simple_stmts.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/reference/simple_stmts.rst Thu Dec 08 23:42:53 2016 -0500 @@ -492,6 +492,10 @@ value (if any) is used as an argument to construct :exc:`StopIteration` and becomes the :attr:`StopIteration.value` attribute. +In an asynchronous generator function, an empty :keyword:`return` statement indicates +that the asynchronous generator is done and will cause :exc:`StopAsyncIteration` +to be raised. A non-empty :keyword:`return` statement is a syntax error in an +asynchronous generator function. .. _yield: