diff -r 4b37328efa08 Doc/glossary.rst --- a/Doc/glossary.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/glossary.rst Mon Dec 12 23:15:54 2016 -0500 @@ -74,6 +74,29 @@ :keyword:`async with` statement by defining :meth:`__aenter__` and :meth:`__aexit__` methods. Introduced by :pep:`492`. + asynchronous generator + A function which returns an :term:`asynchronous generator iterator`. It + looks like a coroutine function defined with :keyword:`async def` except + that it contains :keyword:`yield` expressions for producing a series of + values usable in an :keyword:`async for` loop. + + Usually refers to a asynchronous generator function, but may refer to an + *asynchronous generator iterator* in some contexts. In cases where the + intended meaning isn't clear, using the full terms avoids ambiguity. + + asynchronous generator iterator + An object created by a :term:`asynchronous generator` function. + + This is an :term:`asynchronous iterator` which when called using the + :meth:`__anext__` method returns an awaitable object which will execute + that the body of the asynchronous generator function until the + next :keyword:`yield` expression. + Each :keyword:`yield` temporarily suspends processing, remembering the + location execution state (including local variables and pending + try-statements). When the *asynchronous generator iterator* effectively + resumes with another awaitable returned by :meth:`__anext__`, it + picks-up where it left-off. See :pep:`492` and :pep:`525`. + asynchronous iterable An object, that can be used in an :keyword:`async for` statement. Must return an :term:`asynchronous iterator` from its diff -r 4b37328efa08 Doc/library/asyncio-eventloop.rst --- a/Doc/library/asyncio-eventloop.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/library/asyncio-eventloop.rst Mon Dec 12 23:15:54 2016 -0500 @@ -88,6 +88,24 @@ This is idempotent and irreversible. No other methods should be called after this one. + +.. coroutinemethod:: AbstractEventLoop.shutdown_asyncgens() + + Schedule all currently open :term:`asynchronous generator` objects to + close with an :meth:`~agen.aclose()` call. After calling this method, + the event loop will issue a warning whenever a new asynchronous generator + is iterated. May be used to finalize all scheduled asynchronous generators + reliably, for example:: + + try: + loop.run_forever() + finally: + loop.run_until_complete(loop.shutdown_asyncgens()) + loop.close() + + .. versionadded:: 3.6 + + .. _asyncio-pass-keywords: Calls diff -r 4b37328efa08 Doc/library/inspect.rst --- a/Doc/library/inspect.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/library/inspect.rst Mon Dec 12 23:15:54 2016 -0500 @@ -318,6 +318,27 @@ .. versionadded:: 3.5 +.. function:: isasyncgenfunction(object) + + Return true if the object is an :term:`asynchronous generator` function, + for example:: + + >>> async def agen(): + ... yield 1 + ... + >>> inspect.isasyncgenfunction(agen) + True + + .. versionadded:: 3.6 + + +.. function:: isasyncgen(object) + + Return true if the object is an :term:`asynchronous generator iterator` + created by an :term:`asynchronous generator` function. + + .. versionadded:: 3.6 + .. function:: istraceback(object) Return true if the object is a traceback. diff -r 4b37328efa08 Doc/library/sys.rst --- a/Doc/library/sys.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/library/sys.rst Mon Dec 12 23:15:54 2016 -0500 @@ -603,6 +603,20 @@ .. versionchanged:: 3.6 Added *platform_version* + +.. function:: get_asyncgen_hooks() + + Returns an *asyncgen_hooks* object, which is similar to a + :class:`~collections.namedtuple` of the form `(firstiter, finalizer)`, + where *firstiter* and *finalizer* are expected to be either ``None`` or + functions which take an :term:`asynchronous generator iterator` as an + argument, and are used to schedule finalization of an asychronous + generator by an event loop. + + .. versionadded:: 3.6 + See :pep:`525` for more details. + + .. function:: get_coroutine_wrapper() Returns ``None``, or a wrapper set by :func:`set_coroutine_wrapper`. @@ -1107,6 +1121,19 @@ implementation platform, rather than part of the language definition, and thus may not be available in all Python implementations. +.. function:: set_asyncgen_hooks(firstiter, finalizer) + + Accepts two optional keyword arguments which are callables that accept an + :term:`asynchronous generator iterator` as an argument. The *firstiter* + callable will be called when an asynchronous generator is iterated for the + first time. The *finalizer* will be called when an asynchronous generator + is about to be garbage collected. + + .. versionadded:: 3.6 + See :pep:`525` for more details, and for a reference example of a + *finalizer* method see the implementation of + ``asyncio.Loop.shutdown_asyncgens`` in + :source:`Lib/asyncio/base_events.py` .. function:: set_coroutine_wrapper(wrapper) diff -r 4b37328efa08 Doc/library/types.rst --- a/Doc/library/types.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/library/types.rst Mon Dec 12 23:15:54 2016 -0500 @@ -104,6 +104,14 @@ .. versionadded:: 3.5 +.. data:: AsyncGeneratorType + + The type of :term:`asynchronous generator`-iterator objects, created by + asynchronous generator functions. + + .. versionadded:: 3.6 + + .. data:: CodeType .. index:: builtin: compile diff -r 4b37328efa08 Doc/reference/compound_stmts.rst --- a/Doc/reference/compound_stmts.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/reference/compound_stmts.rst Mon Dec 12 23:15:54 2016 -0500 @@ -697,7 +697,7 @@ Functions defined with ``async def`` syntax are always coroutine functions, even if they do not contain ``await`` or ``async`` keywords. -It is a :exc:`SyntaxError` to use :keyword:`yield` expressions in +It is a :exc:`SyntaxError` to use ``yield from`` expressions in ``async def`` coroutines. An example of a coroutine function:: diff -r 4b37328efa08 Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/reference/datamodel.rst Mon Dec 12 23:15:54 2016 -0500 @@ -627,6 +627,24 @@ as well as :keyword:`async with` and :keyword:`async for` statements. See also the :ref:`coroutine-objects` section. + Asynchronous generator functions + .. index:: + single: asynchronous generator; function + single: asynchronous generator; asynchronous iterator + + A function or method which is defined using :keyword:`async def` and + which uses the :keyword:`yield` statement is called a + :dfn:`asynchronous generator function`. Such a function, when called, + returns an asynchronous iterator object which can be used in an + :keyword:`async for` statement to execute the body of the function. + Calling the asynchronous iterator's :meth:`aiterator.__anext__` method + will return an :term:`awaitable` which when awaited + will execute until it provides a value using the :keyword:`yield` + statement. When the function executes an empty :keyword:`return` + statement or falls off the end, a :exc:`StopAsyncIteration` exception + is raised and the asynchronous iterator will have reached the end of + the set of values to be returned. + Built-in functions .. index:: object: built-in function diff -r 4b37328efa08 Doc/reference/expressions.rst --- a/Doc/reference/expressions.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/reference/expressions.rst Mon Dec 12 23:15:54 2016 -0500 @@ -172,7 +172,7 @@ .. productionlist:: comprehension: `expression` `comp_for` - comp_for: "for" `target_list` "in" `or_test` [`comp_iter`] + comp_for: [ASYNC] "for" `target_list` "in" `or_test` [`comp_iter`] comp_iter: `comp_for` | `comp_if` comp_if: "if" `expression_nocond` [`comp_iter`] @@ -186,6 +186,13 @@ Note that the comprehension is executed in a separate scope, so names assigned to in the target list don't "leak" into the enclosing scope. +If the comprehension appears in an :keyword:`async def` function, any +:keyword:`for` clause may be replaced with an :keyword:`async for` clause. +Additionally, any expression in the display may be an :keyword:`await` +expression. If a display contains either :keyword:`async for` clauses +or :keyword:`await` expressions it is called an +:dfn:`asynchronous comprehension`. An asynchronous comprehension may +suspend the execution of the coroutine function in which it appears. .. _lists: @@ -315,6 +322,14 @@ The parentheses can be omitted on calls with only one argument. See section :ref:`calls` for details. +If the generator appears in an :keyword:`async def` function, +any :keyword:`for` clause may be replaced with an :keyword:`async for` clause. +Additionally, any expression in the display may be an :keyword:`await` +expression. If a generator expression contains either :keyword:`async for` +clauses or :keyword:`await` expressions it is called an +:dfn:`asynchronous generator expression`. An asynchronous generator +expression yields a new asynchronous generator object, +which is an asynchronous iterator (see :ref:`async-iterators`). .. _yieldexpr: @@ -330,9 +345,12 @@ yield_atom: "(" `yield_expression` ")" yield_expression: "yield" [`expression_list` | "from" `expression`] -The yield expression is only used when defining a :term:`generator` function and +The yield expression is used when defining a :term:`generator` function and thus can only be used in the body of a function definition. Using a yield expression in a function's body causes that function to be a generator. +The only other valid use of the yield expression is in defining a +:term:`asynchronous generator` function, which is described separately in +section :ref:`asynchronous-generator-functions`. When a generator function is called, it returns an iterator known as a generator. That generator then controls the execution of the generator function. @@ -496,6 +514,134 @@ For examples using ``yield from``, see :ref:`pep-380` in "What's New in Python." +.. _asynchronous-generator-functions: + +Asynchronous generator functions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The presence of a yield expression in a function or method defined using +:keyword:`async def` further defines the function as a +:term:`asynchronous generator` function. + +When an asynchronous generator function is called, it returns an +asynchronous iterator known as an asynchronous generator object. +That object then controls the execution of the generator function. +An asynchronous generator object is typically used in an +:keyword:`async for` statement in a coroutine function analogously to +how a generator object would be used in a :keyword:`for` statement. + +Calling one of the asynchronous generator's methods returns an +:term:`awaitable` object, and the execution starts when this object +is awaited on. At that time, the execution proceeds to the first yield +expression, where it is suspended again, returning the value of +:token:`expression_list` to the awaiting coroutine. As with a generator, +suspension means that all local state is retained, including the +current bindings of local variables, the instruction pointer, the internal +evaluation stack, and the state of any exception handling. When the execution +is resumed by awaiting on the next object returned by the asynchronous +generator's methods, the function can proceed exactly as if the yield +expression were just another external call. The value of the yield expression +after resuming depends on the method which resumed the execution. If +:meth:`~agen.__anext__` is used then the result is :const:`None`. Otherwise, if +:meth:`~agen.asend` is used, then the result will be the value passed in to +that method. + +In an asynchronous generator function, yield expressions are allowed anywhere +in a :keyword:`try` construct, except for a :keyword:`finally` clause. + +If an asynchronous generator is not resumed before it is +finalized (by reaching a zero reference count or by being garbage collected), +it is the responsibility of the event loop or scheduler running the +asynchronous generator to call the asynchronous +generator-iterator's :meth:`~agen.aclose` method and run the resulting +coroutine object, thus allowing any pending :keyword:`finally` clauses +to execute. To take care of finalization, an event loop should define +a *finalizer* function which takes an asynchronous generator-iterator +and presumably calls :meth:`~agen.aclose` and executes the coroutine. +This *finalizer* may be registered by calling :func:`sys.set_asyncgen_hooks`. +When first iterated over, an asynchronous generator-iterator will store the +registered *finalizer* to be called upon finalization. + +The expression ``yield from `` is a syntax error when used in an +asynchronous generator function. + +.. index:: object: asynchronous-generator +.. _asynchronous-generator-methods: + +Asynchronous generator-iterator methods +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This subsection describes the methods of an asynchronous generator iterator, +which are used to control the execution of a generator function. + + +Note that unlike a generator-iterator, calling any of the asynchronous +generator methods below when will not raise an exception. + + +.. index:: exception: StopAsyncIteration + +.. coroutinemethod:: agen.__anext__() + + Returns an awaitable which when run starts to execute the asynchronous + generator or resumes it at the last executed yield expression. When an + asynchronous generator function is resumed with a :meth:`~agen.__anext__` + method, the current yield expression always evaluates to :const:`None` in + the returned awaitable, which when run will continue to the next yield + expression. The value of the :token:`expression_list` of the yield + expression is the value of the :exc:`StopIteration` exception raised by + the completing coroutine. If the asynchronous generator exits without + yielding another value, the awaitable instead raises an + :exc:`StopAsyncIteration` exception, signalling that the asynchronous + iteration has completed. + + This method is normally called implicitly by a :keyword:`async for` loop. + + +.. coroutinemethod:: agen.asend(value) + + Returns an awaitable which when run resumes the execution of the + asynchronous generator. As with the :meth:`~generator.send()` method for a + generator, this "sends" a value into the asynchronous generator function, + and the *value* argument becomes the result of the current yield expression. + The awaitable returned by the :meth:`asend` method will return the next + value yielded by the generator as the value of the raised + :exc:`StopIteration`, or raises :exc:`StopAsyncIteration` if the + asynchronous generator exits without yielding another value. When + :meth:`asend` is called to start the asynchronous + generator, it must be called with :const:`None` as the argument, + because there is no yield expression that could receive the value. + + +.. coroutinemethod:: agen.athrow(type[, value[, traceback]]) + + Returns an awaitable that raises an exception of type ``type`` at the point + where the asynchronous generator was paused, and returns the next value + yielded by the generator function as the value of the raised + :exc:`StopIteration` exception. If the asynchronous generator exits + without yielding another value, an :exc:`StopAsyncIteration` exception is + raised by the awaitable. + If the generator function does not catch the passed-in exception, or + raises a different exception, then when the awaitalbe is run that exception + propagates to the caller of the awaitable. + +.. index:: exception: GeneratorExit + + +.. coroutinemethod:: agen.aclose() + + Returns an awaitable that when run will throw a :exc:`GeneratorExit` into + the asynchronous generator function at the point where it was paused. + If the asynchronous generator function then exits gracefully, is already + closed, or raises :exc:`GeneratorExit` (by not catching the exception), + then the returned awaitable will raise a :exc:`StopIteration` exception. + Any further awaitables returned by subsequent calls to the asynchronous + generator will raise a :exc:`StopAsyncIteration` exception. If the + asynchronous generator yields a value, a :exc:`RuntimeError` is raised + by the awaitable. If the asynchronous generator raises any other exception, + it is propagated to the caller of the awaitable. If the asynchronous + generator has already exited due to an exception or normal exit, then + further calls to :meth:`aclose` will return an awaitable that does nothing. .. _primaries: diff -r 4b37328efa08 Doc/reference/simple_stmts.rst --- a/Doc/reference/simple_stmts.rst Wed Dec 07 00:37:38 2016 +0100 +++ b/Doc/reference/simple_stmts.rst Mon Dec 12 23:15:54 2016 -0500 @@ -492,6 +492,10 @@ value (if any) is used as an argument to construct :exc:`StopIteration` and becomes the :attr:`StopIteration.value` attribute. +In an asynchronous generator function, an empty :keyword:`return` statement +indicates that the asynchronous generator is done and will cause +:exc:`StopAsyncIteration` to be raised. A non-empty :keyword:`return` +statement is a syntax error in an asynchronous generator function. .. _yield: