From da6fd0e1a062f7bcf758dbb4e0a47635ea83769b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Zbigniew=20J=C4=99drzejewski-Szmek?= Date: Tue, 20 Sep 2011 19:58:22 +0200 Subject: [PATCH 1/2] Document the 'yield from' syntax and StopIteration return enhancements Previously, 'yield' was documented in two places: simple_smtms.rst and expressions.rst. The _statement_ is the more important of the two meanings, so most of the documentation is moved there. Only the things that describe yield as an expression are kept in expressions.rst. --- Doc/library/exceptions.rst | 11 ++- Doc/reference/expressions.rst | 158 +++++++-------------------------- Doc/reference/simple_stmts.rst | 192 ++++++++++++++++++++++++++++++++++------ 3 files changed, 206 insertions(+), 155 deletions(-) diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 528febd..5a17c19 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -262,8 +262,15 @@ The following exceptions are the exceptions that are usually raised. .. exception:: StopIteration Raised by built-in function :func:`next` and an :term:`iterator`\'s - :meth:`__next__` method to signal that there are no further values. - + :obj:`~iterator.__next__` method to signal that there are no + further items in the sequence. + + The exception object has a single attribute :attr:`value`, which is + given as an argument when constructing the exception, and defaults + to :const:`None`. When a generator function returns, a + :exc:`StopIteration` is raised, and the value returned by the + function is used as the :attr:`value` parameter to the constructor + of the exception. .. exception:: SyntaxError diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index 655ebde..4572e50 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -56,7 +56,7 @@ also categorized syntactically as atoms. The syntax for atoms is: .. productionlist:: atom: `identifier` | `literal` | `enclosure` enclosure: `parenth_form` | `list_display` | `dict_display` | `set_display` - : | `generator_expression` | `yield_atom` + : | `generator_expression` | `yield_expression` .. _atom-identifiers: @@ -317,133 +317,41 @@ Yield expressions pair: generator; function .. productionlist:: - yield_atom: "(" `yield_expression` ")" - yield_expression: "yield" [`expression_list`] - -The :keyword:`yield` expression is only used when defining a generator function, -and can only be used in the body of a function definition. Using a -:keyword:`yield` expression in a function definition is sufficient to cause that -definition to create a generator function instead of a normal function. - -When a generator function is called, it returns an iterator known as a -generator. That generator then controls the execution of a generator function. -The execution starts when one of the generator's methods is called. At that -time, the execution proceeds to the first :keyword:`yield` expression, where it -is suspended again, returning the value of :token:`expression_list` to -generator's caller. By suspended we mean that all local state is retained, -including the current bindings of local variables, the instruction pointer, and -the internal evaluation stack. When the execution is resumed by calling one of -the generator's methods, the function can proceed exactly as if the -:keyword:`yield` expression was just another external call. The value of the -:keyword:`yield` expression after resuming depends on the method which resumed -the execution. - -.. index:: single: coroutine - -All of this makes generator functions quite similar to coroutines; they yield -multiple times, they have more than one entry point and their execution can be -suspended. The only difference is that a generator function cannot control -where should the execution continue after it yields; the control is always -transferred to the generator's caller. - -The :keyword:`yield` statement is allowed in the :keyword:`try` clause of a -:keyword:`try` ... :keyword:`finally` construct. If the generator is not -resumed before it is finalized (by reaching a zero reference count or by being -garbage collected), the generator-iterator's :meth:`close` method will be -called, allowing any pending :keyword:`finally` clauses to execute. - -.. index:: object: generator - -The following generator's methods can be used to control the execution of a -generator function: - -.. index:: exception: StopIteration - - -.. method:: generator.__next__() - - Starts the execution of a generator function or resumes it at the last - executed :keyword:`yield` expression. When a generator function is resumed - with a :meth:`__next__` method, the current :keyword:`yield` expression - always evaluates to :const:`None`. The execution then continues to the next - :keyword:`yield` expression, where the generator is suspended again, and the - value of the :token:`expression_list` is returned to :meth:`next`'s caller. - If the generator exits without yielding another value, a :exc:`StopIteration` - exception is raised. - - This method is normally called implicitly, e.g. by a :keyword:`for` loop, or - by the built-in :func:`next` function. - - -.. method:: generator.send(value) - - Resumes the execution and "sends" a value into the generator function. The - ``value`` argument becomes the result of the current :keyword:`yield` - expression. The :meth:`send` method returns the next value yielded by the - generator, or raises :exc:`StopIteration` if the generator exits without - yielding another value. When :meth:`send` is called to start the generator, - it must be called with :const:`None` as the argument, because there is no - :keyword:`yield` expression that could receive the value. - - -.. method:: generator.throw(type[, value[, traceback]]) - - Raises an exception of type ``type`` at the point where generator was paused, - and returns the next value yielded by the generator function. If the generator - exits without yielding another value, a :exc:`StopIteration` exception is - raised. If the generator function does not catch the passed-in exception, or - raises a different exception, then that exception propagates to the caller. - -.. index:: exception: GeneratorExit - - -.. method:: generator.close() - - Raises a :exc:`GeneratorExit` at the point where the generator function was - paused. If the generator function then raises :exc:`StopIteration` (by - exiting normally, or due to already being closed) or :exc:`GeneratorExit` (by - not catching the exception), close returns to its caller. If the generator - yields a value, a :exc:`RuntimeError` is raised. If the generator raises any - other exception, it is propagated to the caller. :meth:`close` does nothing - if the generator has already exited due to an exception or normal exit. - -Here is a simple example that demonstrates the behavior of generators and -generator functions:: - - >>> def echo(value=None): - ... print("Execution starts when 'next()' is called for the first time.") - ... try: - ... while True: - ... try: - ... value = (yield value) - ... except Exception as e: - ... value = e - ... finally: - ... print("Don't forget to clean up when 'close()' is called.") - ... - >>> generator = echo(1) - >>> print(next(generator)) - Execution starts when 'next()' is called for the first time. - 1 - >>> print(next(generator)) - None - >>> print(generator.send(2)) - 2 - >>> generator.throw(TypeError, "spam") - TypeError('spam',) - >>> generator.close() - Don't forget to clean up when 'close()' is called. - + yield_expression: "(" `yield_stmt` ")" + +The :keyword:`yield` statement appears only in generator functions and is used +to pause the execution of a function and pass a value to the caller. The +behaviour of :keyword:`yield` depends on the generator method that was called +and is described in detail `elsewhere `_. Here, only the behaviour of +:keyword:`yield` as an expression is documented. + +When :token:`yield from ` is used, the :attr:`~StopIteration.value` +attribute of :exc:`StopIteration` used to signal the exhaustion of the +sub-generator becomes the value of the yield expression. It can be either set +explicitely when raising :exc:`StopIteration`, or automatically when the +sub-generator is a generator, by returning a value from the sub-generator +function. + +When a :token:`normal yield ` is used, the value of the +:token:`yield_expression` depends on the generator method that was called. When +the generator's :obj:`~generator.__next__` method is used, the value of the +yield expression is :const:`None`, this is the most common case. When the +generator's :obj:`send(value) ` method is used, the ``value`` +passed as an argument becomes the value of the :token:`yield_expression`. + +The other two generator methods cause an exception to be raised at the point of +the yield statement. :meth:`~generator.throw` takes exception details as +arguments and is used to raise an arbitrary expression at the point where the +generator function is stopped after :keyword:`yield`. :meth:`~generator.close` +is used to finalize generators and causes a :exc:`GeneratorExit` exception to be +raised at the point of the :keyword:`yield` statement. + +The parentheses can be ommited when the :token:`yield statement ` +occurs in a top-level expression on the right-hand side of an assignment. .. seealso:: - :pep:`0255` - Simple Generators - The proposal for adding generators and the :keyword:`yield` statement to Python. - - :pep:`0342` - Coroutines via Enhanced Generators - The proposal to enhance the API and syntax of generators, making them - usable as simple coroutines. - + The documentation for the :keyword:`yield` statement. .. _primaries: diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index a375117..17da75b 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -425,10 +425,10 @@ When :keyword:`return` passes control out of a :keyword:`try` statement with a :keyword:`finally` clause, that :keyword:`finally` clause is executed before really leaving the function. -In a generator function, the :keyword:`return` statement is not allowed to -include an :token:`expression_list`. In that context, a bare :keyword:`return` -indicates that the generator is done and will cause :exc:`StopIteration` to be -raised. +In a generator function, the :keyword:`return` statement indicates that the +generator is done and will cause :exc:`StopIteration` to be raised. The returned +value is used as an argument to construct :exc:`StopIteration` and becomes the +:attr:`StopIteration.value` attribute. .. _yield: @@ -444,30 +444,163 @@ The :keyword:`yield` statement exception: StopIteration .. productionlist:: - yield_stmt: `yield_expression` + yield_stmt: `yield_arg` | `yield_from` + yield_arg: "yield" [`expression_list`] + yield_from: "yield" "from" [`expression_list`] + +The :keyword:`yield` expression is only used when defining a generator function, +and can only be used in the body of a function definition. Using a +:keyword:`yield` expression in a function definition is sufficient to cause that +definition to create a generator function instead of a normal function. -The :keyword:`yield` statement is only used when defining a generator function, -and is only used in the body of the generator function. Using a :keyword:`yield` -statement in a function definition is sufficient to cause that definition to -create a generator function instead of a normal function. When a generator function is called, it returns an iterator known as a generator -iterator, or more commonly, a generator. The body of the generator function is -executed by calling the :func:`next` function on the generator repeatedly until -it raises an exception. - -When a :keyword:`yield` statement is executed, the state of the generator is -frozen and the value of :token:`expression_list` is returned to :meth:`next`'s -caller. By "frozen" we mean that all local state is retained, including the -current bindings of local variables, the instruction pointer, and the internal -evaluation stack: enough information is saved so that the next time :func:`next` -is invoked, the function can proceed exactly as if the :keyword:`yield` -statement were just another external call. - -The :keyword:`yield` statement is allowed in the :keyword:`try` clause of a -:keyword:`try` ... :keyword:`finally` construct. If the generator is not -resumed before it is finalized (by reaching a zero reference count or by being -garbage collected), the generator-iterator's :meth:`close` method will be -called, allowing any pending :keyword:`finally` clauses to execute. +iterator, or more commonly, a generator. That generator then controls the +execution of the generator function. The execution starts when one of the +generator's methods is called. At that time, the execution proceeds to the first +:keyword:`yield` expression. What happens next depends on whether the simpler +:token:`yield_arg` or the more complicated :token:`yield_from` form was used. + +In the first case, when ``from`` is not used, the execution of the generator +function is suspended again, returning the value of :token:`expression_list` to +generator's caller. By suspended we mean that all local state is retained, +including the current bindings of local variables, the instruction pointer, and +the internal evaluation stack. When the execution is resumed by calling one of +the generator's methods, the function can proceed exactly as if the +:keyword:`yield` expression was just another external call. The value of the +:keyword:`yield` expression after resuming depends on the method which resumed +the execution. + +In the second case, when ``from`` is used, the :token:`expression_list` is +expected to yield an iterator, a "sub-generator". The evaluation of the +generator is then delegated to this sub-generator. This means that the generator +function is suspended like in the first case, but calls to +:meth:`~generator.__next__`, :meth:`~generator.send`, :meth:`~generator.throw`, +and :meth:`~generator.close` are propagated to the sub-generator until it is +exhausted (and raises :exc:`StopIteration`) or until it terminates through an +exception (which is then re-raised at the point of the original +:token:`yield_from`). This includes the call which causes the original generator +function to reach the :token:`yield_from` statement. If :token:`expression_list` +yields something that is not an iterator, a :exc:`TypeError` is raised. The +behaviour of a :token:`yield_from` expression is similar to repeatedly calling +``return yield`` in a loop, but in contrast also propagates the exceptions and +values injected with :meth:`~generator.throw` and :meth:`~generator.send`. + +.. index:: single: coroutine + +All of this makes generator functions quite similar to coroutines; they yield +multiple times, they have more than one entry point and their execution can be +suspended. The only difference is that a generator function cannot control +where should the execution continue after it yields; the control is always +transferred to the generator's caller. + +The :keyword:`yield` statement is allowed anywhere in the body of the generator +function, also in the :keyword:`try` clause of a :keyword:`try` ... +:keyword:`finally` construct. The :keyword:`finally` clause is still guaranteed +to be executed before the interpreter terminates. Before the generator is +destroyed (after reaching a zero reference count or being garbage collected), +the generator-iterator's :meth:`~generator.close` method is called, allowing any +pending :keyword:`finally` clauses to execute. + +.. index:: object: generator + +The following generator's methods can be used to control the execution of a +generator function: + +.. index:: exception: StopIteration + + +.. method:: generator.__next__() + + Starts the execution of a generator function or resumes it at the last + executed :keyword:`yield` expression. When a generator function is resumed + with a :meth:`__next__` method, the current :keyword:`yield` expression + always evaluates to :const:`None`. The execution then continues to the next + :keyword:`yield` expression, where the generator is suspended again, and the + value of the :token:`expression_list` is returned to :meth:`next`'s caller. + If the generator exits without yielding another value, a :exc:`StopIteration` + exception is raised. + + This method is normally called implicitly, e.g. by a :keyword:`for` loop, or + by the built-in :func:`next` function. + + +.. method:: generator.send(value) + + Resumes the execution and "sends" a value into the generator function. The + ``value`` argument becomes the result of the current :keyword:`yield` + expression. The :meth:`send` method returns the next value yielded by the + generator, or raises :exc:`StopIteration` if the generator exits without + yielding another value. When :meth:`send` is called to start the generator, + it must be called with :const:`None` as the argument, because there is no + :keyword:`yield` expression that could receive the value. + + +.. method:: generator.throw(type[, value[, traceback]]) + + Raises an exception of type ``type`` at the point where generator was paused, + and returns the next value yielded by the generator function. If the generator + exits without yielding another value, a :exc:`StopIteration` exception is + raised. If the generator function does not catch the passed-in exception, or + raises a different exception, then that exception propagates to the caller. + +.. index:: exception: GeneratorExit + + +.. method:: generator.close() + + Raises a :exc:`GeneratorExit` at the point where the generator function was + paused. If the generator function then raises :exc:`StopIteration` (by + exiting normally, or due to already being closed) or :exc:`GeneratorExit` (by + not catching the exception), close returns to its caller. If the generator + yields a value, a :exc:`RuntimeError` is raised. If the generator raises any + other exception, it is propagated to the caller. :meth:`close` does nothing + if the generator has already exited due to an exception or normal exit. + +Here is a simple example that demonstrates the behavior of generators and +generator functions:: + + >>> def echo(value=None): + ... print("Execution starts when 'next()' is called for the first time.") + ... try: + ... while True: + ... try: + ... value = yield value + ... except Exception as e: + ... value = e + ... finally: + ... print("Don't forget to clean up when 'close()' is called.") + ... + >>> generator = echo(1) + >>> next(generator) + Execution starts when 'next()' is called for the first time. + 1 + >>> next(generator) + None + >>> generator.send(2) + 2 + >>> generator.throw(TypeError, "spam") + TypeError('spam',) + >>> generator.close() + Don't forget to clean up when 'close()' is called. + +When using :token:`yield_from`, work is delegated to the subgenerator:: + + >>> def outer(): + ... print("Outer generator started") + ... yield from echo() + ... print("Outer generator done") + >>> generator = outer() + >>> next(generator) + Outer generator started + Execution starts when 'next()' is called for the first time. + >>> generator.send(2) + 2 + >>> generator.throw(TypeError, "spam") + TypeError('spam',) + >>> generator.close() + Don't forget to clean up when 'close()' is called. + Outer generator done # XXX: this doesn't appear in Python 3.3.0a0 (pep380:c589658c828e, Sep 4 2011, 12:53:53) + >>> generator.close() .. seealso:: @@ -475,9 +608,12 @@ called, allowing any pending :keyword:`finally` clauses to execute. The proposal for adding generators and the :keyword:`yield` statement to Python. :pep:`0342` - Coroutines via Enhanced Generators - The proposal that, among other generator enhancements, proposed allowing - :keyword:`yield` to appear inside a :keyword:`try` ... :keyword:`finally` block. + The proposal to enhance the API and syntax of generators, making them + usable as simple coroutines. + :pep:`0380` - Syntax for Delegating to a Subgenerator + The proposal to introduce the :token:`yield_from` syntax, making delegation + to sub-generators easy. .. _raise: -- 1.7.6