classification
Title: Better expose closure, generator & coroutine status of functions
Type: enhancement Stage: patch review
Components: Interpreter Core Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, berker.peksag, ezio.melotti, mark.dickinson, ncoghlan, r.david.murray, rhettinger, serhiy.storchaka, terry.reedy, yselivanov
Priority: normal Keywords: patch

Created on 2015-04-25 06:13 by ncoghlan, last changed 2018-09-14 22:31 by petr.viktorin.

Files
File name Uploaded Description Edit
issue24056.diff berker.peksag, 2015-04-28 11:29 review
issue24056_2.diff yselivanov, 2015-05-21 20:00 review
Messages (14)
msg241994 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-04-25 06:13
From https://mail.python.org/pipermail/python-ideas/2015-April/033177.html, there are some additional details about functions that could be usefully exposed in the function repr, specifically whether or not it's a closure, and whether or not it's a generator function.

Proposed display:

    <function f at 0x7f7dad9f7bf8 (closure)>
    <function f at 0x7f7dad9f7bf8 (generator)>
    <function f at 0x7f7dad9f7bf8 (closure,generator)>

The Python level checks for the two flags:

closure: f.__closure__ is not None
generator: c.__code__.co_flags & inspect.CO_GENERATOR

Actual implementation would be in the C code at https://hg.python.org/cpython/file/default/Objects/funcobject.c#l569
msg242176 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2015-04-28 11:29
Here is a patch with a test.
msg242185 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2015-04-28 14:27
I can see that the `generator` information would be useful.  What's the use-case for reporting that a function is a closure?  I'm having trouble thinking of a case where it's useful to know that a function is a closure without also knowing which locals refer to cells.
msg242214 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-04-29 06:40
It's mostly pedagogical - similar to "normal functions" vs "generator functions", folks talk about functions and closures as different things, even though in Python a closure is just a normal function with one or more references to cells that were defined in outer scopes.

Having that show up in the repr() then becomes a way of clarifying that some, but not all, Python function objects are closures, even though closures aren't represented as a distinct type.

That difference also shows up in the bytecode that creates them (note the MAKE_FUNCTION vs MAKE_CLOSURE):


>>> def outer():
...     x = 1
...     def inner_function():
...         pass
...     def inner_closure():
...         return x
... 
>>> import dis
>>> dis.dis(outer)
  2           0 LOAD_CONST               1 (1)
              3 STORE_DEREF              0 (x)

  3           6 LOAD_CONST               2 (<code object inner_function at 0x7fade75e5c90, file "<stdin>", line 3>)
              9 LOAD_CONST               3 ('outer.<locals>.inner_function')
             12 MAKE_FUNCTION            0
             15 STORE_FAST               0 (inner_function)

  5          18 LOAD_CLOSURE             0 (x)
             21 BUILD_TUPLE              1
             24 LOAD_CONST               4 (<code object inner_closure at 0x7fade75e5a50, file "<stdin>", line 5>)
             27 LOAD_CONST               5 ('outer.<locals>.inner_closure')
             30 MAKE_CLOSURE             0
             33 STORE_FAST               1 (inner_closure)
             36 LOAD_CONST               0 (None)
             39 RETURN_VALUE

One particular case where the distinction matters and is known to be genuinely confusing for new Python users is the late binding behaviour of closures:

    lambda: i # closure
    lambda i=i: i # not a closure
msg242360 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-05-02 00:21
Describing generator functions as such is a great idea.  But how about
    <generator function f at 0x7f7dad9f7bf8>

Marking closure functions as such is a bit more subtle. However, there ia a real point that closure functions have a hidden input.  If it is mutated or rebound, the function will not be deterministic with respect to its overt input arguments.  Closure functions are similar to methods in this respect.

Await functions (Guido's name choice as of today), if the PEP is approved, will also need to be identified as such.  I propose a uniform format of no prefix, a single prefic or a (tuple) of prefixes.
msg242403 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-05-02 15:29
The main reason I suggest using the postfix parenthetical syntax is to make
it clear that we're exposing "behavioural feature flags" for a single
underlying type. A prefix syntax would make them look like distinct types,
which would be misleading in a different way.
msg242406 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-05-02 16:48
Although I like the look of the repr Terry proposes better, I agree with Nick: it would imply that the types were distinct, which they are not.
msg243780 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2015-05-21 19:33
Nick, Berker, please find an updated patch attached (with support for coroutines).  Big +1 on the idea, BTW.
msg243834 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2015-05-22 15:35
Nick, Berker, a kind reminder -- please review the patch if we want to have it in 3.5.
msg243838 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2015-05-22 16:33
I'm not the ideal candidate to review the second patch since I'm not familiar with the best practices of C :)
msg243845 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-05-22 18:39
I like the look of the repr Terry proposes better. For generator objects the repr is either "<coroutine object %S at %p>" or "<generator object %S at %p>". "<coroutine function %S at %p>" and "<generator function %S at %p>" would be consistent with this. It also shows the relation and the difference between the generator function and the generator object.

Yet one argument is that both terms "generator object" and "generator function" are searchable in the documentation.

There are other precedences with exposing flags at the start of the repr. "<built-in function %s>" and "<built-in method %s of %s object at %p>", "<unlocked %s object at %p>" and "<locked %s object at %p>".
msg243861 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-05-22 22:34
> It's mostly pedagogical - similar to "normal functions" 
> vs "generator functions",

I see a need for this but object to calling it a "generator" rather than a "function that makes a generator" or "generator creating function" or somesuch.  There is a huge semantic difference between the two.

Another thought this that I'm not sure that a __repr__ should try usurp something that is the primary responsibility of a docstring or function annotation here.  Whether a function call runs code and returns a value or whether it returns a generator is fundamental to what the function does.  The usual job of the __repr__ is to tell what the object is.  The usual job of a docstring or type annotation to the describe what is returned.

> Marking closure functions as such is a bit more subtle.
> However, there ia a real point that closure functions 
> have a hidden input.

I don't see a need for this and think it make cause more confusion than help.  I try to teach that callables are all conceptually the same thing (something that has a __call__ method).  It matters very little whether a callable is implemented as a closure or using a class with a __call__ method.

So, put me down for -1 on this one.
msg243885 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-05-23 05:00
I don't think we should rush this one, especially as PEP 484 provides the possibility for tools (including educational tools) to infer the appropriate return types for generator and coroutine functions.

Bumping the target version to 3.6 accordingly.
msg243895 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-05-23 08:03
I've also come to agree with Raymond that the repr may not be the best place for this additional information, and have updated the issue title accordingly.

For example, as one possible alternative, we might be able to put something in the inspect module (e.g. "inspect.callable_info()") that's a higher level alternative to "dis.code_info()".

The information displayed could potentially include:

- the object's repr
- the str() of the callable's signature (presented with the name or qualname if it has one)
- the names and current repr of any captured closure variables

More controversially, it might include an inferred return type annotation when there's no explicit annotation to display (defaulting to "typing.Any", but potentially more explicit if it's possible to tell from the code object flags that calling it will return a Coroutine or Generator)
History
Date User Action Args
2018-09-14 22:31:28petr.viktorinsetnosy: - petr.viktorin
2016-01-03 10:11:55ezio.melottisetnosy: + ezio.melotti
2015-07-21 07:10:44ethan.furmansetnosy: - ethan.furman
2015-05-23 08:03:43ncoghlansetmessages: + msg243895
title: Expose closure & generator status in function repr() -> Better expose closure, generator & coroutine status of functions
2015-05-23 05:00:06ncoghlansetmessages: + msg243885
versions: + Python 3.6, - Python 3.5
2015-05-22 22:34:00rhettingersetnosy: + rhettinger
messages: + msg243861
2015-05-22 18:39:12serhiy.storchakasetmessages: + msg243845
2015-05-22 16:33:53berker.peksagsetnosy: + serhiy.storchaka
messages: + msg243838
2015-05-22 15:35:30yselivanovsetmessages: + msg243834
2015-05-21 20:00:37yselivanovsetfiles: + issue24056_2.diff
2015-05-21 19:58:41yselivanovsetfiles: - issue24056_2.diff
2015-05-21 19:33:19yselivanovsetfiles: + issue24056_2.diff

messages: + msg243780
2015-05-03 07:14:32Arfreversetnosy: + Arfrever
2015-05-02 16:48:58r.david.murraysetnosy: + r.david.murray
messages: + msg242406
2015-05-02 15:29:47ncoghlansetmessages: + msg242403
2015-05-02 00:21:26terry.reedysetnosy: + terry.reedy
messages: + msg242360
2015-04-29 06:40:07ncoghlansetmessages: + msg242214
2015-04-28 14:27:11mark.dickinsonsetnosy: + mark.dickinson
messages: + msg242185
2015-04-28 11:29:17berker.peksagsetfiles: + issue24056.diff

components: + Interpreter Core

keywords: + patch
nosy: + berker.peksag
messages: + msg242176
stage: needs patch -> patch review
2015-04-25 15:41:51petr.viktorinsetnosy: + petr.viktorin
2015-04-25 14:05:45ethan.furmansetnosy: + ethan.furman
2015-04-25 06:43:00yselivanovsetnosy: + yselivanov
2015-04-25 06:16:26ncoghlansettype: enhancement
stage: needs patch
2015-04-25 06:13:53ncoghlancreate