classification
Title: @public - an __all__ decorator
Type: enhancement Stage: patch review
Components: Interpreter Core Versions: Python 3.6
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: barry, berker.peksag, eryksun, ethan.furman, jayvdb, leewz, martin.panter, ncoghlan, r.david.murray, rhettinger, zach.ware
Priority: normal Keywords: patch

Created on 2016-03-24 02:33 by barry, last changed 2017-01-10 06:20 by ncoghlan. This issue is now closed.

Files
File name Uploaded Description Edit
26632-in-c.diff barry, 2016-05-06 15:06 review
26632-in-c-2.diff barry, 2016-05-06 15:26 review
26632-in-c-3.diff barry, 2016-05-06 17:04 review
26632-in-c-4.diff barry, 2016-05-08 20:27 review
26632-in-c-5.diff barry, 2016-05-12 21:27 review
Messages (42)
msg262319 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-03-24 02:33
This is probably terrible, but given how difficult it is to keep __all__'s up to date, maybe something like this can be useful.  I literally whipped this up in about 5 minutes, so sit back and watch the bikeshedding!

import sys

def public(thing):
    if isinstance(thing, str):
        mdict = sys._getframe(1).f_globals
        name = thing
    else:
        mdict = sys.modules[thing.__module__].__dict__
        name = thing.__name__
    dunder_all = mdict.setdefault('__all__', [])
    dunder_all.append(name)
    return thing


Then:

@public
def baz(a, b):
    return a + b

@public
def buz(c, d):
    return c / d

def qux(e, f):
    return e * f

class zup:
    pass

@public
class zap:
    pass

public('CONST1')
CONST1 = 3

CONST2 = 4

public('CONST3')
CONST3 = 5

Normally for any callable with an __name__, you can just decorate it with @public to add it to __all__.  Of course that doesn't worth with things like constants, thus the str argument.  But of course that also requires sys._getframe() so blech.
msg262320 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-03-24 02:35
Oh, and it should be a built-in <wink>
msg262321 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2016-03-24 02:48
def public(thing, value=None):
    if isinstance(thing, str):
        mdict = sys._getframe(1).f_globals
        name = thing
        mdict[name] = thing  # no need for retyping! ;)
    else:
        mdict = sys.modules[thing.__module__].__dict__
        name = thing.__name__
    dunder_all = mdict.setdefault('__all__', [])
    dunder_all.append(name)
    return thing

@public
def baz(a, b):
    return a+ b

public('CONST1', 3)

CONST2 = 4

----------------------------

On the down side, you know somebody is going to @public a class' method -- how do we check for that?
msg262346 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-03-24 14:19
On Mar 24, 2016, at 02:48 AM, Ethan Furman wrote:

>On the down side, you know somebody is going to @public a class' method --
>how do we check for that?

Do we need to?  Consenting adults and __all__.

OT1H, you do get an AttributeError if you from-import-* and there are things
in __all__ that aren't in the module.  OTOH, it's a pretty obvious error.
msg262380 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-03-24 22:31
FWIW I already invented this :) as written in Issue 22247. Although I think I only used it once or twice in my own personal librarie(s). So it’s a nice sign that we came up with the same @public name and usage.

I’m not a fan of hacks depending on the calling frame, and I prefer APIs that “only do one thing”. So I am okay with accepting an object and work off its __name__ and __module__, but not okay with also accepting a string and guessing what module it was defined in.
msg262383 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-03-24 22:52
On Mar 24, 2016, at 10:31 PM, Martin Panter wrote:

>FWIW I already invented this :) as written in Issue 22247. Although I think I
>only used it once or twice in my own personal librarie(s). So it’s a nice
>sign that we came up with the same @public name and usage.

Cool!  Consider that bikeshed painted then. :)

>I’m not a fan of hacks depending on the calling frame, and I prefer APIs that
>“only do one thing”. So I am okay with accepting an object and work off its
>__name__ and __module__, but not okay with also accepting a string and
>guessing what module it was defined in.

Yes, but it makes it less convenient to add non-"APIs" to __all__, although I
guess you can just append it at the point of use:

__all__.append('CONST1')
CONST1 = 3

Not as pretty, and now you have two ways of doing it.

Here's another thought:

What if we gave all modules an __all__ automatically, and that was an object
that acted like a list but also had an @public decorator.

import sys

class All(list):
    def public(self, api):
        sys.modules[api.__module__].__all__.append(api.__name__)

__all__ = All()

@__all__.public
def foo():
    pass

@__all__.public
class Bar:
    pass

__all__.append('CONST')
CONST = 1

print(__all__)
msg262389 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2016-03-25 00:14
Not a fan.  :/

How about getting your own copy of the public decorator initialized with the globals you pass in?

class Public:
    def __init__(self, module):
        """
        module should be the globals() dict from the calling module
        """
        self.module = module
        self.module.setdefault('__all__', [])
    def __call__(self, thing, value=None):
        if isinstance(thing, str):
            self.module[thing] = value
        else:
            self.module[thing.__name__] = thing

and in use:

public = Public(globals())

@public
def baz(a, b):
    #blah blah

public('CONST1', 2)
msg262390 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-03-25 00:24
On Mar 25, 2016, at 12:14 AM, Ethan Furman wrote:

>public = Public(globals())
>
>@public
>def baz(a, b):
>    #blah blah
>
>public('CONST1', 2)

I'm not crazy about that, plus I rather don't like the implicit binding of the
name.  I suppose we should just drop the idea of convenience for non-"API".
Just use the defined @public for classes and functions, and an explicit
__all__.append('CONST') for other names.
msg262396 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-03-25 02:12
> work off its __name__ and __module__

Why is __module__ required? It seems to me this should only operate on the current module. 

I added a prototype to Python/bltinmodule.c that gets or creates the __all__ list from the current globals (i.e. PyEval_GetGlobals). It accepts at most one positional argument and any number of keyword arguments. It adds the positional argument's __name__ to __all__, sets it in globals, and returns a reference for use as a decorator. The keyword argument dict is used to update globals and extend __all__. 

    Python 3.6.0a0 (default:3eec7bcc14a4+, Mar 24 2016, 20:40:52) 
    [GCC 4.8.4] on linux
    Type "help", "copyright", "credits" or "license" for more
    information.
    >>> @public
    ... def foo():
    ...     pass
    ... 
    >>> def bar(): pass
    ... 
    >>> public(bar, spam=1, eggs=2)
    <function bar at 0x7efe96ca1048>
    >>> __all__
    ['foo', 'spam', 'eggs', 'bar']
    >>> foo, bar
    (<function foo at 0x7efe96c8af28>, <function bar at 0x7efe96ca1048>)
    >>> spam, eggs
    (1, 2)

Maybe it should be generalized to handle multiple positional arguments. Currently it's an error:

    >>> public(foo, bar)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: public expected at most 1 arguments, got 2

The positional argument must have a __name__ that's a string:

    >>> public('CONST')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: 'str' object has no attribute '__name__'
    >>> class C:
    ...     __name__ = 1
    ... 
    >>> public(C())
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: __name__ must be a string

If it's used to decorate a method definition, it stores a reference to the function in the module's globals. That's not very useful, but at least it won't lead to an error with a star import.
msg262562 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2016-03-28 06:34
[Barry]
> This is probably terrible ...

I have to agree with that part ;-)   Sorry, but this feels "yucky" and is likely to cause more problems than it solves.
msg262564 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-03-28 07:26
Agree with Raymond.
msg262568 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-03-28 12:14
On Mar 28, 2016, at 06:34 AM, Raymond Hettinger wrote:

>I have to agree with that part ;-) Sorry, but this feels "yucky" and is
>likely to cause more problems than it solves.

I've been experimenting with something like this in a Mailman branch and I've
come to like it much more than I did originally.  I'm using the "simple"
implementation, so that means that I have a very few explicit appends to
__all__.

But the use of @public is actually pretty great.  I don't have to worry about
__all__ getting out of sync (and there turned out to be lots of places where
that was happening), it's quite visually appealing (easy to pick out in a
crowded file), and it avoids nasty pep8 conflicts.

The major downside is actually having to import it from a module very early in
the startup sequence.  I stick it in mailman/__init__.py but that kind of
sucks because I want to move to making that a namespace package so I want to
*remove* mailman/__init__.py but there's no other obvious place to put an
early `public` definition.

Thus after experimenting with it, I'm much more in favor of it.  Could you
please describe what you don't like about it and what problems you think it
will cause?

(Plus, I encourage you to try it on a medium to large size project!)
msg262575 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-03-28 16:26
There is a helper in test.support that helps to add a test for __all__, and we slowly add these test for all modules. So this is not large issue for the stdlib.

New module level names are added not very often. Keeping __all__ in sync is not the largest problem in the maintaining.
msg262617 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-03-29 17:38
On Mar 28, 2016, at 04:26 PM, Serhiy Storchaka wrote:

>There is a helper in test.support that helps to add a test for __all__, and
>we slowly add these test for all modules. So this is not large issue for the
>stdlib.
>
>New module level names are added not very often. Keeping __all__ in sync is
>not the largest problem in the maintaining.

stdlib use is not really the point of this proposal.  It's for all those 3rd
party projects that use __all__.
msg262697 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-03-31 17:17
On Mar 25, 2016, at 02:12 AM, Eryk Sun wrote:

>I added a prototype to Python/bltinmodule.c that gets or creates the __all__
>list from the current globals (i.e. PyEval_GetGlobals).

Hi Eryk.  Can you post your diff to bltinmodule.c?  I'd like to see your code.

>It accepts at most one positional argument and any number of keyword
>arguments. It adds the positional argument's __name__ to __all__, sets it in
>globals, and returns a reference for use as a decorator. The keyword argument
>dict is used to update globals and extend __all__.

I like this.  The dual functionality of `public` looks like it will handle
almost all use cases.  I think we're in widespread agreement about the
decorator, and the keyword arguments are a nice approach to public constants.

I guess I'm a little less sure about the positional argument API.  In adding
@public to Mailman, I noticed there are a few public names which are
instances.  These could be "publicized" with the keyword argument approach,
but I don't think they can work as positional arguments, because the instances
themselves don't have __name__s.  For example, currently:

factory = DateFactory()
factory.reset()
today = factory.today
now = factory.now
layers.MockAndMonkeyLayer.register_reset(factory.reset)

__all__.extend([
    'factory',
    'now',
    'today',
    ])

With only keyword arguments, which isn't bad:

public(factory=DateFactory())
factory.reset()
public(today=factory.today, now=factor.now)

What's the use case for positionals?

>The positional argument must have a __name__ that's a string:

Right.  But what class of objects does that cover that isn't already covered
(or that explicitly appending to __all__ is good enough)?
msg262878 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-04-04 23:05
Here's my implementation based on eryksun's idea:

def public(thing=None, **kws):
    mdict = (sys._getframe(1).f_globals
             if thing is None
             else sys.modules[thing.__module__].__dict__)
    dunder_all = mdict.setdefault('__all__', [])
    if thing is not None:
        dunder_all.append(thing.__name__)
    for key, value in kws.items():
        dunder_all.append(key)
        mdict[key] = value
    return thing
msg264977 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-06 15:06
Here's a C implementation.  I'm a bit under the weather so please do double check my refcounting logic. ;)

No tests or docs yet, but those would be easy to add.  Here's an example:

@public
class Foo:
    pass

public(qux=3)

print(qux)

@public
def zzz():
    pass

public(jix=1, jox=2, jrx=3)

print(__all__)
print(jix, jox, jrx)

You could also try to add an explicit __all__ in the module and those names will get appended to it.
msg264980 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-06 15:26
I think I missed a decref.  New diff.
msg264987 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-05-06 15:53
Added a couple of comments on Rietveld.

But sorry, the overall idea looks unpythonic to me. I'm strong -1.
msg264997 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-06 17:04
Updated.
msg265001 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-05-06 17:23
"This will cause more problems than it solves" and "this looks unpythonic" are, IMO, not strong arguments against it without butressing discussion.  If we can have some examples of problems it will cause, or a concrete explanation of wy something that makes code easier to understand and update (by putting the __all__ declaration next to the object being made public) is unpythonic, that would also help.
msg265165 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-08 20:27
One more diff.  As I was working on tests, I realized that the decorator version wasn't returning the thing it was decorating.  Changing this also allowed me to simplify the exit path.

I should be putting up a PyPI package soon which implements this for earlier Pythons (Python 3 only for now though).
msg265198 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-09 15:11
Here's a standalone version for older Python 3's:

http://public.readthedocs.io/en/latest/
https://pypi.python.org/pypi/atpublic
msg265207 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2016-05-09 16:19
For the standalone version I suggest a disclaimer about the `from ... import *` ability.  Something like:

`from ... import *` should not be used with packages that do not have an __all__ unless they support that usage (check their docs).
msg265211 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-09 16:59
On May 09, 2016, at 04:19 PM, Ethan Furman wrote:

>`from ... import *` should not be used with packages that do not have an
>__all__ unless they support that usage (check their docs).

Good idea; I added a warning-ish.
msg265428 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-12 21:27
Now with docs and tests.
msg265910 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-05-20 07:02
+1 for the idea. I saw a lot of different 'all' or 'public' decorators in the wild. It would be nice to have a complete solution in Python. It would be good to add a note to Doc/whatsnew/3.6.rst.
msg265911 - (view) Author: Franklin? Lee (leewz) Date: 2016-05-20 07:21
I like how ``@public`` keeps the declaration close to the definition.

I am iffy about using ``public`` to define other values. That part might be considered unpythonic.


Implementation issues:
    - ``__module__`` is not reliable. ``functools.wraps`` changes it. (Why does it do that, though?)
    - If `__all__` isn't a list, you'd have to make it a list before you mess with it. (Is this possible?)


> > On the down side, you know somebody is going to @public a class' method --
> > how do we check for that?
>
> Do we need to?  Consenting adults and __all__.

It's a silent error waiting to happen. If you never use ``import *`` on it (e.g. because it's your main file), you won't get the error message. Things will work "as expected" (your methods are class-public!) until you give a method the same name as a builtin or something you imported or defined earlier. When that happens, the error message will have nothing to do with the problem.

It might be detectable using ``thing.__qualname__ != thing.__name__``, but this disallows functions decorated without updating __qualname__, and static/class methods exposed in a module's interface.

It might be detectable by checking, on the callstack, whether you're in a module load or a class definition.


Bikeshed
========


How many public module values aren't enum-type constants? It could be useful to be able to dump an enum into a module's space. I mean, a canonical way. With that, maybe maintaining module-level constants in __all__ isn't that big a deal.

    # Rather than:
    globals().update(MyEnum.__members__)
    __all__.extend(MyEnum.__members__)
    # Perhaps allow:
    enum.dump_namespace(MyEnum, globals())


About the cost paid at every load:
    - Should tools update __all__ for you, and comment out the ``@public``s?
        - If so, how would they deal with public non-callable values?
    - When compiling to .pyc, should the compiler remove ``@public`` calls and explicitly add the values to __all__?


API:
    - Alternative syntax for constants, requiring less frame hackery:
        public(globals(), x=1, y=2, z=3)
    - Naming: Is it really "public"? Some names might be public but not in __all__.


P.S. Typo in the ReadTheDocs. ``py_install`` should be a function call, right?

    >>> from public import py_install
    >>> py_install


P.S.: Would OrderedSet (which doesn't exist) be the ideal type for __all__? I mean, if you had to use someone else's __all__, not if you had to maintain it.
msg266107 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-22 20:59
On May 20, 2016, at 07:21 AM, Franklin? Lee wrote:

>I am iffy about using ``public`` to define other values. That part might be
>considered unpythonic.

It's a bit of a stretch.  I like it for the convenience, and the
implementation is simple, but if e.g. Guido disliked this part of it, I'd be
okay dropping it.  I think the use on non-__name__'d things is rare enough
that a little inconvenience wouldn't be a fatal flaw.

>    - ``__module__`` is not reliable. ``functools.wraps`` changes it. (Why
>    - does it do that, though?)

I don't know, but what practical effect will this have?  I.e. under what
conditions would you @public wrap a @functools.wraps function and want it to
show up in __all__?  Do you have a specific use case?

Also note that this is a subtle difference between the C and Python
implementations.  I actually expect that if this were adopted for Python 3.6,
we'd pretty much only use the C version.  In the standalone package, I'm
including the Python versions mostly just for convenience in environments
without a compiler (though maybe a built wheel for some platforms would be
useful).

>    - If `__all__` isn't a list, you'd have to make it a list before you mess
>    - with it. (Is this possible?)

It would be possible.  I'd just do the equivalent of::

    __all__ = list(__all__)

But I actually think that best practice would be not to set __all__ explicitly
if you're going to use @public.  If you really want it to be immutable, you'd
put the following at the bottom of your module:

    __all__ = tuple(__all__)

For now I've added some usage caveats that describe these points.

>> > On the down side, you know somebody is going to @public a class' method --
>> > how do we check for that?  
>>
>> Do we need to?  Consenting adults and __all__.  
>
>It's a silent error waiting to happen. If you never use ``import *`` on it
>(e.g. because it's your main file), you won't get the error message. Things
>will work "as expected" (your methods are class-public!) until you give a
>method the same name as a builtin or something you imported or defined
>earlier. When that happens, the error message will have nothing to do with
>the problem.
>
>It might be detectable using ``thing.__qualname__ != thing.__name__``, but
>this disallows functions decorated without updating __qualname__, and
>static/class methods exposed in a module's interface.
>
>It might be detectable by checking, on the callstack, whether you're in a
>module load or a class definition.

Sure, we could probably add some heuristics, but I still don't see the need
for the extra complexity.  The error will be far from the declaration, but the
exception should make it relatively obvious what's going on.  I also really
don't think folks would naturally be inclined to put @public on anything but a
top-level definition.  You wouldn't ever put such a thing in your __all__ so
why would you decorate it with @public?

In any case, I've added a usage caveat for this case too.

>How many public module values aren't enum-type constants?

These days I bet they are quite a bit more common than enum-types, although I
agree that enums are great and we should use more of them!  Just historically
speaking I don't know how many packages have converted all their constants
over to enums.

Also, I know that I have several cases where constants are actually
instances.  They could be marker objects like::

    MARKER = object()

or system globals::

    configuration = Configuration()

I'd want both of those in __all__.

>It could be useful to be able to dump an enum into a module's space. I mean,
>a canonical way. With that, maybe maintaining module-level constants in
>__all__ isn't that big a deal.
>
>    # Rather than:
>    globals().update(MyEnum.__members__)
>    __all__.extend(MyEnum.__members__)
>    # Perhaps allow:
>    enum.dump_namespace(MyEnum, globals())

It's an interesting thought.

>About the cost paid at every load:
>    - Should tools update __all__ for you, and comment out the ``@public``s?
>        - If so, how would they deal with public non-callable values?
>    - When compiling to .pyc, should the compiler remove ``@public`` calls
>    and explicitly add the values to __all__?

Why?  Aren't those one-time costs only borne when the module is originally
imported?

>API:
>    - Alternative syntax for constants, requiring less frame hackery:
>        public(globals(), x=1, y=2, z=3)

Possibly.  Since this is really only relevant for the pure-Python
implementation, I'm not as keen on the extra cruft.

>    - Naming: Is it really "public"? Some names might be public but not in
>    - __all__.

What does it mean for a name to be "public but not in __all__"?

I'll also note that since the basic API has been independently invented at
least three times, and all of them use @public, it seems like the obvious
choice. ;)

>P.S. Typo in the ReadTheDocs. ``py_install`` should be a function call, right?
>
>    >>> from public import py_install
>    >>> py_install  

Fixed, thanks!

>P.S.: Would OrderedSet (which doesn't exist) be the ideal type for __all__? I
>mean, if you had to use someone else's __all__, not if you had to maintain
>it.

It's an interesting thought, but I don't know if it's enough of a use case to
add collections.OrderedSet.  Traditionally __all__ has been a list, with some
relatively recent moves to making it a tuple.

Thanks for the interesting and useful feedback!
msg266108 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-22 20:59
On May 20, 2016, at 07:02 AM, Berker Peksag wrote:

>+1 for the idea. I saw a lot of different 'all' or 'public' decorators in the
>wild. It would be nice to have a complete solution in Python. It would be
>good to add a note to Doc/whatsnew/3.6.rst.

Is that a pronouncement? :)
msg266117 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-05-22 23:30
Here are two examples where publicly-documented module attributes are intentionally omitted from __all__:

* Issue 26234: typing.re and typing.io
* Issue 23439: HTTP status codes like http.client.NOT_FOUND

Despite these, I think @public is a reasonable name. But I may be biased, because I personally think everything should be included in __all__. Otherwise pydoc does not pick it up.
msg266120 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-23 00:50
On May 22, 2016, at 11:30 PM, Martin Panter wrote:

>Here are two examples where publicly-documented module attributes are
>intentionally omitted from __all__:
>
>* Issue 26234: typing.re and typing.io
>* Issue 23439: HTTP status codes like http.client.NOT_FOUND

Wild.

>Despite these, I think @public is a reasonable name. But I may be biased,
>because I personally think everything should be included in
>__all__. Otherwise pydoc does not pick it up.

I think it's pretty reasonable, and pretty well-established despite some
exceptions, that __all__ names a module's public symbols.
msg266152 - (view) Author: Franklin? Lee (leewz) Date: 2016-05-23 14:43
>I don't know, but what practical effect will this have?  I.e. under what
>conditions would you @public wrap a @functools.wraps function and want it to
>show up in __all__?  Do you have a specific use case?

I sometimes wrap functions that return iterators to make functions that return lists, because I work on the interpreter a lot. From there, it's not much of a stretch to imagine functions which are implemented as decorated versions of other functions.

If @public were only to be used as a decorator, it would not be possible to have `public` called on a function outside of its definition. But someone might call `public(some_decorator(some_function))`.

(@public is really a macro, if you think about it.)

>It would be possible.

(I meant, is it possible for someone to have a non-list __all__?)

If the `.append` fails, I think there should be a meaningful error. Perhaps "'__all__' is not a list."

>Sure, we could probably add some heuristics, but I still don't see the need
>for the extra complexity.  The error will be far from the declaration, but the
>exception should make it relatively obvious what's going on.  I also really
>don't think folks would naturally be inclined to put @public on anything but a
>top-level definition.  You wouldn't ever put such a thing in your __all__ so
>why would you decorate it with @public?

I'm thinking of the people who don't read docs and are coming from other languages. They'd put `@public` over their method, and one day they'd `import *` from that file (whereas they used to only import explicitly), getting an error about a name not being defined in their module. "But why would that name need to be defined? It's a method."

Or worse, the name of the method just happens to be the same as something in some other file, so they'll focus on why that NAME is being expected in THIS file.

>Also, I know that I have several cases where constants are actually
>instances.  They could be marker objects like::
>
>    MARKER = object()

(Here's food for thought: A MARKER could be a one-element enum, both conceptually and by implementation. Just like how the "bool enum" is {True,False} and the "None enum" is {None}.)
msg266159 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-23 15:52
On May 23, 2016, at 02:43 PM, Franklin? Lee wrote:

>I sometimes wrap functions that return iterators to make functions that
>return lists, because I work on the interpreter a lot. From there, it's not
>much of a stretch to imagine functions which are implemented as decorated
>versions of other functions.
>
>If @public were only to be used as a decorator, it would not be possible to
>have `public` called on a function outside of its definition. But someone
>might call `public(some_decorator(some_function))`.

Do you mean, they'd call this is some module other than the one some_function
was defined in?  I don't know that this is a use case we even want to support.

>(@public is really a macro, if you think about it.)

That's true in a sense.  It doesn't change the decorated thing at all.  I
think it's important to keep in mind that @public isn't the only way to add to
__all__.

>
>>It would be possible.  
>
>(I meant, is it possible for someone to have a non-list __all__?)

Yes.  I've seen existing code where __all__ is assigned to a tuple.

>If the `.append` fails, I think there should be a meaningful error. Perhaps
>"'__all__' is not a list."

You should get something like:

AttributeError: 'tuple' object has no attribute 'append'

which seems pretty obvious.

>I'm thinking of the people who don't read docs and are coming from other
>languages. They'd put `@public` over their method, and one day they'd `import
>*` from that file (whereas they used to only import explicitly), getting an
>error about a name not being defined in their module. "But why would that
>name need to be defined? It's a method."
>
>Or worse, the name of the method just happens to be the same as something in
>some other file, so they'll focus on why that NAME is being expected in THIS
>file.

Well, consenting adults and all.  I'm not sure we need to protect ourselves so
strictly against people who don't read the docs and don't understand Python
(i.e. random cargo-culters).

>>Also, I know that I have several cases where constants are actually
>>instances.  They could be marker objects like::
>>
>>    MARKER = object()  
>
>(Here's food for thought: A MARKER could be a one-element enum, both
>conceptually and by implementation. Just like how the "bool enum" is
>{True,False} and the "None enum" is {None}.)

Sure.  I don't think that changes anything here though.  Down the line, it
might be an interesting idiom to experiment with (you can probably start with
the standalone enum34 module).
msg266319 - (view) Author: Franklin? Lee (leewz) Date: 2016-05-25 07:56
>>If @public were only to be used as a decorator, it would not be possible to
>>have `public` called on a function outside of its definition. But someone
>>might call `public(some_decorator(some_function))`.
>
>Do you mean, they'd call this is some module other than the one some_function
>was defined in?  I don't know that this is a use case we even want to support.

I mean they'd define their own function as a wrapped version of another function.


>That's true in a sense.  It doesn't change the decorated thing at all.  I
>think it's important to keep in mind that @public isn't the only way to add to
>__all__.

I mean more in that it acts in the scope of its caller, rather than its definition.


>You should get something like:
>
>AttributeError: 'tuple' object has no attribute 'append'
>
>which seems pretty obvious.

I don't think the C version shows a traceback, so it won't be clear that you're trying to assign to `__all__`.

When I rewrote `public` from memory, I wrote it something like this:
        try:
            dunder_all.append(name)
        except TypeError:
            module.__all__ = [*dunder_all, name]


>Well, consenting adults and all.  I'm not sure we need to protect ourselves so
>strictly against people who don't read the docs and don't understand Python
>(i.e. random cargo-culters).

Python is a popular learning language, and many will be students who haven't yet trained to reflexively look up docs. I saw the lack of such habits in Python's IRC channel.

"Consenting adults", I feel, is a reason to grant power: don't stop people from doing something they might need to do. But @public on a class method is just an error.
msg266403 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-26 02:47
On May 25, 2016, at 07:56 AM, Franklin? Lee wrote:

>>AttributeError: 'tuple' object has no attribute 'append'
>>which seems pretty obvious.  
>
>I don't think the C version shows a traceback, so it won't be clear that
>you're trying to assign to `__all__`.

Thanks!  I'll fix both the C and Python versions to raise a ValueError when
__all__ isn't a list.  I'll make the change in my atpublic PyPI project
(version 0.3) and will update the CPython patch if and when it gets accepted.

I think for now at least I won't autoconvert it to a list.  Happy to change
that if e.g. Guido prefers. :)

>>Well, consenting adults and all.  I'm not sure we need to protect ourselves
>>so strictly against people who don't read the docs and don't understand
>>Python (i.e. random cargo-culters).
>
>Python is a popular learning language, and many will be students who haven't
>yet trained to reflexively look up docs. I saw the lack of such habits in
>Python's IRC channel.
>
>"Consenting adults", I feel, is a reason to grant power: don't stop people
>from doing something they might need to do. But @public on a class method is
>just an error.

Would you consider submitting a patch on the project?

https://gitlab.com/warsaw/public
msg266411 - (view) Author: Franklin? Lee (leewz) Date: 2016-05-26 05:21
I probably won't submit a patch, but I can definitely write a bunch of private notes toward a patch and forget about them. >_>

Idea 1: Crawl up the call tree and see whether you hit a module definition or a class definition first.

Idea 2: Inspect __qualname__ to determine whether it's a method/inner function. That might have the same problems with @wraps, though, and you wouldn't be allowed to use a classmethod as a module's function.

Idea 2 seems to be the right thing. If ".__qualname__ != .__name__", it's an error. If you really want to use `public` in the above cases, you should use the assignment form of `public` (whatever it turns out to be), or explicitly append to __all__.

Rule: The programmer should not explicitly use `public(a_func)`, and should use assignment-form `public` if `@public` is not possible (== the decorator form is not being used at the point of definition). I think this way won't have errors passing silently and failing mysteriously.

First draft of error message (not considering assignment-form):
    "'public' should only be used as a decorator for a function/class in module scope."
msg266746 - (view) Author: Franklin? Lee (leewz) Date: 2016-05-31 09:38
BIKESHEDDING

Here is another concern with decorators and `.__module__` (or `inspect.getmodule`). (Or is it the original concern?)

If an earlier decorator creates a wrapper and doesn't set .__module__, you'll make the function public in the wrong module.

    @public
    @bad_wrapper
    def f(...): ...


IMO, the correct solution is a new stdlib function, to get the currently-loading module. You wouldn't need explicit frame hacking, and it will be portable between different Python implementations because each implementation will have to define it correctly.

The problem with this solution is the lack of other usecases for such a function, though maybe someone in python-ideas can think of one.

Candidate owners: sys, importlib.

Candidate names:
- sys.getloadingmodule()
- importlib.currentimport()
- importlib.???()

Effect:
- Returns the "innermost" loading module. If no module is loading, returns None?
- In a thread, return the module being loaded in that thread (possibly None, even if other threads are loading modules).
- (I don't know what I'm talking about:) If a Python implementation uses multithreaded loading, each thread whose result could immediately be loaded into the module's namespace (i.e. it's close to the "surface" of the load) is considered to be loading that module.
- Needs to handle re-imports, such as from `importlib.reload`.

Other solutions include:
- Require explicit `@public(__all__)` (or `__module__`)
- Special `super()`-like treatment of `public` which automatically inserts the `__all__` (or whatever).
- Hope that bad wrappers don't come up.


----

I think nonexistence of module.__all__ should be an error.

The behavior with a module with __all__ is very different from one without, so I think that if you want your module to have __all__, you should be required to create it, even if it's empty.

Assuming that __all__ is, by convention, always near the top, someone reading your code would know whether the first dozen or so functions are part of the public interface, without searching for `@public`.
msg266756 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-05-31 16:10
I'm going to go ahead and close this issue.  There wasn't much positive reception to @public in the Pycon 2016 language summit (where I presented it as a lightning talk).

https://gitlab.com/warsaw/public

Some of the other ideas for changes to Python may still be valid, but they would be best discussed under a new issue.
msg267305 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-06-04 19:47
I had seen this go by on new-bugs-announce, but hadn't formed an opinion about it.  I had also noticed several issues about __all__, but hadn't realized how widespread those issues were.  Now I've just been introduced to #23883 via #26809, and was sad to find this closed when I came to give my +1.

So, +1 anyway.  I think this would be rather worthwhile, especially in the stdlib.
msg267734 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2016-06-07 19:29
On Jun 04, 2016, at 07:47 PM, Zachary Ware wrote:

>So, +1 anyway.  I think this would be rather worthwhile, especially in the
>stdlib.

Thanks!

I still like it and plan on continuing to use it in my code.  I would
recommend you playing with the third party module:

https://pypi.python.org/pypi/atpublic

and seeing how you like it.  If you do like it maybe help get some momentum
behind it.  Then we can approach python-dev and try to get it into builtins.
I'd be willing to bring it up there (it didn't get such an overwhelming
reception at the Pycon 2016 language summit ;).
msg285092 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-01-10 06:20
Chiming in so Barry & Zach can take this feedback into account for any future proposal:

- I think the specific term "public" has too much baggage from other languages, especially in the sense that it implies that "private" is the default. In reality, all attributes in Python are public, with __all__ serving as executable documentation and a constraint on wildcard exports

- the notion of an automatically supplied __all__ object that defaults to reporting [name for name in dir(module) if not name.startswith("_")] has a lot more to recommend it (as it encodes the default wildcard export behaviour directly into the language implementation, rather than requiring that module analysis tools implement that default themselves)

- given "__all__" as the base name, the method name for class and function registration could just be the generic "__all__.register" (although "__all__.export" would also work, since it's the counterpart of "import *")

- using an object namespace would allow for other registration methods if that seemed appropriate

Even then, I'm at best +0 on the proposal, but I also don't make heavy use of code development helpers (IDEs, etc)
History
Date User Action Args
2017-01-10 06:20:18ncoghlansetnosy: + ncoghlan
messages: + msg285092
2016-06-07 19:29:41barrysetmessages: + msg267734
2016-06-04 19:47:43zach.waresetnosy: + zach.ware
messages: + msg267305
2016-05-31 16:10:36barrysetstatus: open -> closed
resolution: wont fix
messages: + msg266756

title: __all__ decorator -> @public - an __all__ decorator
2016-05-31 09:39:00leewzsetmessages: + msg266746
2016-05-26 05:21:28leewzsetmessages: + msg266411
2016-05-26 02:47:27barrysetmessages: + msg266403
2016-05-25 07:56:17leewzsetmessages: + msg266319
2016-05-23 15:52:19barrysetmessages: + msg266159
2016-05-23 14:43:17leewzsetmessages: + msg266152
2016-05-23 00:50:41barrysetmessages: + msg266120
2016-05-22 23:30:53martin.pantersetmessages: + msg266117
2016-05-22 20:59:44barrysetmessages: + msg266108
2016-05-22 20:59:23barrysetmessages: + msg266107
2016-05-20 07:21:42leewzsetnosy: + leewz
messages: + msg265911
2016-05-20 07:07:58serhiy.storchakasetnosy: - serhiy.storchaka
2016-05-20 07:02:49berker.peksagsetnosy: + berker.peksag
messages: + msg265910

components: + Interpreter Core
type: enhancement
stage: patch review
2016-05-12 21:27:16barrysetfiles: + 26632-in-c-5.diff

messages: + msg265428
2016-05-10 16:26:50jayvdbsetnosy: + jayvdb
2016-05-09 16:59:09barrysetmessages: + msg265211
2016-05-09 16:19:13ethan.furmansetmessages: + msg265207
2016-05-09 15:11:26barrysetmessages: + msg265198
2016-05-08 20:27:21barrysetfiles: + 26632-in-c-4.diff

messages: + msg265165
2016-05-06 17:23:22r.david.murraysetnosy: + r.david.murray
messages: + msg265001
2016-05-06 17:04:07barrysetfiles: + 26632-in-c-3.diff

messages: + msg264997
2016-05-06 15:53:31serhiy.storchakasetmessages: + msg264987
2016-05-06 15:26:35barrysetfiles: + 26632-in-c-2.diff

messages: + msg264980
2016-05-06 15:06:45barrysetfiles: + 26632-in-c.diff
keywords: + patch
messages: + msg264977
2016-04-04 23:05:02barrysetmessages: + msg262878
2016-03-31 17:17:16barrysetmessages: + msg262697
2016-03-29 17:38:04barrysetmessages: + msg262617
2016-03-28 16:26:46serhiy.storchakasetmessages: + msg262575
2016-03-28 12:14:04barrysetmessages: + msg262568
2016-03-28 07:26:43serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg262564
2016-03-28 06:34:34rhettingersetnosy: + rhettinger
messages: + msg262562
2016-03-25 02:12:46eryksunsetnosy: + eryksun
messages: + msg262396
2016-03-25 00:24:34barrysetmessages: + msg262390
2016-03-25 00:14:52ethan.furmansetmessages: + msg262389
2016-03-24 22:52:32barrysetmessages: + msg262383
2016-03-24 22:31:02martin.pantersetnosy: + martin.panter
messages: + msg262380
2016-03-24 14:19:49barrysetmessages: + msg262346
2016-03-24 02:48:08ethan.furmansetmessages: + msg262321
2016-03-24 02:38:46ethan.furmansetnosy: + ethan.furman
2016-03-24 02:35:23barrysetmessages: + msg262320
2016-03-24 02:33:17barrycreate