classification
Title: help() not helpful with enum
Type: Stage: resolved
Components: Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ethan.furman Nosy List: barry, eli.bendersky, ethan.furman, ned.deily, python-dev, ronaldoussoren
Priority: normal Keywords: patch

Created on 2013-08-09 01:59 by ethan.furman, last changed 2013-09-16 00:03 by ethan.furman. This issue is now closed.

Files
File name Uploaded Description Edit
issue18693.stoneleaf.01.patch ethan.furman, 2013-09-04 13:39 review
issue18693.stoneleaf.02.patch ethan.furman, 2013-09-15 02:38 review
Messages (24)
msg194714 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-08-09 01:59
help(), when used on an enum member or class, returns almost nothing.  I suspect the custom __dir__ is at fault, but whatever is causing the problem needs fixing.
msg194717 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-08-09 05:20
With custom __dir__:

Help on class Enum in module enum:

Enum = <enum 'Enum'>

----------------------------------------------------

Without custom __dir__:

Help on class Enum in module enum:

class Enum(builtins.object)
 |  Generic enumeration.
 |  
 |  Derive from this class to define new enumerations.
 |  
 |  Methods defined here:
 |  
 |  __eq__(self, other)
 |  
 |  __getnewargs__(self)
 |  
 |  __hash__(self)
 |  
 |  __repr__(self)
 |  
 |  __str__(self)
 |  
 |  ----------------------------------------------------------------------
 |  Static methods defined here:
 |  
 |  __new__(cls, value)
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors defined here:
 |  
 |  __dict__
 |      dictionary for instance variables (if defined)
 |  
 |  __weakref__
 |      list of weak references to the object (if defined)
 |  
 |  name
 |      Route attribute access on a class to __getattr__.
 |      
 |      This is a descriptor, used to define attributes that act differently when
 |      accessed through an instance and through a class.  Instance access remains
 |      normal, but access to an attribute through a class will be routed to the
 |      class's __getattr__ method; this is done by raising AttributeError.
 |  
 |  value
 |      Route attribute access on a class to __getattr__.
 |      
 |      This is a descriptor, used to define attributes that act differently when
 |      accessed through an instance and through a class.  Instance access remains
 |      normal, but access to an attribute through a class will be routed to the
 |      class's __getattr__ method; this is done by raising AttributeError.

---------------------------------------------------------------

I'm thinking we should drop the custom __dir__.  help() is far more important than not seeing some things with dir().
msg194830 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-08-10 20:07
New changeset 5d417257748e by Ethan Furman in branch 'default':
Close #18693: __dir__ removed from Enum; help() now helpful.
http://hg.python.org/cpython/rev/5d417257748e
msg194834 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-08-10 21:08
Less than two days passed since this issue was opened. No waiting for feedback? No patch? No code review?

Maybe there's another solution? Personally I find dir() often more useful than help(). Maybe help() can be adapted to behave nicely for classes with custom __dir__ and it could be useful elsewhere?

Is this such a burning immediate need that we forego all the usual discussion channels?
msg194864 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-08-11 05:57
Sorry, sorry.

Long week, felt like more than two days, mild sense of unease and stress due to non-functioning Release Schedule on python.org, and wanting to get Enum used /somewhere/ in the stdlib before 3.4 is locked down.

Making help() better would be a better solution, especially if we enhanced it to show docstrings on instances.

I'll see what I can figure out.  Feel free to beat me to it.  :)
msg194872 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-08-11 08:28
Ethan, http://www.python.org/dev/peps/pep-0429/#release-schedule
TL;DR - no new features after beta 1 (2013-11-24), no non-release-critical bug fixes after rc1 (2014-01-19)
msg194894 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-08-11 12:42
Ethan, as Ned said (and I think you got this answer in the list before), the real feature cutoff is Beta 1. So we have time until the end of November. Note that even new PEPs (like the statistics one) can go in before that. Even after beta, things that appear to be bugs (like this issue) can usually be fixed before the RCs come. So there is no time pressure whatsoever.
msg194916 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-08-12 00:05
So what do we want Enum's __dir__ to report?

Normally we see things like __eq__, __dict__, __getnewargs__, etc.

For IntEnum there would be __abs__, __floor__, __div__, etc.

Do we want to worry about those kinds of differences?  I think we do.

And if we do, then we are looking at removing items to make our custom __dir__, and with each release we would have to revisit the blacklist of items we don't want (the tests would catch that for us, but it would still be effort to update the code).

What if we took what object.__dir__ gave us, then added the Enum members while removing the private, er, non-public data structures?

In other words, this dir in EnumMeta:

    def __dir__(cls):
        items = set(super().__dir__())
        disgard = set([m for m in items if _is_sunder(m)])
        members = set(cls.__members__)
        return sorted((items | members) ^ disgard)

with this Enum:

>>> class Color(enum.Enum):
...  RED = 1
...  BLUE = 2
...  GREEN = 3
... 

gives us this result:

>>> dir(Color)
['BLUE', 'GREEN', 'RED', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'name', 'value']
msg194928 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-08-12 06:11
Huh.  I just checked `help(Color)` on my proposed __dir__ and got the two-line, rather useless, response.  Perhaps help() is broken with all custom __dir__s.
msg194951 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-08-12 12:47
Ethan, please revert your commit first. I liked the previous dir. The current one is useless.

I think you may be right about help, but I didn't dig deep enough to be sure.
msg194961 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-08-12 13:51
New changeset 39697dcd97e3 by Ethan Furman in branch 'default':
Issue 18693: Put custom __dir__ back in place.  Will instead look at fixing `help()`.
http://hg.python.org/cpython/rev/39697dcd97e3
msg196757 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-09-01 23:51
Do we have enough evidence to open a new bug vs. help() ?
msg196760 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-09-02 00:29
I've done some more investigation today but I can't tell if the issue is solely in help or if it's some wierd 
interaction between help and _EnumMeta.
msg196762 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-09-02 00:39
On Sun, Sep 1, 2013 at 5:29 PM, Ethan Furman <report@bugs.python.org> wrote:

>
> Ethan Furman added the comment:
>
> I've done some more investigation today but I can't tell if the issue is
> solely in help or if it's some wierd
> interaction between help and _EnumMeta.
>

So you can't reproduce it with other classes that have a custom __dir__ ?
msg196763 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-09-02 01:04
Nope, not yet.  And even a simple dummy metaclass with a custom __dir__ worked fine.
msg196764 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-09-02 01:22
What I know for sure:

   1) if something is added to dir() that does not live in __dict__, help() breaks.

   2) if a certain something or some things are removed from dir(), help() breaks.  I'm not yet certain which somethings 
apply here.
msg196781 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-09-02 12:34
That help() is confused by __dir__ is documented in #16938. 

AFAIK help is fairly fragile in its expectations of the attributes present on classes and the correspondence between dir(cls) and list(cls.__dict__), and that is something that could be fixed in pydoc and/or inspect.
msg196913 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-09-04 13:39
Found it!

It was a combination of __objclass__ not being defined on the enum mmebers, and the metatype not being searched in the __mro__ by inspect.  Thanks, Ronald, for the necessary clues.

Patch attached.

I'm not sure if I have the method wowser showing up in the correct dir().  Thoughts?
msg196917 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-09-04 13:51
Great, Ethan.

I'd say the inspect fix has to be reviewed and committed separately. Maybe #16938 is the right place to post the patch for it. Once that's in, we can review/commit the enum parts.
msg196924 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-09-04 15:08
help() won't really be fixed with the inspect patch.  If no objections within a few hours I'll open a new issue for it.
msg197743 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-09-15 02:38
Okay, here's the patch after the inspect portion was committed in #18929.

Summary:
  - added __objclass__ to each enum member
  - added __module__ to the class dir
  - aded any extra methods defined to the instance dir
  - changed _RouteClassAttributeToGetattr to use the wrapped doc if it exists
  - added more useful __doc__s to value and name
  - added another test :)
msg197842 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-09-15 23:51
Two issues still remain:

  - custom behavior, as well as value and name, don't show in help
  - value and name, if defined as enum members, show up as data 
    descriptors in help

=======================================================================================
--> class Test(enum.Enum):
...   this = 'that'
...   these = 'those'
...   whose = 'mine'
...   name = 'Python'
...   value = 'awesome'
...   def what(self):
...     return "%s is %s!" % (self.name, self.value)
... 

--> dir(Test)
['__class__', '__doc__', '__members__', '__module__', 'name', 'these', 'this', 'value', 'whose']

--> dir(Test.this)
['__class__', '__doc__', '__module__', 'name', 'value', 'what']


--> help(Test)
Help on Test in module __main__ object:

class Test(enum.Enum)
 |  Method resolution order:
 |      Test
 |      enum.Enum
 |      builtins.object
 |  
 |  Data and other attributes defined here:
 |  
 |  these = <Test.these: 'those'>
 |  
 |  this = <Test.this: 'that'>
 |  
 |  whose = <Test.whose: 'mine'>
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from enum.Enum:
 |  
 |  name
 |      The name of the Enum member.
 |  
 |  value
 |      The value of the Enum member.
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors inherited from enum.EnumMeta:
 |  
 |  __members__
 |      Returns a mapping of member name->value.
 |      
 |      This mapping lists all enum members, including aliases. Note that this
 |      is a read-only view of the internal mapping.
(END)
=======================================================================================

At this point, dir() on an Enum member shows what can be done with the member, and dir() on an Enum class shows what can be done with the class.

I'll create new issues to track changes for inspect and help.
msg197843 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-09-16 00:00
New changeset 353ced6ae182 by Ethan Furman in branch 'default':
Close #18693: Enum is now more help() friendly.
http://hg.python.org/cpython/rev/353ced6ae182
msg197846 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-09-16 00:03
Tracking inspect in issue19030.

Tracking help in issue19031.
History
Date User Action Args
2013-09-16 00:03:39ethan.furmansetmessages: + msg197846
2013-09-16 00:00:20python-devsetstatus: open -> closed
resolution: fixed
messages: + msg197843

stage: patch review -> resolved
2013-09-15 23:51:49ethan.furmansetmessages: + msg197842
2013-09-15 02:39:00ethan.furmansetfiles: + issue18693.stoneleaf.02.patch
resolution: fixed -> (no value)
messages: + msg197743
2013-09-04 15:08:53ethan.furmansetmessages: + msg196924
2013-09-04 13:51:39eli.benderskysetmessages: + msg196917
2013-09-04 13:40:00ethan.furmansetfiles: + issue18693.stoneleaf.01.patch
keywords: + patch
messages: + msg196913

stage: resolved -> patch review
2013-09-02 12:34:51ronaldoussorensetmessages: + msg196781
2013-09-02 01:22:26ethan.furmansetmessages: + msg196764
2013-09-02 01:04:02ethan.furmansetmessages: + msg196763
2013-09-02 00:39:55eli.benderskysetmessages: + msg196762
2013-09-02 00:29:05ethan.furmansetmessages: + msg196760
2013-09-01 23:51:19eli.benderskysetmessages: + msg196757
2013-08-12 13:51:48python-devsetmessages: + msg194961
2013-08-12 12:51:27ronaldoussorensetnosy: + ronaldoussoren
2013-08-12 12:47:59eli.benderskysetmessages: + msg194951
2013-08-12 06:11:17ethan.furmansetmessages: + msg194928
2013-08-12 00:05:23ethan.furmansetmessages: + msg194916
2013-08-11 12:42:00eli.benderskysetmessages: + msg194894
2013-08-11 08:28:50ned.deilysetnosy: + ned.deily
messages: + msg194872
2013-08-11 05:57:32ethan.furmansetstatus: closed -> open

messages: + msg194864
2013-08-10 21:08:13eli.benderskysetmessages: + msg194834
2013-08-10 20:07:57python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg194830

resolution: fixed
stage: resolved
2013-08-09 05:20:21ethan.furmansetmessages: + msg194717
2013-08-09 01:59:52ethan.furmancreate