classification
Title: help() on enum34 enumeration class creates only a dummy documentation
Type: behavior Stage: test needed
Components: Library (Lib) Versions: Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: ethan.furman Nosy List: andymaier, ethan.furman, terry.reedy
Priority: normal Keywords: patch

Created on 2014-05-23 17:27 by andymaier, last changed 2014-06-17 19:12 by andymaier.

Files
File name Uploaded Description Edit
bug1.py andymaier, 2014-05-23 17:27 Source file that produces the bug, using enum34
pydoc_fix2.py andymaier, 2014-06-06 17:58 fixed version of pydoc.py for Python 2.7.6; still has some debug prints; both fixes and debug code are marked with @AM@
bug2.py andymaier, 2014-06-06 18:06 Simple standalone source file that produces the bug
pydoc.py.patch andymaier, 2014-06-17 19:10 Patch for Lib/pydoc.py, relative to the tip of 2.7. review
test_pydoc.py.patch andymaier, 2014-06-17 19:12 Patch for Lib/test/test_pydoc.py, relative to the tip of 2.7. review
Messages (12)
msg218979 - (view) Author: Andy Maier (andymaier) * Date: 2014-05-23 17:27
Using the enum34 backport of enums, the help() function on an enum class Colors displays only:


-------
Help on class Colors in module __main__:

Colors = <enum 'Colors'>
-------

Source code to reproduce:
-------
from enum import Enum # enum34 package

class Colors(Enum):
    """docstring for enumeration Colors"""
    RED = 1
    GREEN = "2"
    BLUE = [3]

help(Colors)
-------

Versions used:
  python 2.7.6
  enum34 1.0
Platform: Windows 7

I debugged the issue, found the place where it breaks down, and made a fix. However, it may well be that the fix is just a cure to a symptom, and that a better fix is possible.

Here is the fix:
In pydoc.py, class TextDoc, method docclass():
Change this code on line 1220+:
-------
            if thisclass is __builtin__.object:
                attrs = inherited
                continue
            elif thisclass is object:
                tag = "defined here"
            else:
                tag = "inherited from %s" % classname(thisclass,
                                                      object.__module__)
-------
to this, by adding the two lines marked with "added":
-------
            if thisclass is __builtin__.object:
                attrs = inherited
                continue
            elif thisclass is object:
                tag = "defined here"
            elif thisclass is None:            # <-- added
                tag = "inherited from TBD"     # <-- added
            else:
                tag = "inherited from %s" % classname(thisclass,
                                                      object.__module__)
-------

It breaks down during the last round through the 'while attrs' loop, where thisclass is None. I did not investigate why thisclass is None.

Without the fix, this causes an AttributeError to be raised by the classname() function, which then further on causes the dummy documentation to be generated.

With the fix, the help(Colors) output becomes:

-------
Help on class Colors in module __main__:

class Colors(enum.Enum)
 |  docstring for enumeration Colors
 |
 |  Method resolution order:
 |      Colors
 |      enum.Enum
 |      __builtin__.object
 |
 |  Data and other attributes defined here:
 |
 |  BLUE = <Colors.BLUE: [3]>
 |
 |  GREEN = <Colors.GREEN: '2'>
 |
 |  RED = <Colors.RED: 1>
 |
 |  ----------------------------------------------------------------------
 |  Data and other attributes inherited from TBD:
 |
 |  __members__ = {'BLUE': <Colors.BLUE: [3]>, 'GREEN': <Colors.GREEN: '2'...
-------
msg218982 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2014-05-23 18:20
Good work.

This bug was fixed in 3.4 with the inclusion of enum.

It would definitely be good to fix in 2.7 as well.
msg218990 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-05-23 19:19
If the problem reported here applies only to the 2.7 backport of enum, which is not part of the Python standard library, shouldn't this issue be closed?
msg218993 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2014-05-23 19:35
The problem will affect anything that uses the same mechanism as enum.  It also affects (not verified) all versions of python up to 3.4 where it was fixed because enum exposed it.

Besides which, I did not think a bug had to affect stdlib code in order to be fixed -- or this just a 2.7 restriction since it's basically end-of-lifed?
msg218994 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-05-23 19:38
Sorry, I skimmed over the issue and didn't notice that the fix applied to pydoc.py, not enum.
msg219088 - (view) Author: Andy Maier (andymaier) * Date: 2014-05-25 13:25
The pydoc.py of Python 3.4 that supposedly has been fixed has a lot of changes compared to 2.7, but the place where I applied my "fix" in TextDoc.docclass() is unchanged.

So it seems that my fix should be regarded only to be a quick fix, and the real fix would be somewhere in the 3.4 pydoc.py. I tried to understand the changes but gave up after a while. My quick fix (with a better text than one that contains "TBD") is still better than not having it fixed, but more ideally the real fix should be rolled back to the 2.7 pydoc.py.

Is there anything else I can do to help with this bug?
msg219404 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-05-30 16:57
Devise a simple test (fail before, work after) that does not require enum34. If this fix is committed, the message could note that the same issue was fixed differently in 3.4 mixed in with other changes.
msg219599 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2014-06-02 16:05
I think something like the following, taken from http://bugs.python.org/issue19030#msg199920, shoud do the trick for something to test against:

    class Meta(type):
        def __getattr__(self, name):
            if name == 'ham':
                return 'spam'
            return super().__getattr__(name)

    class VA(metaclass=Meta):
        @types.DynamicClassAttribute
        def ham(self):
            return 'eggs'

which should result in:

    VA_instance = VA()
    VA_instance.ham  # should be 'eggs'
    VA.ham           # should be 'spam'

Combining all that with the DynamicClassAttribute should make a fitting test.  Thanks, Andreas, for working on that.
msg219894 - (view) Author: Andy Maier (andymaier) * Date: 2014-06-06 17:58
Using Ethan's sample code (Thanks!!), I was pointed in the right direction and was able to produce a simple piece of code that reproduces the behavior without depending on enum34, as well as a proposal for a fix in pydoc.py.

The problem can be reproduced with a class that specifies a metaclass that has class attributes and includes these class attributes in its __dir__() result, which causes them to "propagate" as class attributes to the class using that metaclass, at least for the purposes of pydoc.

In the course of the processing within pydoc.py, the tuple returned by inspect.classify_class_attrs() for such class attributes then has None for its class item, which triggers the issue I originally reported (pydoc traps and produces just a dummy help description).

In my original problem with the enum34 module, the same thing happens, just here the "propagated" class attributes include the __members__ list, which is defined as a property. So I do believe that my simple code represents the same error situation as the original enum34 issue.

Because pydoc.py already has its own classify_class_attrs() function that wrappers inspect.classify_class_attrs() for the purpose of treating data descriptors correctly, I think it would be acceptable to continue down the path of fixing it up, just this case for class attributes propagated by the metaclass. That's what my proposed fix does.

Unfortunately, it seems one can attach only one file, so I paste the reproduction code in here, and attach the fixed pydoc.py.

Here is the code that reproduces the issue:
---------------- bug2.py
# Boolean test switches that control whether a class variable of the metaclass
# is added to the dir() result.
# If the fix is not present, then enabling each one triggers the error
# behavior; If none of them is enabled, the error behavior is not triggered.
with_food = True      # Add the 'normal' class attribute 'food'
with_drink = True     # Add the property-based class attribute 'drink'

class FoodMeta(type):
    """Metaclass that adds its class attributes to dir() of classes using it."""

    food = 'ham'      # 'normal' class attribute
    
    @property
    def drink(cls):   # property-based class attribute
        return 'beer'

    def __dir__(cls):
        ret = [name for name in cls.__dict__] # the normal list
        if with_food:
            ret += ['food']
        if with_drink:
            ret += ['drink']
        print "bug2.FoodMeta.__dir__(): return=%s" % (repr(ret),)
        return ret

class Food(object):
    """docstring for Food class"""
    __metaclass__ = FoodMeta

    def diet(self):
        return "no!"

if __name__ == '__main__':
    print "bug2: Calling help(Food):"
    help(Food)

----------------

-> Please review the reproduction code and the fix and let me know how to proceed.

Andy
msg219895 - (view) Author: Andy Maier (andymaier) * Date: 2014-06-06 18:06
Here is the bug2.py file pasted into the previous message, for convenience.
msg220861 - (view) Author: Andy Maier (andymaier) * Date: 2014-06-17 19:10
Attaching the patch for pydoc.py, relative to the tip of 2.7. the patch contains just the proposed fix, and no longer the debug prints.
msg220862 - (view) Author: Andy Maier (andymaier) * Date: 2014-06-17 19:12
Attaching the patch for Lib/test/test_pydoc.py, relative to the tip of 2.7. The patch adds a testcase test_class_with_metaclass(), which defines a class that provokes the buggy behavior, and verifies the fix.
History
Date User Action Args
2014-06-17 19:12:26andymaiersetfiles: + test_pydoc.py.patch

messages: + msg220862
2014-06-17 19:10:19andymaiersetfiles: + pydoc.py.patch
keywords: + patch
messages: + msg220861
2014-06-06 18:06:32andymaiersetfiles: + bug2.py

messages: + msg219895
2014-06-06 17:58:16andymaiersetfiles: + pydoc_fix2.py

messages: + msg219894
2014-06-02 22:04:53ned.deilysetnosy: - ned.deily
2014-06-02 16:05:07ethan.furmansetassignee: ethan.furman
messages: + msg219599
2014-05-30 16:58:00terry.reedysetnosy: + terry.reedy

messages: + msg219404
stage: test needed
2014-05-25 13:25:07andymaiersetmessages: + msg219088
2014-05-23 19:38:24ned.deilysetnosy: ned.deily, ethan.furman, andymaier
messages: + msg218994
2014-05-23 19:35:40ethan.furmansetmessages: + msg218993
2014-05-23 19:19:18ned.deilysetnosy: + ned.deily
messages: + msg218990
2014-05-23 18:20:43ethan.furmansetnosy: + ethan.furman
messages: + msg218982
2014-05-23 17:27:11andymaiercreate