classification
Title: Enhance inspect.getdoc to follow inheritance chains
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Claudiu.Popa, berker.peksag, eric.snow, martin.panter, merwok, ncoghlan, python-dev, serhiy.storchaka, yselivanov
Priority: normal Keywords: patch

Created on 2012-08-08 02:52 by ncoghlan, last changed 2015-07-30 15:12 by berker.peksag. This issue is now closed.

Files
File name Uploaded Description Edit
inspect_getdoc.patch Claudiu.Popa, 2013-12-13 10:20 review
issue15582.patch Claudiu.Popa, 2014-03-16 20:04 Add documentation. review
issue15582_1.patch Claudiu.Popa, 2014-04-05 10:48 Ups, remove unintented change. review
inspect_getdoc_inherited.patch serhiy.storchaka, 2015-03-09 22:24 review
getdoc-news.patch martin.panter, 2015-05-15 06:30 review
getdoc-news.v2.patch martin.panter, 2015-07-30 01:04
Messages (26)
msg167654 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-08-08 02:52
Currently, if you override a method from a base class, the docstring is not inherited, even if it remains accurate.

This issue proposes an enhancement to inspect.getdoc() that allows the docstring to be retrieved from the inheritance hierarchy in the case where it is not overridden in the subclass by providing an explicit docstring.

Specifically, in the case where obj.__doc__ is None, and either the first parameter is a bound method, or a class object is passed in as the second parameter, inspect.getdoc will search the MRO based on obj.__name__ until it finds an attribute with a non-None __doc__ value.

(In Python 2, this could have been automatic for both bound and unbound methods. Unfortunately, there are no unbound methods in Python 3, so the second parameter is needed to handle the unbound method case)
msg167909 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-08-10 19:31
I have just the thing for this, but haven't had a chance to put a patch together.  I might be able to get to it in the next week if Dave "bitey" Beazley isn't too much of a distraction.  :)
msg206045 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2013-12-13 10:20
Hello!

Here's a patch. Currently it lacks doc updates, but if the solution is okay, then I could provide them.
msg215593 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2014-04-05 10:48
Yury, Nick, how is my latest patch?
msg232322 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2014-12-08 19:45
@Claudiu, you should also add this test and make sure that it passes:

   class Parent:
      __doc__ = 'some documentation'

   class Child(Parent):
      __doc__ = None

   assert getdoc(Child) is None

In other words -- we use __doc__ defined in parent classes only when there was no __doc__ in children's __dict__s.
msg232323 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2014-12-08 19:49
Alright, I'll update my patch later this week.
msg234171 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2015-01-17 15:19
Yury, regarding your last message, is it actually possible to have a subclass which doesn't have a __doc__ attribute in its __dict__, except using slots? __doc__ seems to be set to None every time if it's not specified, so I don't know how could I detect the case where the client sets '__doc__ = None' himself.

The following example might be more explanatory:

>>> class A:
...    __doc__ = "a"
...
>>> inspect.getdoc(A)
'a'
>>> inspect.getdoc(A())
'a'
>>> class B(A):
...   __doc__ = None
...
>>> vars(B)
mappingproxy({'__doc__': None, '__module__': '__main__'})
>>> B.__dict__
mappingproxy({'__doc__': None, '__module__': '__main__'})
>>> class C(A): pass
...
>>> vars(C)
mappingproxy({'__doc__': None, '__module__': '__main__'})
>>>

Nevertheless, my patch ignores this case, since it operates only on methods. When trying to do inspect.getdoc(Child, parent=Parent), it will try to look for an attribute 'Child' in the mro of Parent and thus it will return None, since this doesn't exist (this can actually be a problem, if that attribute actually exist).
msg237691 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-09 18:56
The patch adds support only for unbound methods and requires additional parameter for this. It is not what should be done in this issue at all.

I'm interested in this issue so I'll write a patch. It should be easier to write it myself than describe it.
msg237694 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2015-03-09 19:06
> The patch adds support only for unbound methods and requires additional parameter for this.

Just curios, which part works only for unbound methods?

> It is not what should be done in this issue at all.
> It should be easier to write it myself than describe it.

Well, some hints for the right way to do this would have been appreciated.
msg237716 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-09 22:24
Here is a patch that supports classes, bound and unbound methods, class methods, properties, method and class descriptors.
msg237760 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-03-10 12:37
Serhiy's patch looks good to me.

I'd completely missed that __qualname__ could be used to avoid needing a second argument even when handling objects other than bound methods. That's very cool, and I can see why you figured it was easier to just write the patch than explain what you had in mind.
msg240027 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-03 19:39
New changeset 157b7055bb9d by Serhiy Storchaka in branch 'default':
Issue #15582: inspect.getdoc() now follows inheritance chains.
https://hg.python.org/cpython/rev/157b7055bb9d
msg240041 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-03 21:58
Unfortunately this enhancement breaks test_enum.

======================================================================
FAIL: test_pydoc (test.test_enum.TestStdLib)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_enum.py", line 1609, in test_pydoc
    self.assertEqual(result, expected_text)
AssertionError: 'Help[68 chars]n |  Generic enumeration.\n |  \n |  Derive fr[876 chars]ing.' != 'Help[68 chars]n |  Method resolution order:\n |      Color\n[782 chars]ing.'
  Help on class Color in module test.test_enum:
  
  class Color(enum.Enum)
-  |  Generic enumeration.
-  |  
-  |  Derive from this class to define new enumerations.
-  |  
   |  Method resolution order:
   |      Color
   |      enum.Enum
   |      builtins.object
   |  
   |  Data and other attributes defined here:
   |  
   |  blue = <Color.blue: 3>
   |  
   |  green = <Color.green: 2>
   |  
   |  red = <Color.red: 1>
   |  
   |  ----------------------------------------------------------------------
   |  Data descriptors inherited from enum.Enum:
   |  
   |  name
   |      The name of the Enum member.
   |  
   |  value
   |      The value of the Enum member.
   |  
   |  ----------------------------------------------------------------------
   |  Data descriptors inherited from enum.EnumMeta:
   |  
   |  __members__
   |      Returns a mapping of member name->value.
   |      
   |      This mapping lists all enum members, including aliases. Note that this
   |      is a read-only view of the internal mapping.

----------------------------------------------------------------------
msg240057 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-04 09:49
New changeset 47a61a1c97b3 by Serhiy Storchaka in branch 'default':
Fixed test_enum for issue #15582.
https://hg.python.org/cpython/rev/47a61a1c97b3
msg240058 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-04 09:53
Changed test_enum to make buildbots green, but perhaps the docstring of Enum should be changed, because it now is used for all Enum subclasses that doesn't define a docstring explicitly. An alternative solution is to set __doc__ of Enum subclasses to an empty string if the docstring is not defined explicitly.
msg240349 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-04-09 16:43
I filed issue #23900 to consider the question of the default docstring for Enum subclasses.
msg241489 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-04-19 06:23
Sometimes the doc string for the overridden method does not make much sense in the context of the subclass. Just wondering if this was considered; it seems like a fairly serious downside to this new feature. E.g. in a package I am reviewing, there is a class that inherits HTMLParser and converts HTML to PDF. There is no doc string, so previously there was just the signature in the “pydoc” output. Now the output looks like:

 |  __init__(self, pdf, image_map=None)
 |      Initialize and reset this instance.
 |      
 |      If convert_charrefs is True (the default), all character references
 |      are automatically converted to the corresponding Unicode characters.

The second paragraph mentions parameters and settings of the internal base class, which doesn’t make much sense for the subclass.
msg241520 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2015-04-19 16:27
Sounds like good incentive to add good docstrings.  :)
msg241608 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-20 05:52
I were aware that this can propagate some not well appropriate docstrings from abstract base classes, but Martin expose worse problem: inheriting a docstring by the method with changed signature.

Perhaps we should check if a signature of overriding method is compatible with a signature of base method. But this is hard to implement, and impossible in some cases until all builtins will converted to Argument Clinic.
msg241726 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2015-04-21 17:26
I know it will take some time to add doc strings to places where they are missing, but I would not roll-back the patch for that.  File new issues and get the over-riding methods properly documented.
msg241753 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-04-21 23:14
My example was taken from a package that tends to “properly document” everything in separate documentation files, rather than in the source code. I don’t really think Python should dictate if and how one document their code based on what base classes they use, and pydoc is still useful with code that has no doc strings.

But if this is to be the way of the future, perhaps a warning in the “What’s New” page might be a good idea. The inspect.getdoc() documentation should probably also have a “Changed in version 3.5” warning.

I guess it would be less automatic, but maybe another option might have been to add an @inherit_docstring decorator, or some explicit way to say a particular API (re)implements an (abstract) API documented elsewhere.
msg242005 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-04-25 07:50
Yes, this was a deliberate change to "flip the default" as to which subclasses get bad docstring behaviour.

In the status quo, if you provide a subclass method which does basically the same thing as the parent class, you lose the docstring unless you duplicate it, meaning you have to choose between bad docstrings and a maintainability problem due to content duplication.

With this change, you only need a custom docstring in the subclass if you've changed the method behaviour enough that the parent class docstring no longer makes any sense.

If you just want to suppress the docstring entirely, then you'll need to specify an empty docstring.

This is potentially worth a note in the "Porting to Python 3.5" section of the What's New document, but it's an intended consequence of the change.
msg243248 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-05-15 06:30
getdoc-news.patch suggests some wording to add to What’s New, and also adds a “Changed in version 3.5” note to inspect.getdoc().

BTW I also noticed that the class doc strings are not inherited from object.__doc__, although method doc strings _are_ inherited from object(), such as object.__init__.__doc__. The current documentation suggests that the class doc string “The most base type” should also be inherited.

$ cat module.py
class SomeClass:
    '''CLASS DOCSTRING'''
    def __init__(self):
        '''METHOD DOCSTRING'''
$ ./python -m pydoc module.SomeClass  # Doc strings intact
[. . .]
module.SomeClass = class SomeClass(builtins.object)
 |  CLASS DOCSTRING
 |  
 |  Methods defined here:
 |  
 |  __init__(self)
 |      METHOD DOCSTRING
 |  [. . .]
$ ./python -OOm pydoc module.SomeClass  # Method inherited, class stripped
[. . .]
module.SomeClass = class SomeClass(builtins.object)
 |  Methods defined here:
 |  
 |  __init__(self)
 |      Initialize self.  See help(type(self)) for accurate signature.
 |  [. . .]

I also wonder how well this feature would work when someone tries to override a base method by using a mix-in type class.
msg247645 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-07-30 01:04
Merged the current What’s New page in getdoc-news.v2.patch. Is there any interest in applying this?
msg247680 - (view) Author: Roundup Robot (python-dev) Date: 2015-07-30 15:06
New changeset 4476b578b8fd by Berker Peksag in branch '3.5':
Issue #15582: Add a whatsnew entry for inspect.getdoc() changes in 3.5.
https://hg.python.org/cpython/rev/4476b578b8fd

New changeset e0f4a5f09bfa by Berker Peksag in branch 'default':
Issue #15582: Add a whatsnew entry for inspect.getdoc() changes in 3.5.
https://hg.python.org/cpython/rev/e0f4a5f09bfa
msg247681 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2015-07-30 15:12
Applied, thanks!
History
Date User Action Args
2015-07-30 15:12:21berker.peksagsetstatus: open -> closed

nosy: + berker.peksag
messages: + msg247681

stage: patch review -> resolved
2015-07-30 15:06:55python-devsetmessages: + msg247680
2015-07-30 01:04:25martin.pantersetfiles: + getdoc-news.v2.patch

messages: + msg247645
stage: resolved -> patch review
2015-07-21 07:12:28ethan.furmansetnosy: - ethan.furman
2015-05-15 06:30:38martin.pantersetfiles: + getdoc-news.patch

messages: + msg243248
2015-04-25 07:50:46ncoghlansetmessages: + msg242005
2015-04-21 23:14:20martin.pantersetmessages: + msg241753
2015-04-21 17:26:55ethan.furmansetmessages: + msg241726
2015-04-20 05:52:29serhiy.storchakasetmessages: + msg241608
2015-04-19 16:27:57ethan.furmansetmessages: + msg241520
2015-04-19 06:26:32serhiy.storchakasetstatus: closed -> open
2015-04-19 06:23:34martin.pantersetnosy: + martin.panter
messages: + msg241489
2015-04-09 17:40:12serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2015-04-09 16:43:59ncoghlansetmessages: + msg240349
2015-04-04 09:53:49serhiy.storchakasetmessages: + msg240058
2015-04-04 09:49:19python-devsetmessages: + msg240057
2015-04-03 21:58:28serhiy.storchakasetnosy: + ethan.furman
messages: + msg240041
2015-04-03 19:39:35python-devsetnosy: + python-dev
messages: + msg240027
2015-03-10 12:37:09ncoghlansetmessages: + msg237760
2015-03-09 22:24:34serhiy.storchakasetfiles: + inspect_getdoc_inherited.patch

messages: + msg237716
2015-03-09 19:06:56Claudiu.Popasetmessages: + msg237694
2015-03-09 18:56:53serhiy.storchakasetassignee: serhiy.storchaka
messages: + msg237691
2015-03-09 18:30:29serhiy.storchakasetnosy: + serhiy.storchaka
2015-01-17 15:19:58Claudiu.Popasetmessages: + msg234171
2014-12-08 19:49:38Claudiu.Popasetmessages: + msg232323
2014-12-08 19:45:38yselivanovsetnosy: + yselivanov
messages: + msg232322
2014-06-17 09:58:14Claudiu.Popasetstage: needs patch -> patch review
2014-04-05 10:48:04Claudiu.Popasetfiles: + issue15582_1.patch

messages: + msg215593
2014-03-16 20:04:11Claudiu.Popasetfiles: + issue15582.patch
2014-01-28 21:34:15yselivanovsetversions: + Python 3.5, - Python 3.4
2013-12-13 10:20:38Claudiu.Popasetfiles: + inspect_getdoc.patch

nosy: + Claudiu.Popa
messages: + msg206045

keywords: + patch
2012-08-10 19:31:17eric.snowsetmessages: + msg167909
2012-08-08 22:34:54merwoksetnosy: + merwok
2012-08-08 02:52:13ncoghlancreate