On multiple occasions I have wanted to add documentation not only to Python classes and functions, but also instance variables. This seems to involve (at least) two orthogonal questions:

1) what is the proper syntax to associate documentation (docstrings ?) to objects ?
2) what changes need to be applied to Python's infrastructure (e.g., the help system) to support it ?


I have attempted to work around 1) in my custom code by explicitly setting an object's `__doc__` attribute. However, calling `help()` on such an object would simply ignore that attribute, and instead list the documentation associated with the instance type.

Am I missing something here, i.e. am I approaching the problem the wrong way, or am I the first to want to use object-specific documentation ?

Minor note on terminology: classes and functions are themselves objects.

I think that help() (or in particular PyDoc in general) should support any instance with a __doc__ attribute. Its failure to do so is causing pain, see #12154.

Exactly ! I'm fully aware of the ubiquity of objects in Python, and it is for that reason that I had naively expected `pydoc` to simply DoTheRightThing when encountering an object containing a `__doc__` attribute. rather than only working for types and function objects.

OK, assuming that this is a recognized bug / limitation, it seems easy to address.

Is there any discussion concerning what syntax might be used for docstrings associated with objects ? (There seem to be some partial solutions added on top of the Python parser (I think `epydoc` offered one), but it would be nice to have a built-in solution to avoid having to re-invent wheels.

> Is there any discussion concerning what syntax might be used for 
> docstrings associated with objects ?

I don't know about PyDoc in general, but I would expect help(obj) to 
just use obj.__doc__ which will return the instance docstring if it 
exists, and if not, the type docstring (if it exists). No new syntax is 
required, the standard ``help(obj)`` is sufficient.

> (There seem to be some partial 
> solutions added on top of the Python parser (I think `epydoc` offered 
> one), but it would be nice to have a built-in solution to avoid having 
> to re-invent wheels.

Are you suggesting we need new syntax to automatically assign docstrings 
to instances? I don't think we do.

I expect that if you want to set a custom instance docstring, you would 
just say ``instance.__doc__ = "The doc string"`` after creating the 
instance, or ``self.__doc__ = "..."`` inside the __init__ method.

On 2018-12-09 18:35, Steven D'Aprano wrote:
> Steven D'Aprano <steve+python@pearwood.info> added the comment:
>
>> Is there any discussion concerning what syntax might be used for 
>> docstrings associated with objects ?
> I don't know about PyDoc in general, but I would expect help(obj) to 
> just use obj.__doc__ which will return the instance docstring if it 
> exists, and if not, the type docstring (if it exists). No new syntax is 
> required, the standard ``help(obj)`` is sufficient.

That's why I distinguished between points 1) and 2) in my original mail:
The syntax is about how certain tokens in the parse tree are associated
as "docstring" with a given object (i.e., point 1), while the pydoc's
behaviour (to either accept any `__doc__` attributes, or only those of
specific types of objects) is entirely orthogonal to that (thus point 2).

I now understand that the current `pydoc` behaviour is considered
erroneous, and it sounds like a fix would be simple and focused in scope.

>> (There seem to be some partial 
>> solutions added on top of the Python parser (I think `epydoc` offered 
>> one), but it would be nice to have a built-in solution to avoid having 
>> to re-invent wheels.
> Are you suggesting we need new syntax to automatically assign docstrings 
> to instances? I don't think we do.

No, I'm not suggesting that. I'm suggesting that within the current
syntax, some additional semantic rules might be required to bind
comments (or strings) to objects as "docstrings". For example:

```

foo = 123

"""This is foo's docstring"""

```

might be one convention to add a docstring to a variable.

```

foo = 123

# This is foo's docstring

```

might be another.

None of this is syntactically new, but the construction of the AST from
the parse tree is. (I have seen both of these conventions used in custom
tools to associate documentation to variables, which of course requires
hacking into the parser internals, to add the given docstring to the
object's `__doc__` attribute.

It would be great to establish a convention for this, so in the future
tools don't have to invent their own (non-portable) convention.

There was a related proposal in https://www.python.org/dev/peps/pep-0258/#attribute-docstrings

On 2018-12-09 19:48, Karthikeyan Singaravelan wrote:
> There was a related proposal in https://www.python.org/dev/peps/pep-0258/#attribute-docstrings

Right, but that was rejected (for unrelated reasons). The idea itself
was rejected by Guido
(https://www.python.org/dev/peps/pep-0224/#comments-from-our-bdfl), and
I'm not aware whether anyone has addressed his concerns by proposing a
different syntax.

It's sad, as right now there doesn't appear to be any way to address
this need...

Stefan

-- 

      ...ich hab' noch einen Koffer in Berlin...

I asked:
> > Are you suggesting we need new syntax to automatically assign docstrings 
> > to instances?

Stefan replied:
> No, I'm not suggesting that.

And then immediately went on to suggest new syntax for automatically 
binding a string to objects as docstrings. I am amused :-)

Whether you want to call it "new semantics for existing syntax" or "new 
syntax" is a matter of terminology. The point is, you are suggesting 
something that requires dedicated support from the interpreter, as 
opposed to merely writing some Python code.

> I'm suggesting that within the current
> syntax, some additional semantic rules might be required to bind
> comments (or strings) to objects as "docstrings".

To my mind, it makes sense to have dedicated docstring syntax for 
classes, modules and functions: I expect that they will make up in 
excess of 95% of use-cases for docstrings. In this case, special cases 
*are* special enough to change the rules.

But for the rare(?) cases of wanting to add docstrings to arbitrary 
instances, I don't think it is justified to have dedicated syntax to do 
it. It's easy enough and more than flexible enough to just do an 
instance attribute assignment:

    instance.__doc__ = """This is the instance docstring."""

I strongly oppose any suggestion that *comments* be treated as code:

> foo = 123
> # This is foo's docstring

Deleting comments shouldn't make have any runtime effect on the code, 
but in this case it would. (I'm willing to make an exception for the 
optional encoding cookie at the beginning of modules, as a rather 
special case.)

So I think there are three related but separate issues here:

1. help(obj) and possibly PyDoc in general ought to support per-instance 
docstrings. I think that is uncontroversial and we just need somebody to 
do the work to make it happen.

2. Add special syntactic sugar to automatically associate a string with 
arbitrary instances as a docstring. I think that's overkill and 
unnecessary, but I'd be willing to be convinced otherwise.

3. Your examples suggest that even built-in immutable objects like ints 
should be able to take docstrings. I don't think that idea is going to 
fly, but I'm also not sure how serious you are about that. It runs into 
the problem that small ints and certain strings are cached, so your 
docstring could clobber my docstring. It is also going to require the 
addition of an extra attribute slot to every int, str, etc for the 
docstring, even if 99.999% of them never use it.

> It would be great to establish a convention for this, so in the future
> tools don't have to invent their own (non-portable) convention.

``instance.__doc__ = "docstring"`` seems pretty portable to me :-)

ad 3) sorry, I picked a bad example - I didn't mean to suggest that immutable objects should in fact become mutable by modifying their `__doc__` attribute.

ad 1) good, glad to hear that.

ad 2) fine. In fact, I'm not even proposing that per-instance docstring generation should be "on" by default. I'm merely asking whether the Python community can't (or even shouldn't) agree on a single convention for how to represent them, such that special tools can then support them, rather than different tools supporting different syntax / conventions.

The reason that modules, classes, and functions need a special rule for assigning the .__doc__ attribute is that one cannot get a reference to the module, class, or function within the body of its definition.  And putting the docstring at the top of a file or after a header is usually the best place.

For modules, the body is the file, leaving nowhere else to put the docstring.  For classes and functions, the alternative of an assignment elsewhere, after the object is created, remains.

>>> def f(): pass

>>> f.__doc__ = 'Docstring outside f body'
>>> help(f)
Help on function f in module __main__:

f()
    Docstring outside f body

This alternative is used in functools.partial and decorators that wrap functions with a function and then copy the original docstring to the wrapper.

I think object.doc = 'docstring' is sufficient for other objects and that PEP 224 was wrong to propose otherwise and should have been rejected.

So I think that this issue should propose what Steven said:  pydoc/help() should be simplified to fetch object.__doc__ with normal lookup, instead of bypassing object if not one of the special types.  Stefan Seefeld, can you try patching pydoc to do this?

If one wants to add docstrings to builtins, a subclass can work.

>>> class Docint(int): pass

>>> i,j = Docint(1), Docint(2)
>>> i+j
3
>>> i.__doc__ = 'one'

We just need help to print i.__doc__ instead of int.__doc__.