This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Set docstrings to empty string when optimizing with -OO.
Type: enhancement Stage:
Components: Interpreter Core Versions: Python 3.5
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: barry, erik.bray, jvs, rhettinger
Priority: normal Keywords:

Created on 2015-01-08 09:41 by jvs, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (7)
msg233634 - (view) Author: Jarle Selvåg (jvs) Date: 2015-01-08 09:41
Python code byte-compiled with -OO has doc-strings stripped out. 

This creates problems when compiling different packages which changes the doc-strings by doing something like this:
     __doc__ += "additional text"
(when the docstring is 'None', this will fail).

The packages "lmfit 0.8.1" and "Patsy 0.3.0" have this problem, and must be patched before compilation.

See related discussion on Stackoverflow:
http://stackoverflow.com/questions/22299532/unsupported-operand-types-for-nonetype-and-str-winappdbg-error-after-c

Proposal: Set the doc-strings to empty string ("") instead of removing them completely during optimization with -OO. The memory footprint would anyway be the same.
msg233636 - (view) Author: Jarle Selvåg (jvs) Date: 2015-01-08 09:57
This issue is only relevant for classes that have this construct:

class MyClass(object):
    __doc__ += '''Some more text'''
    ....
msg233666 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2015-01-08 15:01
That's a backward compatibility break since existing code may be expecting None.  At least it needs to be carefully considered, and should have no possibility of be applied to anything before Python 3.5.
msg233731 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-01-09 08:36
It seems to me that the problem here lies with the packages that use __doc__+="sometext" rather than with -OO which is doing exactly what it is supposed to.
msg233885 - (view) Author: Jarle Selvåg (jvs) Date: 2015-01-12 15:17
I agree that -OO does what (people have agreed) it's supposed to do.

Many packages manipulates the docstring without checking for 'None' (see list below). For many package developers, it seems hard to remember that the docstrings may disappear after optimization. This behavior is not intuitive. At least in cases where the docstring is set to a string value, the expected behavior in my opinion would be to set its value to an empty string during optimization with -OO.

Here are some of the packages that run into trouble when the docstrings are set to 'None' by the -OO optimization:
algopy
astropy
lmfit
pyamg
pyvisa
mpl_toolkits
sympy
statsmodels
patsy
sklearn
msg234007 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-01-14 05:58
> Here are some of the packages that run into trouble
> when the docstrings are set to 'None' by the -OO optimization

I think you should file bug reports for those packages.

We've agreed here that the current behavior is correct and that the proposed change in backward incompatible, so I'm gone to close this one as "not a bug".   Perhaps, some not in the docs for "packaging and distributing python would be appropriate" and perhaps some PEP-8 guidance for avoiding "__doc__+='moredocs'" without first checking whether __doc__ is None.
msg246470 - (view) Author: Erik Bray (erik.bray) * (Python triager) Date: 2015-07-08 20:12
For what it's worth (since it was mentioned toward the end of this thread as an offending package) Astropy has a patch now to fix its misbehavior with respect to this issue.

I do feel like this would be nice to fix though, since I think it's awkward to have code that on its face clearly has a docstring, but still have to check that the docstring may be None.  Just to put it out there if anyone were really interested, I identified two ways this could be changed:

1) For functions add a new co_flags flag indicating that docstrings were optimized out.  This is at the cost of a co_flags flag.

2) Keep an empty string as the first element in co_consts, at the const of one additional constant per function that previously had a docstring (still cheaper than the docstring itself in most cases).

Of course, it would still break backward compatibility for code that expects None for an optimized out docstring, I guess, and probably isn't worth either of the above two sacrifices at the end of the day.
History
Date User Action Args
2022-04-11 14:58:11adminsetgithub: 67378
2015-07-08 20:12:32erik.braysetnosy: + erik.bray
messages: + msg246470
2015-01-14 05:58:23rhettingersetstatus: open -> closed
resolution: not a bug
messages: + msg234007
2015-01-12 15:17:46jvssetmessages: + msg233885
2015-01-09 08:36:27rhettingersetnosy: + rhettinger
messages: + msg233731
2015-01-08 15:01:22barrysetnosy: + barry

messages: + msg233666
versions: - Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.6
2015-01-08 09:57:50jvssetmessages: + msg233636
2015-01-08 09:41:45jvscreate