classification
Title: Remove COUNT_ALLOCS special build
Type: Stage: resolved
Components: Build Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: rhettinger, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2020-01-29 18:16 by vstinner, last changed 2020-02-03 14:21 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 18259 merged vstinner, 2020-01-29 18:22
Messages (12)
msg360978 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-29 18:16
Python has a COUNT_ALLOCS special build which adds sys.getcounts() function and shows statistics on Python types at exit if -X showalloccount command line option is used.

I never ever used this feature and I don't know anyone using it.

But "#ifdef COUNT_ALLOCS" code is scattered all around the code. It requires maintenance. I propose to remove the code to ease maintenance.

Attached PR shows how much code is requires to support this special build.

There are now more advanced tools to have similar features:

* tracemalloc can be used to track memory leaks
* gc.getobjects() can be called frequently to compute statistics on Python types
* There are many tools built around gc.getobjects()

The previous large change related to COUNT_ALLOCS was done in Python 3.6 by bpo-23034:

"The output of a special Python build with defined COUNT_ALLOCS, SHOW_ALLOC_COUNT or SHOW_TRACK_COUNT macros is now off by default. It can be re-enabled using the -X showalloccount option. It now outputs to stderr instead of stdout. (Contributed by Serhiy Storchaka in bpo-23034.)"

https://docs.python.org/dev/whatsnew/3.6.html#changes-in-python-command-behavior
msg360979 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-29 18:21
See also bpo-19527 which discussed COUNT_ALLOCS in 2013.

COUNT_ALLOCS build is documented in Fedora "DebugPythonStacks" wiki page:
https://fedoraproject.org/wiki/Features/DebugPythonStacks#Verify_COUNT_ALLOCS
msg360981 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-01-29 19:26
I have never used this feature either.
msg360985 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-29 21:29
As far as I known, the Fedora package of Python is the most known user of COUNT_ALLOCS special build, and maybe the only user.

The Fedora package of Python 2.7 builds two binaries:

* python2.7: release mode, optimized
* python2.7-debug: debug mode, not optimized, built with COUNT_ALLOCS defined

Sadly, in practice, python2.7-debug is basically unusable since C extensions providing by Fedora are not compatible: the release and the debug modes have a different ABI. I only fixed this recently in Python 3.8.

It uses COUNT_ALLOCS since the following patch (package version 2.6.5-10, in 2010) written by Dave Malcolm.


(1) There is a downstream patch adding --with-count-allocs flag to configure. Patch added to package version 2.6.5-10, in 2010:
https://src.fedoraproject.org/rpms/python2/c/ab11e4c10f6fef4e2e993ef446953df0f0dbb840


(2) Another downstream patch adds PYTHONDUMPCOUNTS environment variable to only dump statistics if the variable is set. Patch added to package version 2.7-8, in 2010:
https://src.fedoraproject.org/rpms/python2/c/5810c5d8b1c876ccc4c547cc71cf20520dd27d85

=> this patch was proposed upstream in bpo-19527. I merged it in Python 2.7.15 (in 2017), but with a different environment variable name: PYTHONSHOWALLOCCOUNT.


(3) Finally, a 3rd downstream patch fix test_abc when COUNT_ALLOCS is defined. Patch added to package version 2.7.1-1, in 2010:
https://src.fedoraproject.org/rpms/python2/c/4b97eebe22c8c5db58ae65cdc7e79c3ccd45b0a4

=> Bohuslav "Slavek" Kabrda created bpo-19527 to propose to make multiple test fixes upstream. Serhiy Storchaka modified the proposed patch by adding @test.support.requires_type_collecting decorator. Change merged in 3.5 and default branches, but not in 2.7.


Another example of Fedora contribution: bpo-31692 reports that test_regrtest fails when Python 2.7.14 is built with COUNT_ALLOCS.

---

Now, the interesting part.

IMHO COUNT_ALLOCS was basically kept between 2010 and 2015 because "it was there" and nobody asked if it still made sense to use it. It wasn't too expensive to maintain in the Fedora package... until someone asked if it's still worth it to maintain it in 2015.

The COUNT_ALLOCS macro has been removed from the Fedora package of Python 3 in 2015:

* https://bugzilla.redhat.com/show_bug.cgi?id=1291325
* "[cleanup] Remove COUNT_ALLOCS patches, see rhbz#1291325" commit:
  https://src.fedoraproject.org/rpms/python3/c/19aade22cbfa57dc500f5fa82229b4b9ed0b4c30

Extract of the bugzilla, "Reply from Dave Malcolm":
"""
My hazy recollection is that at the time I was dealing with lots of
memory leak issues in Python 2, so I was keen to add as much help as
possible in tracking them down to Python 2 and Python 3.

I don't think this patch ever really bought us much, and it sounds like
there are better tools for this now, so feel free to drop the
COUNT_ALLOC patches.
"""

It confirms what I wrote in the initial message of this issue: there are now better tool to track Python memory leaks.

Supporting COUNT_ALLOC in the Fedora package was motivated by Dave Malcolm use case of tracking memory leak, but even Dave wrote: "I don't think this patch ever really bought us much".
msg360986 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-29 21:31
Python 2.7.15 got a PYTHONSHOWALLOCCOUNT environment variable to dump statistics on types at exit, if Python is built with COUNT_ALLOCS macro defined.

Example with Fedora python2.7-debug (package python2-debug-2.7.17-1.fc31.x86_64):

$ PYTHONSHOWALLOCCOUNT=1 python2.7-debug -c pass
exceptions.ImportError alloc'd: 2, freed: 2, max in use: 1
symtable entry alloc'd: 3, freed: 3, max in use: 1
enumerate alloc'd: 2, freed: 2, max in use: 1
dict alloc'd: 459, freed: 220, max in use: 346
str alloc'd: 13981, freed: 12142, max in use: 5991
tuple alloc'd: 5088, freed: 4096, max in use: 2337
(...)
fast tuple allocs: 2434, empty: 1753
fast int allocs: pos: 875, neg: 74
null strings: 81, 1-strings: 328
msg360987 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-29 21:40
I looked for "Python COUNT_ALLOCS" on the Internet. This special build seems to be very badly documented on the Internet. Outside Python own documentation, I found almost zero reference to it. IMHO it's basically unused.


(*) bpo-33058: Two years ago (2018), attempt to make COUNT_ALLOCS build ABI compatible with release build. Extract:

> > Could tracemalloc help you?
> tracemalloc can't distinguish between the usage of gc allocs, normal mallocs, and free list reuse.


(*) Documentation of all Python builds:

https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-count-allocs-label
msg360988 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-29 21:43
The COUNT_ALLOCS feature itself is quite old. It was added 27 years ago (in 1993):

commit a9c3c22c33762699b362e7598268442fd2df9eb6
Author: Sjoerd Mullender <sjoerd@acm.org>
Date:   Mon Oct 11 12:54:31 1993 +0000

    * Extended X interface: pixmap objects, colormap objects visual objects,
      image objects, and lots of new methods.
    * Added counting of allocations and deallocations of builtin types if
      COUNT_ALLOCS is defined.  Had to move calls to NEWREF down in some
      files.
    * Bug fix in sorting lists.
msg361036 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-01-30 11:02
I used COUNT_ALLOCS for curiosity. But I needed slightly different information, so in any case I patched the code.

AFAIK COUNT_ALLOCS is used in some large projects (maybe Fedora). It was already discussed somewhere on the tracker, but I have no a link.
msg361046 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-30 11:57
I started a thread on the python-dev mailing list:
https://mail.python.org/archives/list/python-dev@python.org/thread/YEY2TZIWC6HGMM5Y3QKWKMEBT5Y5C324/

Serhiy:
> I used COUNT_ALLOCS for curiosity. But I needed slightly different information, so in any case I patched the code.

Did you use it recently? I understand that the unmodified feature didn't help for your use case. Was it to debug memory leaks?


Serhiy:
> AFAIK COUNT_ALLOCS is used in some large projects (maybe Fedora). It was already discussed somewhere on the tracker, but I have no a link.

Did you read previous comments, especially msg360985? It's no longer used in Fedora :-) Technically, it's still used in the python2 package, but it's gone from python3 package. But this issue is about Python 3.9, not Python 2 ;-)

FYI I'm now working in the Python Maintenance team at Red Hat which maintains python2 and python3 packages of Fedora.

By the way, Charalampos Stratakis is also in my team and he replied on python-dev:

"I've never used this feature and it was quite the hassle to maintain those patches downstream, so in my biased opinion, I would welcome the removal."

https://mail.python.org/archives/list/python-dev@python.org/message/4BI64IXNZMJGXNHOA34UJ35V74IGJNZF/
msg361051 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-01-30 12:14
It was many years ago, and I used it to get some statistics about using Python objects of some types. But it did not work as I wanted for cached integers etc. In any case I can patch Python again if I need such information.

Seems Fedora was the main user of this feature. If it is no longer used in Fedora, I think it can be safely removed.
msg361294 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-03 14:17
New changeset c6e5c1123bac6cbb4c85265155af5349dcea522e by Victor Stinner in branch 'master':
bpo-39489: Remove COUNT_ALLOCS special build (GH-18259)
https://github.com/python/cpython/commit/c6e5c1123bac6cbb4c85265155af5349dcea522e
msg361297 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-03 14:21
Done, I remove COUNT_ALLOCS macro and all associated code.
History
Date User Action Args
2020-02-03 14:21:24vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg361297

stage: patch review -> resolved
2020-02-03 14:17:22vstinnersetmessages: + msg361294
2020-01-30 12:14:21serhiy.storchakasetmessages: + msg361051
2020-01-30 11:57:56vstinnersetmessages: + msg361046
2020-01-30 11:02:25serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg361036
2020-01-29 21:43:31vstinnersetmessages: + msg360988
2020-01-29 21:40:00vstinnersetmessages: + msg360987
2020-01-29 21:31:14vstinnersetmessages: + msg360986
2020-01-29 21:29:33vstinnersetmessages: + msg360985
2020-01-29 19:26:13rhettingersetnosy: + rhettinger
messages: + msg360981
2020-01-29 18:22:17vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request17634
2020-01-29 18:21:50vstinnersetmessages: + msg360979
2020-01-29 18:16:16vstinnercreate