Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance Python's Memory Instrumentation with COUNT_ALLOCS #77239

Closed
eduardo-elizondo mannequin opened this issue Mar 12, 2018 · 6 comments
Closed

Enhance Python's Memory Instrumentation with COUNT_ALLOCS #77239

eduardo-elizondo mannequin opened this issue Mar 12, 2018 · 6 comments
Labels
3.8 only security fixes type-feature A feature request or enhancement

Comments

@eduardo-elizondo
Copy link
Mannequin

eduardo-elizondo mannequin commented Mar 12, 2018

BPO 33058
Nosy @vstinner, @serhiy-storchaka, @eduardo-elizondo
PRs
  • bpo-33058: Enhanced COUNT_ALLOCS to be ABI Compatible #6091
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2018-03-12.19:31:15.946>
labels = ['type-feature', '3.8']
title = "Enhance Python's Memory Instrumentation with COUNT_ALLOCS"
updated_at = <Date 2018-03-12.22:05:43.281>
user = 'https://github.com/eduardo-elizondo'

    bugs.python.org fields:

    activity = <Date 2018-03-12.22:05:43.281>
actor = 'eelizondo'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2018-03-12.19:31:15.946>
creator = 'eelizondo'
dependencies = []
files = []
hgrepos = []
issue_num = 33058
keywords = ['patch']
message_count = 4.0
messages = ['313684', '313685', '313686', '313691']
nosy_count = 3.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'eelizondo']
pr_nums = ['6091']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue33058'
versions = ['Python 3.8']
    @eduardo-elizondo eduardo-elizondo mannequin added 3.8 only security fixes type-feature A feature request or enhancement labels Mar 12, 2018
    @eduardo-elizondo eduardo-elizondo mannequin changed the title Enhancing Python [WIP] Enhancing Python's COUNT_ALLOCS Mar 12, 2018
    @eduardo-elizondo eduardo-elizondo mannequin changed the title [WIP] Enhancing Python's COUNT_ALLOCS [WIP] Enhance Python's Memory Instrumentation with COUNT_ALLCOS Mar 12, 2018
    @eduardo-elizondo
    Copy link
    Mannequin Author

    eduardo-elizondo mannequin commented Mar 12, 2018

    [WIP]

    @eduardo-elizondo eduardo-elizondo mannequin changed the title [WIP] Enhance Python's Memory Instrumentation with COUNT_ALLCOS Enhance Python's Memory Instrumentation with COUNT_ALLOCS Mar 12, 2018
    @eduardo-elizondo
    Copy link
    Mannequin Author

    eduardo-elizondo mannequin commented Mar 12, 2018

    Currently, Python has very few instrumentation when it comes to the types of allocations that it's doing. For instance, we currently can't identify how many times an object uses free lists vs actual mallocs.

    Currently, there's a special build which can be used by compiling with "-DCOUNT_ALLOCS". However, this build is not ABI compatible with extensions. Meaning you have to recompile all the modules that are used. Doing this on a large scale scenario (1000+ modules) is not feasible.

    Thus, I propose the following enhancements:

    • Make COUNT_ALLOCS ABI compatible

    Then:

    • Expand the counters to not only track allocs and frees but also distinguish between:
    • GC Counts: Allocs/Frees coming from PyObject_GC_Malloc PyObject_GC_Del
    • Memory Counts: Allocs/Frees coming from PyMem_Malloc/PyObject_Malloc PyObject_Free (modulo GC objects).
    • Free Lists: "Allocs/Frees" coming from specific object's free_lists/caches.
    • Misc: Extra kinds of "Allocs/Frees" such as code's zombie frame.

    @serhiy-storchaka
    Copy link
    Member

    Could tracemalloc help you?

    @eduardo-elizondo
    Copy link
    Mannequin Author

    eduardo-elizondo mannequin commented Mar 12, 2018

    @serhiy.storchaka tracemalloc can't distinguish between the usage of gc allocs, normal mallocs, and free list reuse.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @iritkatriel
    Copy link
    Member

    The PR was closed by the OP in 2019. Closing this as well.

    @iritkatriel iritkatriel closed this as not planned Won't fix, can't repro, duplicate, stale Jan 31, 2023
    @vstinner
    Copy link
    Member

    vstinner commented Apr 5, 2023 •
    edited

    The COUNT_ALLOCS special build was removed in Python 3.9: #83670

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.8 only security fixes type-feature A feature request or enhancement
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @vstinner @iritkatriel @serhiy-storchaka