Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use "Low-fragmentation Heap" memory allocator on Windows #70439

Closed
vstinner opened this issue Jan 31, 2016 · 8 comments
Closed

Use "Low-fragmentation Heap" memory allocator on Windows #70439

vstinner opened this issue Jan 31, 2016 · 8 comments
Labels
OS-windows performance Performance or resource usage

Comments

@vstinner
Copy link
Member

BPO 26251
Nosy @pfmoore, @vstinner, @tjguk, @zware, @zooba

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-29.22:11:35.960>
created_at = <Date 2016-01-31.17:55:40.898>
labels = ['OS-windows', 'performance']
title = 'Use "Low-fragmentation Heap" memory allocator on Windows'
updated_at = <Date 2018-05-29.22:11:35.960>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-05-29.22:11:35.960>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-29.22:11:35.960>
closer = 'vstinner'
components = ['Windows']
creation = <Date 2016-01-31.17:55:40.898>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 26251
keywords = []
message_count = 8.0
messages = ['259293', '259294', '259296', '297106', '297209', '297594', '297610', '318122']
nosy_count = 5.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'zach.ware', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue26251'
versions = ['Python 3.6']
@vstinner
Copy link
Member Author

Python has a memory allocator optimized for allocations <= 512 bytes: PyObject_Malloc(). It was discussed to replace it by the native "Low-fragmentation Heap" memory allocator on Windows.

I'm not aware of anyone who tried that. I would nice to try, especially to run benchmarks.

See also the issue bpo-26249: "Change PyMem_Malloc to use PyObject_Malloc allocator?".

@vstinner vstinner added OS-windows performance Performance or resource usage labels Jan 31, 2016
@vstinner
Copy link
Member Author

"Low-fragmentation Heap":
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366750%28v=vs.85%29.aspx

@vstinner
Copy link
Member Author

The issue bpo-19246 "high fragmentation of the memory heap on Windows" was rejected but discussed the Windows Low Fragmented Heap.

@vstinner
Copy link
Member Author

Is there anyway interested to experiment to write such change and run benchmarks with it?

@zooba
Copy link
Member

zooba commented Jun 28, 2017

We tried it at one point, but it made very little difference because we don't use the Windows heap for most allocations. IIRC, replacing Python's optimised allocator with the LFH was a slight performance regression, but I'm not sure the benchmarks were reliable enough back then to be trusted. I'm also not sure what optimisations have been performed in Windows 8/10.

Since the LFH is the default though, it really should just be a case of replacing Py_Malloc with a simple HeapAlloc shim and testing it. The APIs are nearly the same (the result of GetProcessHeap() will be stable for the lifetime of the process, and there's little value in creating specific heaps unless you intend to destroy it rather than free each allocation individually).

@vstinner
Copy link
Member Author

vstinner commented Jul 3, 2017

Steve: "We tried it at one point, but it made very little difference (...)"

Ok. Can I close the issue?

@zooba
Copy link
Member

zooba commented Jul 3, 2017

I wouldn't be opposed to seeing it tried again, but I have no strong opinion. I don't think this is a major performance bottleneck right now.

@vstinner
Copy link
Member Author

I failed to find the bandwidth to work on this issue since 2 years, so I just abandon this idea. However the performance benefit is non obvious.

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
OS-windows performance Performance or resource usage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vstinner @zooba