classification
Title: mmap should expose madvise()
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: giampaolo.rodola, josh.r, larry, miss-islington, ned.deily, neologix, pitrou, ronaldoussoren
Priority: normal Keywords: patch

Created on 2018-02-24 21:15 by pitrou, last changed 2019-05-27 17:57 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 6172 merged ZackerySpytz, 2018-03-21 03:11
PR 13596 merged pitrou, 2019-05-27 17:37
Messages (6)
msg312758 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-02-24 21:15
On POSIX, mmap objects could expose a method wrapping the madvise() library call.  I suggest the following API

  mmap_object.madvise(option[, start[, length]])

If omitted, *start* and *length* would span the whole memory area described by the mmap object.  *option* must be a recognized OS option for the madvise() library call.

The mmap module would expose the various MADV_* options available on the current platform.

Open question: should we expose madvise() or posix_madvise()? (these are two different calls, at least on Linux)  posix_madvise() is arguably more portable, but madvise() is much more powerful, so I'd lean towards madvise().
msg312759 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-02-24 21:15
For information:
- macOS manpage for madvise():
http://www.manpages.info/macosx/madvise.2.html
- FreeBSD manpage for madvise():
https://www.freebsd.org/cgi/man.cgi?query=madvise&sektion=2
msg339375 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2019-04-03 02:42
It might be nice to expose a more limited API to prefetch memory in bulk while we're at it. Windows 8 and higher offers PrefetchVirtualMemory ( https://docs.microsoft.com/en-us/windows/desktop/api/memoryapi/nf-memoryapi-prefetchvirtualmemory ) which fills roughly the same niche as madvise/posix_madvise with the WILLNEED hint.

I kept meaning to file an issue to suggest this, but kept getting hung up on how to make it as portable as possible and as powerful as possible, and where to implement it.

It could go on mmap, but it's also useful for just about any contiguous buffer-supporting object, so it almost seems like adding an os.madvise (and/or os.prefetch) would make more sense than specifically tying it to the mmap module.
msg343659 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-05-27 16:48
New changeset 02db696732c031d9a0265dc9bbf4f5b1fad042b3 by Antoine Pitrou (Zackery Spytz) in branch 'master':
bpo-32941: Add madvise() for mmap objects (GH-6172)
https://github.com/python/cpython/commit/02db696732c031d9a0265dc9bbf4f5b1fad042b3
msg343660 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-05-27 16:53
Josh, sorry, I hadn't seen your message.

Those are low-levels operations, so I don't know if it makes sense to implement madvise() in terms of PrefetchVirtualMemory(), or expose a separate wrapper to PrefetchVirtualMemory().

One complication is that we currently don't expose public wrappers over Windows API functions (there's a private _winapi module).
msg343667 - (view) Author: miss-islington (miss-islington) Date: 2019-05-27 17:57
New changeset 695b1dd8cbf3a48fdb30ab96918a49b20b7ec3e7 by Miss Islington (bot) (Antoine Pitrou) in branch 'master':
bpo-32941:  Fix test_madvise failure when page size >= 8kiB (GH-13596)
https://github.com/python/cpython/commit/695b1dd8cbf3a48fdb30ab96918a49b20b7ec3e7
History
Date User Action Args
2019-05-27 17:57:26miss-islingtonsetnosy: + miss-islington
messages: + msg343667
2019-05-27 17:37:30pitrousetpull_requests: + pull_request13503
2019-05-27 16:53:22pitrousetmessages: + msg343660
2019-05-27 16:48:56pitrousetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-05-27 16:48:19pitrousetmessages: + msg343659
2019-04-03 02:42:45josh.rsetnosy: + josh.r
messages: + msg339375
2019-04-03 01:24:53giampaolo.rodolasetnosy: + giampaolo.rodola
2018-03-21 03:11:59ZackerySpytzsetkeywords: + patch
stage: patch review
pull_requests: + pull_request5927
2018-02-24 21:15:55pitrousetmessages: + msg312759
2018-02-24 21:15:23pitroucreate