Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mmap should expose madvise() #77122

Closed
pitrou opened this issue Feb 24, 2018 · 6 comments
Closed

mmap should expose madvise() #77122

pitrou opened this issue Feb 24, 2018 · 6 comments
Labels
3.8 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@pitrou
Copy link
Member

pitrou commented Feb 24, 2018

BPO 32941
Nosy @ronaldoussoren, @pitrou, @larryhastings, @giampaolo, @ned-deily, @MojoVampire, @miss-islington
PRs
  • bpo-32941: Add madvise() for mmap objects #6172
  • bpo-32941: Fix test_madvise failure when page size >= 8kiB #13596
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2019-05-27.16:48:56.032>
created_at = <Date 2018-02-24.21:15:23.475>
labels = ['3.8', 'type-feature', 'library']
title = 'mmap should expose madvise()'
updated_at = <Date 2019-05-27.17:57:26.307>
user = 'https://github.com/pitrou'

    bugs.python.org fields:

    activity = <Date 2019-05-27.17:57:26.307>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-27.16:48:56.032>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2018-02-24.21:15:23.475>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 32941
keywords = ['patch']
message_count = 6.0
messages = ['312758', '312759', '339375', '343659', '343660', '343667']
nosy_count = 8.0
nosy_names = ['ronaldoussoren', 'pitrou', 'larry', 'giampaolo.rodola', 'ned.deily', 'neologix', 'josh.r', 'miss-islington']
pr_nums = ['6172', '13596']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue32941'
versions = ['Python 3.8']
    @pitrou
    Copy link
    Member Author

    pitrou commented Feb 24, 2018

    On POSIX, mmap objects could expose a method wrapping the madvise() library call. I suggest the following API

      mmap_object.madvise(option[, start[, length]])

    If omitted, *start* and *length* would span the whole memory area described by the mmap object. *option* must be a recognized OS option for the madvise() library call.

    The mmap module would expose the various MADV_* options available on the current platform.

    Open question: should we expose madvise() or posix_madvise()? (these are two different calls, at least on Linux) posix_madvise() is arguably more portable, but madvise() is much more powerful, so I'd lean towards madvise().

    @pitrou pitrou added 3.8 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Feb 24, 2018
    @pitrou
    Copy link
    Member Author

    pitrou commented Feb 24, 2018

    For information:

    @MojoVampire
    Copy link
    Mannequin

    MojoVampire mannequin commented Apr 3, 2019

    It might be nice to expose a more limited API to prefetch memory in bulk while we're at it. Windows 8 and higher offers PrefetchVirtualMemory ( https://docs.microsoft.com/en-us/windows/desktop/api/memoryapi/nf-memoryapi-prefetchvirtualmemory ) which fills roughly the same niche as madvise/posix_madvise with the WILLNEED hint.

    I kept meaning to file an issue to suggest this, but kept getting hung up on how to make it as portable as possible and as powerful as possible, and where to implement it.

    It could go on mmap, but it's also useful for just about any contiguous buffer-supporting object, so it almost seems like adding an os.madvise (and/or os.prefetch) would make more sense than specifically tying it to the mmap module.

    @pitrou
    Copy link
    Member Author

    pitrou commented May 27, 2019

    New changeset 02db696 by Antoine Pitrou (Zackery Spytz) in branch 'master':
    bpo-32941: Add madvise() for mmap objects (GH-6172)
    02db696

    @pitrou pitrou closed this as completed May 27, 2019
    @pitrou
    Copy link
    Member Author

    pitrou commented May 27, 2019

    Josh, sorry, I hadn't seen your message.

    Those are low-levels operations, so I don't know if it makes sense to implement madvise() in terms of PrefetchVirtualMemory(), or expose a separate wrapper to PrefetchVirtualMemory().

    One complication is that we currently don't expose public wrappers over Windows API functions (there's a private _winapi module).

    @miss-islington
    Copy link
    Contributor

    New changeset 695b1dd by Miss Islington (bot) (Antoine Pitrou) in branch 'master':
    bpo-32941: Fix test_madvise failure when page size >= 8kiB (GH-13596)
    695b1dd

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.8 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @pitrou @miss-islington