This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Use malloc_usable_size() in pymalloc for realloc
Type: performance Stage: resolved
Components: Interpreter Core Versions: Python 3.7
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: pitrou, serhiy.storchaka, skrah, vstinner
Priority: normal Keywords:

Created on 2017-10-31 17:57 by vstinner, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (3)
msg305319 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-10-31 17:57
Objects/obmalloc.c contains an interesting comment:

    if (!address_in_range(p, pool)) {
        /* pymalloc is not managing this block.

           If nbytes <= SMALL_REQUEST_THRESHOLD, it's tempting to try to take
           over this block.  However, if we do, we need to copy the valid data
           from the C-managed block to one of our blocks, and there's no
           portable way to know how much of the memory space starting at p is
           valid.

           As bug 1185883 pointed out the hard way, it's possible that the
           C-managed block is "at the end" of allocated VM space, so that a
           memory fault can occur if we try to copy nbytes bytes starting at p.
           Instead we punt: let C continue to manage this block. */
        return 0;
    }

See also bpo-1185883.

We don't have to guess, it's possible to get the size of a memory block allocated by malloc() with:

* malloc_usable_size(ptr): available at least on Linux
* _msize(ptr): Windows

Maybe we could add a "msize" field to PyMemAllocatorEx? See also bpo-18835 whic adds PyMem_AlignedAlloc() and so already modify (and rename PyMemAllocatorEx).

See also bpo-31626, but I'm not sure that it would help for _PyMem_DebugRawRealloc().
msg306402 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-11-16 23:11
What would be the point of this?  This would complicate the code with platform-specific code and it's not obvious that it would change runtime performance in any significant way.
msg306870 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-24 01:26
I agree. I don't think that it's worth it to overcompilates the API.

I proposed this change since we may change the API to add PyMem_AlignedAlloc(). Since I just rejected bpo-18835, it's no more the case.

I close this issue as rejected.
History
Date User Action Args
2022-04-11 14:58:53adminsetgithub: 76092
2017-11-24 01:26:37vstinnersetstatus: open -> closed
resolution: rejected
messages: + msg306870

stage: resolved
2017-11-16 23:11:46pitrousetnosy: + pitrou
messages: + msg306402
2017-11-03 11:15:49vstinnersettitle: Use malloc_usable_size() is pymalloc for realloc -> Use malloc_usable_size() in pymalloc for realloc
2017-10-31 17:57:44vstinnercreate