This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author neologix
Recipients BreamoreBoy, flub, loewis, neologix, pitrou, sable, tim.peters
Date 2011-05-04.18:46:40
SpamBayes Score 1.66533e-16
Marked as misclassified No
Message-id <>
In-reply-to <>
> Also that addresses the issue of "two threads inside different malloc implementations at the same time": it is currently not allowed with PyMem_Malloc.

That's not true.
You can perfectly have one thread inside PyMem_Malloc while another
one is inside libc's malloc.
For example, posix_listdir does:

     dirp = opendir(name);

Where opendir calls malloc internally. Since the GIL is released, you
can have another thread inside PyMem_Malloc at the same time. This is
perfectly safe, as long as the libc's malloc version is thread-safe.

But with your patch, such code wouldn't be thread-safe anymore. This
patch implies that a thread can't call malloc directly or indirectly
(printf, opendir, and many others) while it doesn't hold the GIL. This
is going to break a lot of existing code.
This thread-safety issue is not theoretical: I wrote up a small
program with two threads, one allocating/freeing memory in loop with
glibc's malloc and the other one with dlmalloc: it crashes immediately
on a Linux box.

> Most python objects will be allocated in pymalloc arenas (if they are smaller than 256 bytes) which (if compiled with --with-pymalloc-mmap) will be directly allocated by calling mmap, or (without --with-pymalloc-mmap) will be allocated in dlmalloc by calling mmap (because arenas are 256KB).
> So most of the python objects will end up in mmap segments separate from the heap.
> The only allocations that will end up in the heap are for the medium python objects (>256 bytes and <256KB) or for allocations directly by calling  PyMem_Malloc (and for a size <256KB).

Note that there are actually many objects falling into this category:
for example, on 64-bit, a dictionary exceeds 256B, and is thus
allocated directly from the heap (well, it changed really recently
actually), the same holds for medium-sized lists and strings. So,
depending on your workload, the heap can extend and shrink quite a

> If you are really concerned about mixing 2 malloc implementations in the heap, you can define "HAVE_MORECORE 0" in dlmalloc and that way dlmalloc will always use mmap and not use the heap at all.

It will also be slower, and consume more memory.
Date User Action Args
2011-05-04 18:46:43neologixsetrecipients: + neologix, tim.peters, loewis, pitrou, sable, flub, BreamoreBoy
2011-05-04 18:46:41neologixlinkissue3526 messages
2011-05-04 18:46:40neologixcreate