Author neologix
Recipients josh.r, jtaylor, neologix, njs, pitrou, skrah, vstinner
Date 2014-04-27.13:19:02
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <CAH_1eM0iQysGLgkkyLwb=FtY4W+UWa_kysjCV+bDQiv8y+d1vw@mail.gmail.com>
In-reply-to <1398597164.17.0.618208848538.issue21233@psf.upfronthosting.co.za>
Content
> It looks like calloc-3.patch is wrong: it modify _PyObject_GC_Malloc() to fill the newly allocated buffer with zeros, but _PyObject_GC_Malloc() is not only called by PyType_GenericAlloc(): it is also used by _PyObject_GC_New() and _PyObject_GC_NewVar(). The patch is maybe a little bit slower because it writes zeros twice.

Exactly (sorry, I thought you'd already seen that, otherwise I could
have told you!)

> Actually, I think we have to match the C-API:  For instance, in
> Modules/_decimal/_decimal.c:5527 the libmpdec allocators are
> set to the Python allocators.

Hmm, ok then, I didn't know we were plugging our allocators for
external libraries: that's indeed a very good reason to keep the same
prototype.

But I still find this API cumbersome: calloc is exactly like malloc
except for the zeroing, so the prototype could be simpler (a quick
look at Victor's patch shows a lot of calloc(1, n), which is a sign
something's wrong). Maybe it's just me ;-)

Otherwise, a random thought: by changing PyType_GenericAlloc() from
malloc() + memset(0) to calloc(), there could be a subtle side effect:
if a given type relies on the 0-setting (which is documented), and
doesn't do any other work on the allocated area behind the scenes
(think about a mmap-like object), we could lose our capacity to detect
MemoryError, and run into segfaults instead.

Because if a code creates many such objects which basically just do
calloc(), on operating systems with memory overommitting (such as
Linux), the calloc() allocations will pretty much always succeed, but
will segfault when the page is first written to in case of low memory.

I don't think such use cases should be common: I would expect most
types to use tp_alloc(type, 0) and then use an internal additional
pointer for the allocations it needs, or immediately write to the
allocated memory area right after allocation, but that's something to
keep in mind.
History
Date User Action Args
2014-04-27 13:19:03neologixsetrecipients: + neologix, pitrou, vstinner, njs, skrah, jtaylor, josh.r
2014-04-27 13:19:03neologixlinkissue21233 messages
2014-04-27 13:19:02neologixcreate