Add the set_nomemory_allocator() function to _testcapi that sets a no memory allocator.

I think the allocators must just return NULL, without setting the error.

Nice. Do you know my https://pypi.python.org/pypi/pyfailmalloc project? It
helped me to identify and fix dozens of code which didn't handle properly
memory allocation failures. Seach for "pyfailmalloc" in the bug tracker
(closed issues) ;-)

Yes, actually I thought first about re-opening issue 19817 instead of opening this new issue.

> I think the allocators must just return NULL, without setting the error.
You are right, thanks.

The main difference between your change and pyfailmalloc is the ability in pyfailmalloc to only fail in N allocations. If you want to test various code paths, you need to allow N allocations to reach the deep code that you want to test.

My code is something like 100 lines of C code.
https://bitbucket.org/haypo/pyfailmalloc/src/cdaf3609a30b88243d04e79d6074f3b28d9b64e3/failmalloc.c?at=default&fileviewer=file-view-default

IMHO it's small enough to fit directly into _testcapi.

The question is more what do you want to do with it? It was proposed to "include" pyfailmalloc directly in the Python test suite. Maybe add an option to enable it?

I ran the Python test suite with pyfailmalloc in gdb using this script:
https://bitbucket.org/haypo/pyfailmalloc/src/cdaf3609a30b88243d04e79d6074f3b28d9b64e3/debug_cpython.gdb?at=default&fileviewer=file-view-default

./python -m test -F -r -v -x test_tracemalloc

I waited for the next *crash* and ignored all expected tests failing with MemoryError. --forever (-F) is nice to run random tests until you get a crash.

The chain of events following a call to set_nomemory_allocator() is deterministic, this is another difference with pyfailmalloc. Also what happens then after this call is not very close to real life as memory is never freed.

Having the possibility to trigger memory exhaustion in N allocations instead of rightaway would be very useful. Maybe this could be done by including pyfailmalloc in _testcapi with some changes to enable multiple behaviors, among which one of them would be deterministic.

Your code can be reproduced with pyfailmalloc using N=0 :-)

I am not sure, failmalloc.enable(range) accepts only range > 1, hook_malloc() fails only once instead of permanently and hook_free() does free memory.

> I am not sure, failmalloc.enable(range) accepts only range > 1, hook_malloc() fails only once instead of permanently and hook_free() does free memory.

All these things can change :-)

> Also what happens then after this call is not very close to real life as memory is never freed.

Why not freeing memory?

Good point. Not freeing the memory made the implementation of set_nomemory_allocator() simpler, no need to call PyMem_GetAllocator(). Forget about this point, sorry :-)

I prefer your new PR. But is it useful to test N memory allocation failures in a row? The API and the code would be simpler if we would only test a single failure. What do you think?

I know that pyfailmalloc is different, but I'm not sure that pyfailmalloc design is right neither :-)

For the record, while working on the test case of PR 2406, I found by chance that the following script:

# Script start.
import _testcapi
class C(): pass
_testcapi.set_nomemory(0, 5)
C()
# Script end.

fails with:

python: Objects/call.c:89: _PyObject_FastCallDict: Assertion `!PyErr_Occurred()' failed.
Aborted (core dumped)

I will create a new issue when the current issue is closed.

> But is it useful to test N memory allocation failures in a row?

I think it is useful. See my previous post.

2017-06-26 15:34 GMT+02:00 Xavier de Gaye <report@bugs.python.org>:
> python: Objects/call.c:89: _PyObject_FastCallDict: Assertion `!PyErr_Occurred()' failed.
> Aborted (core dumped)
>
> I will create a new issue when the current issue is closed.

Oh, I'm curious about that one :-)

> Oh, I'm curious about that one :-)

This is issue 30817.

New changeset 85f643023fed3d4e2fb8e399f9ad57f3a65ef237 by xdegaye in branch 'master':
bpo-30695: Add set_nomemory(start, stop) to _testcapi (GH-2406)
https://github.com/python/cpython/commit/85f643023fed3d4e2fb8e399f9ad57f3a65ef237

Test cases in issues #30697 and #30817, back ported to 3.6, need _testcapi.set_nomemory().

New changeset aaf6a3dbbdb9754f98d480b468adfcae0f66e3a2 by xdegaye (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-30695: Add set_nomemory(start, stop) to _testcapi (GH-2406) (#4083)
https://github.com/python/cpython/commit/aaf6a3dbbdb9754f98d480b468adfcae0f66e3a2