New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test leaks of memory not managed by Python allocator #79237
Comments
Would be nice to add a possibility to test memory leaks if memory is allocated not by Python allocators, but inside external libraries. This would allow to catch leaks on the bridge between Python and external libraries. See for example bpo-34794. |
You can try to use Valgrind for that? |
Can it be used on buildbots and in CI tests? |
These kind of tool produces a lot of false alarms :-( They are also hard to use. |
Before seeing how to automate it, first someone should try to detect the bpo-34794 memory leak manually ;-) By the way, when I implemented the PEP-445, I tried to configure OpenSSL to use Python memory allocators (to benefit of tracemalloc), but the OpenSSL API for that didn't work at all: bpo-18227. |
About automation, regrtest -R checks memory leaks using sys.getallocatedblocks(). But this function only tracks PyMem_Malloc() and PyObject_Malloc() (which is now technically the same memory allocator, since Python 3.6: https://docs.python.org/dev/c-api/memory.html#default-memory-allocators ) I tried to track PyMem_RawMalloc() using regrtest but... the raw memory allocator is not really "deterministic", it's hard to get reliable behavior. See: bpo-26850. Tracking memory leaks is an hard topic :-) I'm happy that I finally fixed tracemalloc to track properly objects in free lists: bpo-35053! |
It seems like the easiest thing to do thta would directly benefit (to tracemalloc users) is to continue the implementation of bpo-18227:
Python modules already using Python memory allocators:
Using Python memory allocators gives access to Python builtin "memory debugger", even in release mode using PYTHONMALLOC=debug or -X dev. |
SQLite requires the xSize member of sqlite3_mem_methods to be implemented for this to work, so we'd have to implement msize(). The msize() idea seems to have been rejected in PEP-445, though it mentions using debug hooks to implement it. See also https://www.sqlite.org/c3ref/mem_methods.html Anyway, attached is a PoC patch with a fixed 10k mem pool for the sqlite3 module :) |
Victor, Serhiy: Would this issue be "large" or important enough to re-raise the debate about implementing an msize() function in the PyMem_ API, or is it not worth it? I guess no; it has been discussed numerous times before. Else, I can start a thread on Discourse. |
Multiple C library don't provide msize() function. If you seriously consider to add it, you should conduct a study on all platforms. |
No, I did not mean using msize() or something like. Since memory is managed outside of Python, we have no a list of allocated blocks. I meant that we can get the total memory used by the Python process (using OS-specific methods) and compare it between iterations. If it continues to grow, there is a leak. It perhaps is not able to detect small leaks (less than the page size), but large leaks are more important. |
patch-with-simple-msize.diff: I suggest you opening a new issue to propose using the Python memory allocators in the sqlite module. This issue is about C extension modules which don't use the Python memory allocator. |
Yes, I know. Your proposal in msg328533 is to continue the implementation of bpo-18227:
Thus, I thought using this issue would be ok, but I can split the sqlite3 details out in a separate issue. Using sqlite3_config(SQLITE_CONFIG_MALLOC, ...) _requires_ msize(). |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: