New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use Python memory allocators in external libraries like zlib or OpenSSL #62427
Comments
With the PEP-445 and the issue bpo-3329, Python will get an API to setup custom memory allocators. To be able to configure how memory is handled in external libraries, some libraries allow to setup a custom allocator too. New functions PyMem_RawMalloc(), PyMem_GetRawAllocators() PyMem_SetRawAllocators() can be used for that. The safest option is to only reuse custom allocators if a library allows to setup them for a specfic function call or a specific object, and not replace the memory allocators globally. For example, the lzma library allows to set memory allocators only for one compressor object: LzmaEnc_Create(&SzAllocForLzma); We might change the global allocators of a library if Python is not embedded, but Python *is* the application (the standard "python" program). I don't know yet if it is safe to reuse custom memory allocators. Windows has for example a special behaviour: each DLL (dynamic library) has its own heap, memory allocator in a DLL cannot be released from another DLL. Would this issue introduce such bug? |
See also the issue bpo-18203: "Replace calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc()". |
Uncomplete(?) list of external libraries used by Python:
|
It looks like CRYPTO_set_mem_functions() of OpenSSL 1.0.1e-4.fc18 does not work: CRYPTO_set_mem_functions() calls indirectly CRYPTO_malloc() which sets "allow_customize = 0;" and so CRYPTO_set_mem_functions() does nothing (just return 0, instead of 1). Gdb trace with a modified _ssl module: #0 0x0000003803463100 in CRYPTO_malloc () from /lib64/libcrypto.so.10 See the code: |
Here is an initial attempt: set a custom allocator for bz2, lzma and zlib modules. The allocator is only replaced for an instance of a compressor or decompress, the change does not affect the library globally. PyMem_RawMalloc() is used instead of PyMem_Malloc() because the GIL is always released. |
New changeset a876d9d2e4fc by Victor Stinner in branch 'default': |
New changeset 12f26c356611 by Victor Stinner in branch 'default': |
New changeset 7f17c67b5bf6 by Christian Heimes in branch 'default': |
I modified modules when it was possible and easy to do. More modules should be modified, but it's more tricky. If you are interested, please open new issues. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: