classification
Title: Replacing CPython memory allocation
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: matrixise, pablogsal, paul.harvey@rakuten.com, vstinner
Priority: normal Keywords:

Created on 2018-11-12 08:50 by paul.harvey@rakuten.com, last changed 2018-11-13 03:16 by pablogsal. This issue is now closed.

Messages (6)
msg329725 - (view) Author: paul (paul.harvey@rakuten.com) Date: 2018-11-12 08:50
Hi all, 

I am trying to replace the version of malloc/free...etc with my own function suit. I am have issues with loading the initial library setup. 

I am looking for wisdom from the community as to why this may be the case.

Facts:
- i just grabbed the latest cpython repo
- my memory suit seem to be working from independent testing on other code
- i am working on linux
- i went into obmalloc.c and replaced the malloc, free, realloc, calloc with my own functions. 
- i changed the mmap/unmap to use my malloc and free in obmalloc.c
- my allocated produces aligned allocations.
- i dump the exceptions text being generated to see what is happening:
EXCEPTION:: module 'sys' has no attribute '__file__'
EXCEPTION:: type object 'BuiltinImporter' has no attribute '_ORIGIN'
EXCEPTION:: module 'sys' has no attribute '__cached__'
EXCEPTION:: module 'sys' has no attribute '__path__'
EXCEPTION:: module 'builtins' has no attribute '__file__'
EXCEPTION:: type object 'BuiltinImporter' has no attribute '_ORIGIN'
EXCEPTION:: module 'builtins' has no attribute '__cached__'
EXCEPTION:: module 'builtins' has no attribute '__path__'
EXCEPTION:: module '_frozen_importlib' has no attribute '__file__'
EXCEPTION:: type object 'FrozenImporter' has no attribute '_ORIGIN'
EXCEPTION:: module '_frozen_importlib' has no attribute '__cached__'
EXCEPTION:: module '_frozen_importlib' has no attribute '__path__'
EXCEPTION:: module '_imp' has no attribute '__file__'
EXCEPTION:: type object 'BuiltinImporter' has no attribute '_ORIGIN'
EXCEPTION:: module '_imp' has no attribute '__cached__'
EXCEPTION:: module '_imp' has no attribute '__path__'
EXCEPTION:: name '_bootstrap' is not defined
EXCEPTION:: name '_bootstrap' is not defined
EXCEPTION:: name '_bootstrap' is not defined
EXCEPTION:: name '_bootstrap' is not defined
Fatal Python error: initfsencoding: failed to get the Python codec of the filesystem encoding
Traceback (most recent call last):
  File "/home/paul/fresh_cpython/debug/../Lib/encodings/__init__.py", line 31, in <module>
  File "<frozen importlib._bootstrap>", line 989, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 773, in exec_module
  File "<frozen importlib._bootstrap_external>", line 909, in get_code
  File "<frozen importlib._bootstrap_external>", line 966, in get_data
OSError: [Errno 14] Bad address
Aborted
msg329726 - (view) Author: paul (paul.harvey@rakuten.com) Date: 2018-11-12 08:53
I suspect that this _bootstrap library is not being loaded correctly, and i can only assume that this is somehow because of my memory suit, but i am not really sure where to start hunting, as there is a lot of code. 

My goal is just to replace malloc. If there is a better way or i am making some wrong assumption, please let me know. 

Best, 
Paul
msg329737 - (view) Author: St├ęphane Wirtel (matrixise) * (Python committer) Date: 2018-11-12 15:24
You have the issue with your functions, not with Python. Are you sure you do not have an issue with your code?
msg329756 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-11-12 21:38
Hi @paul,

have you already checked https://docs.python.org/3/c-api/memory.html#customize-memory-allocators ?
msg329778 - (view) Author: paul (paul.harvey@rakuten.com) Date: 2018-11-13 00:26
Hi guys, 

First, thanks for the prompt replies. 

@matrixise: If there is i can't find it, but i am happy to accept it as a possibility. Equally, maybe there is an assumption about memory in the cpython implementation somewhere that goes against my memory allocator -> this is obviously not cpythons fault. The point of this 'issue' is to ask the community to suggest how i can go about finding where the problem is. As i said, there is a lot of code :)

@pablogsal: Yeah. I had a look in there. In cpython, i have made all allocations from obmalloc use my memory allocation functions, including the arenas. 

Perhaps a starting point would be if anyone has ideas about how to go about debugging this??

best, 

Paul
msg329799 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-11-13 03:16
> The point of this 'issue' is to ask the community to suggest how i can go about finding where the problem is. As I said, there is a lot of code :)

Sadly, this tracker is to track bugs related to CPython. Modifying the interpreter to change the memory allocation scheme (replacing the code in obmalloc.c) is not supported, so I have to close this issue :(

You can try to ask the question on some of the mailing lists (like python-dev) to see is someone can help you with this. I would advice to also provide some minimal reproducer/example code, as is very difficult to recommend anything without knowing what exactly are you modifying.

If you think there is something wrong/we can improve around this area, please, feel free to reopen this issue or open a new one :)
History
Date User Action Args
2018-11-13 03:16:43pablogsalsetstatus: open -> closed
resolution: not a bug
messages: + msg329799

stage: resolved
2018-11-13 00:26:47paul.harvey@rakuten.comsetmessages: + msg329778
2018-11-12 21:38:45pablogsalsetmessages: + msg329756
2018-11-12 21:38:37pablogsalsetmessages: - msg329755
2018-11-12 21:38:19pablogsalsetnosy: + pablogsal
messages: + msg329755
2018-11-12 15:26:01vstinnersetnosy: + vstinner
2018-11-12 15:24:15matrixisesetnosy: + matrixise
messages: + msg329737
2018-11-12 08:53:50paul.harvey@rakuten.comsetmessages: + msg329726
2018-11-12 08:50:08paul.harvey@rakuten.comcreate