New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[subinterpreters] Explicitly track object ownership (and allocator). #77788
Comments
When an object is created it happens relative to the current Regarding tracking the interpreter, that originating interpreter Regarding the allocator, there used to be just a single global To sort all this out we would need to track per-object:
Either we'd add 2 pointer-size fields to PyObject or we would |
"Either we'd add 2 pointer-size fields to PyObject or we would keep a separate hash table (or two) pointing from each object to the info (...)" The expect a huge impact on the memory footprint. I dislike the idea. Currently, the smallest Python object is: >>> sys.getsizeof(object())
16 It's just two pointers. Adding two additional pointers would simply double the size of the object. "Now the C-API offers a way to switch the allocator, so there's no guarantee I would expect that either all interpreters use the same memory allocator, or that each interpreter uses its own allocator. If you use one allocator per interpreter, calling PyMem_Free() from the wrong interpreter would just crash. As you get a crash when you call free() on an object allocated by PyMem_Free(). You can extend PYTHONMALLOC=debug to detect bugs. This builtin debugger is already able to catch misuses of allocators. Adding extra pointers to this debugger is acceptable since it doesn't modify the footprint of the default mode. |
Rather than tracking this per object, you could potentially track it per arena at the memory allocator level instead. Then if you really need the info (e.g. when running the debug allocator), you can check it in a reliable way, but in the normal case, you assume the associations are being managed correctly and avoid any significant bookkeeping overhead. |
Note that I wouldn't call this issue absolutely specific to subinterpreters. The "ownership" part is, but tracking the allocator has practical application under a single interpreter. I suppose I could split this issue apart. I lumped the two together because I expected the solution would be the same for both. However, that's not necessarily the case. Would it help to open a separate issue for tracking the allocator? |
I agree with Victor, we shouldn't add PyObject fields that only have use in certain (minority) situations. The idea of tracking per arena will be non-trivial to implement, as only small objects (smaller than 512 bytes) use our own allocator; larger objects go to the system allocator. Can I ask why you're considering this? I thought you didn't want to transfer ownership between interpreters. |
I see two options:
See also bpo-40514: [subinterpreters] Add --experimental-isolated-subinterpreters build option. Antoine: "Can I ask why you're considering this? I thought you didn't want to transfer ownership between interpreters." I guess that the purpose is to ensure that: detect when an object is shared between two interpreters. Currently, tons of objects are still shared between interpreters. Starting with static types: see bpo-40601 "[C API] Hide static types from the limited C API". |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: