classification
Title: Add _PySys_GetSizeOf()
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Arfrever, christian.heimes, loewis, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2014-08-14 10:24 by serhiy.storchaka, last changed 2014-11-15 12:13 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
_PySys_GetSizeOf.patch serhiy.storchaka, 2014-08-14 10:24 review
_PySys_GetSizeOf_overflow.patch serhiy.storchaka, 2014-08-15 09:32 review
_PySys_GetSizeOf_overflow_2.patch serhiy.storchaka, 2014-08-17 13:45 review
Messages (13)
msg225294 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-08-14 10:24
Proposed patch adds private function _PySys_GetSizeOf(). It is needed for correct implementing of __sizeof__() methods: issue15490, issue15513, issue15381 (BytesIO.__sizeof__ is broken in 3.5). See discussion about it in issue15490. I extracted this patch in separate issue for its simplicity and because it is needed in other issues.
msg225301 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-08-14 15:14
LGTM
msg225310 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-08-14 19:34
New changeset b5114747d3ed by Serhiy Storchaka in branch '2.7':
Issue #22193: Added private function _PySys_GetSizeOf() needed to implement
http://hg.python.org/cpython/rev/b5114747d3ed

New changeset 46504edf7594 by Serhiy Storchaka in branch '3.4':
Issue #22193: Added private function _PySys_GetSizeOf() needed to implement
http://hg.python.org/cpython/rev/46504edf7594

New changeset e23999892e58 by Serhiy Storchaka in branch 'default':
Issue #22193: Added private function _PySys_GetSizeOf() needed to implement
http://hg.python.org/cpython/rev/e23999892e58
msg225312 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-08-14 19:40
Thanks Martin.
msg225319 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2014-08-14 21:10
The code is missing an overflow check. There should be some verification that __sizeof__ returns a value smaller than SIZE_T_MAX - 24. Also 24 for GC overhead looks strange. IMHO it should be sizeof(PyGC_Head) which can be smaller than 24 bytes.

Python 3.5.0a0 (default:601045ceff94, Aug 14 2014, 22:59:49) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class Overflow:
...     def __sizeof__(self):
...         return (1<<64)-1
... 
>>> sys.getsizeof(Overflow())
23
msg225341 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-08-15 05:39
This was discussed in issue15490. Such implementation is just incorrect, because we cannot allocate such much memory.
msg225346 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-08-15 09:32
Here is a patch which adds overflow check. The question is: should sys.getsizeof with the default argument catch OverflowError in additional to TypeError and replace it by default value?
msg225352 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2014-08-15 15:00
Christian: I don't see why 24 bytes overhead sounds strange. GC_Head is

typedef union _gc_head {
    struct {
        union _gc_head *gc_next;
        union _gc_head *gc_prev;
        Py_ssize_t gc_refs;
    } gc;
    double dummy;  /* force worst-case alignment */
} PyGC_Head;

which happens to be 3*8=24 bytes on a 64-bit system.
msg225422 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2014-08-17 01:28
Revision b5114747d3ed added useless '#ifndef Py_LIMITED_API' check in 2.7 branch.
Support for Py_LIMITED_API was introduced in Python 3.2.
msg225432 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-08-17 10:31
Fixed in changeset 2e417d9a2b1c. Thank you Arfrever.
msg225442 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-08-17 13:45
Here is alternative patch. May be it looks more clear for you.
msg231205 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-11-15 12:05
New changeset 3537994fa43b by Serhiy Storchaka in branch '2.7':
Issue #22193: Fixed integer overflow error in sys.getsizeof().
https://hg.python.org/cpython/rev/3537994fa43b

New changeset df5c6b05238e by Serhiy Storchaka in branch '3.4':
Issue #22193: Fixed integer overflow error in sys.getsizeof().
https://hg.python.org/cpython/rev/df5c6b05238e

New changeset b7651f9be4a1 by Serhiy Storchaka in branch 'default':
Issue #22193: Fixed integer overflow error in sys.getsizeof().
https://hg.python.org/cpython/rev/b7651f9be4a1
msg231207 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-11-15 12:13
Also fixed an error in _PySys_GetSizeOf declaration (thanks yomgui1). Thanks all for reviews and found bugs.
History
Date User Action Args
2014-11-15 12:13:53serhiy.storchakasetstatus: open -> closed

messages: + msg231207
stage: patch review -> resolved
2014-11-15 12:05:37python-devsetmessages: + msg231205
2014-08-17 13:45:05serhiy.storchakasetfiles: + _PySys_GetSizeOf_overflow_2.patch

messages: + msg225442
stage: needs patch -> patch review
2014-08-17 10:31:35serhiy.storchakasetmessages: + msg225432
2014-08-17 01:28:44Arfreversetnosy: + Arfrever
messages: + msg225422
2014-08-15 15:00:46loewissetmessages: + msg225352
2014-08-15 09:32:07serhiy.storchakasetfiles: + _PySys_GetSizeOf_overflow.patch

messages: + msg225346
2014-08-15 05:39:03serhiy.storchakasetmessages: + msg225341
2014-08-14 21:10:47christian.heimessetstatus: closed -> open

nosy: + christian.heimes
messages: + msg225319

stage: resolved -> needs patch
2014-08-14 19:40:40serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg225312

stage: commit review -> resolved
2014-08-14 19:34:51python-devsetnosy: + python-dev
messages: + msg225310
2014-08-14 15:14:54loewissetmessages: + msg225301
stage: patch review -> commit review
2014-08-14 10:24:53serhiy.storchakacreate