classification
Title: "s#" and friends can silently truncate buffer length
Type: behavior Stage: needs patch
Components: Interpreter Core Versions: Python 3.1, Python 3.2, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: haypo, loewis, nadeem.vawda, pitrou, python-dev, rcoyner
Priority: normal Keywords: patch

Created on 2010-05-07 17:24 by pitrou, last changed 2011-05-03 13:11 by haypo. This issue is now closed.

Files
File name Uploaded Description Edit
getarg.patch haypo, 2010-05-26 21:54
Messages (11)
msg105217 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-05-07 17:24
When PY_SSIZE_T isn't defined and a format such as "s#" receives an object whose length fits in a Py_ssize_t but not in an int, the buffer length is silently truncated:

>>> s = 'x' * (4 * 1024**3 + 100)
>>> t = zlib.compress(s, 1)
>>> len(t)
12
>>> len(zlib.decompress(t))
100

(from issue8650)
msg106582 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-05-26 21:54
getarg.patch fixes STORE_SIZE macro used in convertsimple(). If the input size is bigger than INT_MAX, it raises an OverflowError("size does not fit in an int") and calls converterr() which expected="".

The value of expected is useless because converterr() is only used to notice that an error occured. I think that return msgbuf instead of calling converterr() would be enough, but I don't know this code very well and so i copied the code used to raise an OverflowError for the 'b' format.
msg106583 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2010-05-26 21:55
Another test (only requires ~2 GB of memory, not 4 GB or more) for the patch:

import _elementtree
def test():
    parser=_elementtree.XMLParser()
    text='s' * (2**31 + 10)
    try:
        parser.feed(text)
    except OverflowError as err:
        print("ok: %s" % err)
        return
    except:
        pass
    print("error: OverflowError not raised")
test()
msg125261 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-04 00:36
Could you add a proper unit test?
msg125282 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-01-04 02:08
Fixed by r87728.

Wait for the buildbots before backporting to other versions.
msg128983 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-02-21 19:27
I removed Antoine's message because it was related to issue #8650.
msg131607 - (view) Author: Roundup Robot (python-dev) Date: 2011-03-21 02:26
New changeset d9633064458c by Victor Stinner in branch '3.1':
Issue #8651: PyArg_Parse*() functions raise an OverflowError if the file
http://hg.python.org/cpython/rev/d9633064458c
msg131609 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-03-21 02:30
Ok, I backported the fix to 3.1.

Reopen the issue if you would like a port to 2.7 (I am too lazy to do it).
msg135033 - (view) Author: Roundup Robot (python-dev) Date: 2011-05-03 13:08
New changeset 509f1c15a1e1 by Victor Stinner in branch '2.7':
Issue #8651: Fix "z#" format of PyArg_Parse*() function: the size was not
http://hg.python.org/cpython/rev/509f1c15a1e1
msg135034 - (view) Author: Roundup Robot (python-dev) Date: 2011-05-03 13:09
New changeset a0681e7a6ded by Victor Stinner in branch '2.7':
Issue #8651: PyArg_Parse*() functions raise an OverflowError if the file
http://hg.python.org/cpython/rev/a0681e7a6ded
msg135035 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-05-03 13:11
> Reopen the issue if you would like a port to 2.7
> (I am too lazy to do it)

I backported the fix to help issue #11277. While backporting the fix, I found another bug fixed by 509f1c15a1e1.
History
Date User Action Args
2011-05-03 13:11:10hayposetmessages: + msg135035
2011-05-03 13:09:32python-devsetmessages: + msg135034
2011-05-03 13:08:12python-devsetmessages: + msg135033
2011-03-21 02:30:24hayposetstatus: open -> closed

messages: + msg131609
resolution: fixed
nosy: loewis, pitrou, haypo, nadeem.vawda, rcoyner, python-dev
2011-03-21 02:26:21python-devsetnosy: + python-dev
messages: + msg131607
2011-02-21 19:27:37hayposetnosy: loewis, pitrou, haypo, nadeem.vawda, rcoyner
messages: + msg128983
2011-02-21 19:27:13hayposetnosy: loewis, pitrou, haypo, nadeem.vawda, rcoyner
messages: - msg128976
2011-02-21 19:19:02nadeem.vawdasetnosy: + nadeem.vawda
2011-02-21 18:26:44pitrousetnosy: loewis, pitrou, haypo, rcoyner
messages: + msg128976
2011-01-04 02:08:29hayposetnosy: loewis, pitrou, haypo, rcoyner
messages: + msg125282
2011-01-04 00:36:06pitrousetnosy: loewis, pitrou, haypo, rcoyner
messages: + msg125261
2010-05-29 15:51:54rcoynersetnosy: + rcoyner
2010-05-26 21:55:00hayposetmessages: + msg106583
2010-05-26 21:54:06hayposetfiles: + getarg.patch
keywords: + patch
messages: + msg106582
2010-05-07 17:24:54pitroucreate