classification
Title: Get rid of dangerous integer overflow tricks
Type: behavior Stage: resolved
Components: Extension Modules, Interpreter Core Versions: Python 3.3, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Arfrever, asvetlov, mark.dickinson, python-dev, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2012-09-30 18:34 by serhiy.storchaka, last changed 2013-01-26 18:56 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
size_overflow-3.3.patch serhiy.storchaka, 2012-09-30 18:34 Patch for 3.3 and 3.4 review
size_overflow-3.2.patch serhiy.storchaka, 2012-09-30 18:34 Patch for 3.2 review
size_overflow-2.7.patch serhiy.storchaka, 2012-09-30 18:35 Patch for 2.7 review
size_overflow-2.7_2.patch serhiy.storchaka, 2013-01-15 11:14 review
size_overflow-3.2_2.patch serhiy.storchaka, 2013-01-15 11:14 review
Messages (16)
msg171657 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-09-30 18:34
In several places such dungerous code used to check the integer overflow:

  size = n * itemsize;
  if (size / itemsize != n) raise exception...

Because these values are signed, this results in undefined behavior.

The proposed patches replace similar unsafe code to safe one. Note that the patches for the different versions are substantially different.
msg171993 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-10-04 20:04
The patches looks good for me, but I like to double check before commit.
Let's wait for a week for other reviewers.
msg171994 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2012-10-04 20:19
It's maybe safer (and simpler) to not touch such code in Python older than 3.4.
msg172215 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-10-06 16:14
> It's maybe safer (and simpler) to not touch such code in Python
> older than 3.4.

So far, I've been fixing these overflow bugs only in the development branches, unless they can be shown to cause actual bugs.  That said, I think it's probably okay to apply these for 3.3 as well as 3.4, especially since the 3.3 patch is smaller than the others.  I'll review and apply.
msg172218 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-10-06 16:33
> especially since the 3.3 patch is smaller than the others.

It's becouse 3.3 already contains some fixes which was not be backported to 
older versions.
msg172219 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-10-06 16:39
> It's becouse 3.3 already contains some fixes which was not be backported
> to older versions.

Yes, exactly!  That's what I meant when I said:

"So far, I've been fixing these overflow bugs only in the development branches"

There were lots of integer overflow occurrences like these found by John Regehr in issue 9530.  I chose to fix those only in the current development branch, which was 3.3 at the time.  Since we've made an effort to clean up 3.3 in that respect, I think it's worth finishing that job off by applying your patch both to 3.3 and 3.4.
msg172220 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-10-06 16:42
> unless they can be shown to cause actual bugs.

See issue14700.
msg172221 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-10-06 16:54
Serhiy, I don't understand what you're getting at.  Can you explain?
msg172223 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-10-06 17:05
New changeset 152d85b2da3a by Mark Dickinson in branch '3.3':
Issue #16096: Fix several occurrences of potential signed integer overflow.  Thanks Serhiy Storchaka.
http://hg.python.org/cpython/rev/152d85b2da3a

New changeset faae99459b43 by Mark Dickinson in branch 'default':
Issue #16096: Merge fixes from 3.3.
http://hg.python.org/cpython/rev/faae99459b43
msg172224 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-10-06 17:07
Applied the 3.3 patch to 3.3 and default, with some minor changes:

 - revert the Objects/longobject.c changes, since they don't depend
   on signed overflow

 - fix the second change in Objects/tupleobject.c so that the overflow check happens before the multiplication rather than after.
msg172227 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-10-06 17:32
Whoops.  I take it back about the Objects/longobject.c bit.  Fixing ...
msg172228 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-10-06 17:50
New changeset 906ae6485cb8 by Mark Dickinson in branch '3.3':
Issue #16096: Fix signed overflow in Objects/longobject.c.  Thanks Serhiy Storchaka.
http://hg.python.org/cpython/rev/906ae6485cb8

New changeset b728aac3bdb3 by Mark Dickinson in branch 'default':
Issue #16096: port fix from 3.3
http://hg.python.org/cpython/rev/b728aac3bdb3
msg172229 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-10-06 17:58
In issue14700 were fixed two actual bugs. The fix was not be backported to older 
versions (and this changes included in patches for this issue). I think it is 
better to reopen issue14700 for backporting fixes to 2.7 and 3.2?
msg172283 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-10-07 09:41
Yes, reopening issue 14700 sounds good to me.

I'm not against fixing these issues in the bugfix branches, but we need to do it carefully (which unfortunately probably also means slowly).  I think that for the bugfix branches, each fix should be accompanied by a test that exercises the original bug.  I'd also suggest having a separate issue for each bug, for ease of review.

I'd probably also prioritise those bugs that can be triggered without having huge structures in memory:  e.g., the issue 14700 bug seems more important to fix than the PyTuple_New bug.
msg180008 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-01-15 11:14
Here are updated to current codebase patches for 2.7 and 3.2. It seems that 
all the rest of overflows are hypothetical bugs and do not appear on the 
current supported platforms. Fix them is not necessary (rather for purity). If 
no one can see visible bugs, I'll close this issue soon.
msg180699 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-01-26 18:56
I withdraw my patches for 2.7 and 3.2 due to the fact that they have no visible effect on supported platforms. Patches for 3.3+ already committed, therefore I close this issue.
History
Date User Action Args
2013-01-26 18:56:30serhiy.storchakasetstatus: open -> closed
versions: + Python 3.3, Python 3.4, - Python 2.7, Python 3.2
messages: + msg180699

resolution: fixed
stage: resolved
2013-01-15 11:14:42serhiy.storchakasetfiles: + size_overflow-2.7_2.patch, size_overflow-3.2_2.patch

messages: + msg180008
2013-01-07 18:33:46serhiy.storchakasetassignee: serhiy.storchaka
2012-10-28 10:33:24mark.dickinsonsetassignee: mark.dickinson -> (no value)
2012-10-07 11:16:58mark.dickinsonsetversions: - Python 3.3, Python 3.4
2012-10-07 09:41:21mark.dickinsonsetmessages: + msg172283
2012-10-07 02:55:45Arfreversetnosy: + Arfrever
2012-10-06 17:58:44serhiy.storchakasetmessages: + msg172229
2012-10-06 17:50:33python-devsetmessages: + msg172228
2012-10-06 17:32:14mark.dickinsonsetmessages: + msg172227
2012-10-06 17:07:22mark.dickinsonsetmessages: + msg172224
2012-10-06 17:05:31python-devsetnosy: + python-dev
messages: + msg172223
2012-10-06 16:54:14mark.dickinsonsetmessages: + msg172221
2012-10-06 16:42:13serhiy.storchakasetmessages: + msg172220
2012-10-06 16:39:37mark.dickinsonsetmessages: + msg172219
2012-10-06 16:33:18serhiy.storchakasetmessages: + msg172218
2012-10-06 16:14:41mark.dickinsonsetassignee: mark.dickinson
messages: + msg172215
2012-10-04 20:19:18vstinnersetmessages: + msg171994
2012-10-04 20:13:43vstinnersetnosy: + vstinner
2012-10-04 20:04:16asvetlovsetnosy: + asvetlov
messages: + msg171993
2012-09-30 18:35:34serhiy.storchakasetfiles: + size_overflow-2.7.patch
2012-09-30 18:34:50serhiy.storchakasetfiles: + size_overflow-3.2.patch
2012-09-30 18:34:11serhiy.storchakacreate