classification
Title: JSON crashes during encoding resized lists
Type: crash Stage: resolved
Components: Extension Modules Versions: Python 3.4, Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, bob.ippolito, ezio.melotti, jcea, pitrou, python-dev, rhettinger, serhiy.storchaka
Priority: normal Keywords: needs review, patch

Created on 2012-10-14 09:56 by serhiy.storchaka, last changed 2012-11-01 20:03 by asvetlov. This issue is now closed.

Files
File name Uploaded Description Edit
json_encode_resized_list.patch serhiy.storchaka, 2012-10-14 10:02 review
Messages (14)
msg172867 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-10-14 09:56
JSON encoding crash if the source list resized in process of encoding. This can be happen unintentionally in multithreaded code.

Simple crash code:

import json
a = [object()] * 10
def crasher(obj):
    del a[-1]

json.dumps(a, default=crasher)
msg172869 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-10-14 10:02
Here is a patch.
msg174414 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-11-01 13:19
The patch LGTM except I cannot reproduce crash on unmodified sources with running applied test.
msg174422 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-11-01 14:48
Of course, this kind of bugs can cause unpredictable behavior, they do not have to lead to an immediate crash.  This depends from the platform, the compiler and its options.  On my computers the test always crashed, this is the maximum that I can say.
msg174439 - (view) Author: Roundup Robot (python-dev) Date: 2012-11-01 19:04
New changeset 3672db224eb3 by Antoine Pitrou in branch '3.2':
Issue #16228: Fix a crash in the json module where a list changes size while it is being encoded.
http://hg.python.org/cpython/rev/3672db224eb3

New changeset 7528c02b8d52 by Antoine Pitrou in branch '3.3':
Issue #16228: Fix a crash in the json module where a list changes size while it is being encoded.
http://hg.python.org/cpython/rev/7528c02b8d52

New changeset ab65509b8443 by Antoine Pitrou in branch 'default':
Issue #16228: Fix a crash in the json module where a list changes size while it is being encoded.
http://hg.python.org/cpython/rev/ab65509b8443
msg174442 - (view) Author: Roundup Robot (python-dev) Date: 2012-11-01 19:08
New changeset 33ae62a4ecf5 by Antoine Pitrou in branch '2.7':
Issue #16228: Fix a crash in the json module where a list changes size while it is being encoded.
http://hg.python.org/cpython/rev/33ae62a4ecf5
msg174443 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-11-01 19:08
Thank you for noticing this, and for writing a patch.
msg174445 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-11-01 19:19
Hi, I'm ok with patch but I guess to add some comment in C code to prevent, hmm, back optimization from upcoming contributor. 
The same for #16230
msg174451 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-11-01 19:36
> Hi, I'm ok with patch but I guess to add some comment in C code to
> prevent, hmm, back optimization from upcoming contributor. 
> The same for #16230

But that's what tests are for.
msg174454 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-11-01 19:38
As I said tests was not crashed on my linux 64 bit Ubuntu with buggy code.
msg174455 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-11-01 19:42
> As I said tests was not crashed on my linux 64 bit Ubuntu with buggy code.

They crashed duly here in debug mode (64-bit Linux).
msg174461 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-11-01 19:57
I have python built with Py_DEBUG (./configure --with-pydebug)
What I' missed?
I'm ok with status quo but just want to understand how to configure my build properly.
msg174465 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-11-01 20:01
> I have python built with Py_DEBUG (./configure --with-pydebug)
> What I' missed?
> I'm ok with status quo but just want to understand how to configure my build properly.

Well, I don't know. Perhaps a different compiler version, a different
libc, or perhaps you got (un)lucky.
msg174467 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2012-11-01 20:03
Ok. Thanks.
History
Date User Action Args
2012-11-01 20:03:22asvetlovsetmessages: + msg174467
2012-11-01 20:01:50pitrousetmessages: + msg174465
2012-11-01 19:57:39asvetlovsetmessages: + msg174461
2012-11-01 19:42:49pitrousetmessages: + msg174455
2012-11-01 19:38:00asvetlovsetmessages: + msg174454
2012-11-01 19:36:18pitrousetmessages: + msg174451
2012-11-01 19:19:10asvetlovsetmessages: + msg174445
2012-11-01 19:08:54pitrousetstatus: open -> closed
resolution: fixed
messages: + msg174443

stage: patch review -> resolved
2012-11-01 19:08:20python-devsetmessages: + msg174442
2012-11-01 19:04:06python-devsetnosy: + python-dev
messages: + msg174439
2012-11-01 14:48:07serhiy.storchakasetmessages: + msg174422
2012-11-01 13:19:46asvetlovsetnosy: + asvetlov
messages: + msg174414
2012-10-24 08:59:29serhiy.storchakasetstage: patch review
2012-10-21 17:17:30serhiy.storchakasetkeywords: + needs review
2012-10-15 02:04:14jceasetnosy: + jcea
2012-10-14 10:02:19serhiy.storchakasetfiles: + json_encode_resized_list.patch
keywords: + patch
messages: + msg172869
2012-10-14 09:56:18serhiy.storchakacreate