classification
Title: improve json decoding performance
Type: performance Stage: committed/rejected
Components: Library (Lib) Versions: Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, bob.ippolito, pitrou, rhettinger, vkuznet
Priority: normal Keywords: patch

Created on 2009-12-07 23:08 by pitrou, last changed 2010-09-04 20:17 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
json-opts2.patch pitrou, 2009-12-07 23:08
json-opts3.patch pitrou, 2009-12-08 00:54
json-opts4.patch pitrou, 2010-09-04 20:12
Messages (11)
msg96082 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-12-07 23:08
This patch does two things:
- it speeds up decoding of JSON objects by doing fewer temporary allocations
- it reduces memory use of decoded JSON objects by reusing key strings
when they appear several times in a JSON string
msg96093 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2009-12-08 00:08
The problem with this is that it makes json in Python diverge even more
from the simplejson trunk.
msg96097 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-12-08 00:42
Unless Bob applies it to simplejson as well, that is.
msg96098 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-12-08 00:53
I've just noticed that there is a persistent decoder instance in
`json._default_decoder`. To avoid keeping references to past keys
forever, this updated patch clears the memo dict when a string has
finished decoding.
msg96099 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-12-08 00:54
Here is the patch.
msg96169 - (view) Author: Valentin Kuznetsov (vkuznet) Date: 2009-12-09 15:23
I wonder if you can make a patch for 2.6 python branch.
msg96171 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-12-09 15:58
> I wonder if you can make a patch for 2.6 python branch.

No, 2.6 doesn't have the same C accelerator in the first place.
msg96903 - (view) Author: Bob Ippolito (bob.ippolito) * (Python committer) Date: 2009-12-27 11:10
I applied most of this patch to r206 of simplejson trunk
msg96908 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-12-27 12:02
Le dimanche 27 décembre 2009 à 11:10 +0000, Bob Ippolito a écrit :
> Bob Ippolito <bob@redivi.com> added the comment:
> 
> I applied most of this patch to r206 of simplejson trunk

Thank you. Will you port it to CPython yourself or would you prefer
someone else to do it?
msg115611 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-04 20:12
Updated patch against py3k.
msg115612 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-04 20:17
Committed in r84505.
History
Date User Action Args
2010-09-04 20:17:11pitrousetstatus: open -> closed
resolution: fixed
messages: + msg115612

stage: commit review -> committed/rejected
2010-09-04 20:12:45pitrousetfiles: + json-opts4.patch

stage: patch review -> commit review
messages: + msg115611
versions: - Python 2.7
2009-12-27 12:02:23pitrousetmessages: + msg96908
2009-12-27 11:10:33bob.ippolitosetmessages: + msg96903
2009-12-09 15:58:56pitrousetmessages: + msg96171
2009-12-09 15:23:23vkuznetsetmessages: + msg96169
2009-12-08 00:54:35pitrousetfiles: + json-opts3.patch

messages: + msg96099
2009-12-08 00:53:32pitrousetmessages: + msg96098
2009-12-08 00:42:47pitrousetmessages: + msg96097
2009-12-08 00:08:24benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg96093
2009-12-07 23:08:19pitroucreate