Author vkuznet
Recipients bob.ippolito, pitrou, rhettinger, swalker, vkuznet
Date 2009-12-02.18:47:05
SpamBayes Score 0.00119802
Marked as misclassified No
Message-id <1259779628.08.0.457204452131.issue6594@psf.upfronthosting.co.za>
In-reply-to
Content
Hi,
I'm sorry for delay, I was busy. Here is a test data file:
http://www.lns.cornell.edu/~vk/files/mangled.json

Its size is 150 MB, 50MB less of original, due to scrambled values I was 
forced to do.

The tests with stock json module in python 2.6.2 is 2GB
source = open('mangled.json', 'r')
data = json.load(source)

Using simplejson 2.0.9 from PyPi I saw the same performance, please note 
_speedups.so C module was compiled.

Using cjson module, I observed 180MB of RAM utilization
source = open('mangled.json', 'r')
data = cjson.encode(source.read())

cjson is about 10 times faster!

I re-factor code which deals with XML version of the same data and I was 
able to process it using cElementTree only using 20MB (!) of RAM.
History
Date User Action Args
2009-12-02 18:47:08vkuznetsetrecipients: + vkuznet, rhettinger, bob.ippolito, pitrou, swalker
2009-12-02 18:47:08vkuznetsetmessageid: <1259779628.08.0.457204452131.issue6594@psf.upfronthosting.co.za>
2009-12-02 18:47:06vkuznetlinkissue6594 messages
2009-12-02 18:47:05vkuznetcreate