This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author neologix
Recipients amaury.forgeotdarc, daniel.urban, docs@python, dsvensson, loewis, neologix, pitrou, vstinner
Date 2011-08-18.14:02:27
SpamBayes Score 2.129384e-06
Marked as misclassified No
Message-id <1313676148.95.0.290979143574.issue12775@psf.upfronthosting.co.za>
In-reply-to
Content
There's been some work done on the GC some time ago to address this type of pattern (I think to reduce from quadratic complexity to amortized linear complexity). I'm adding Antoine and Martin who were involved in this.

> Here is a short python script that demonstrates the issue, the JSON
> file in this case is 1.2GB large:

A couple remarks:
- please indicate the Python version you're using
- you should put your test in a script, one with gc.disable() at the top, to avoid using any cache that might be used internally by cjson
- you should perform "echo 3 > /proc/sys/vm/drop_cache" before each run to make sure you start with a cold page/buffer cache (see how fast your file got read the second time you called read_json_blob()?)
History
Date User Action Args
2011-08-18 14:02:29neologixsetrecipients: + neologix, loewis, amaury.forgeotdarc, pitrou, vstinner, dsvensson, daniel.urban, docs@python
2011-08-18 14:02:28neologixsetmessageid: <1313676148.95.0.290979143574.issue12775@psf.upfronthosting.co.za>
2011-08-18 14:02:28neologixlinkissue12775 messages
2011-08-18 14:02:27neologixcreate