Message 143598 - Python tracker

➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author	pitrou
Recipients	mark.dickinson, pitrou, rhettinger
Date	2011-09-06.12:20:08
SpamBayes Score	0.0013593889
Marked as misclassified	No
Message-id	<1315311609.99.0.705451675521.issue12911@psf.upfronthosting.co.za>
In-reply-to

Content
In 47176e8d7060, I fixed json to not blow memory when serializing a large container of small objects. It turns out that the repr() of tuple objects (and, very likely, list objects and possibly other containers) has the same problem. For example, Martin's 256GB machine can't serialize a two billion-element tuple: http://www.python.org/dev/buildbot/all/builders/AMD64%20debian%20parallel%20custom/builds/6/steps/test/logs/stdio So I propose to expose a private C API for the "accumulator" pattern introduced in 47176e8d7060 (with, e.g., the _PyAccumulator prefix), and to use that API from relevant code.

In 47176e8d7060, I fixed json to not blow memory when serializing a large container of small objects.
It turns out that the repr() of tuple objects (and, very likely, list objects and possibly other containers) has the same problem. For example, Martin's 256GB machine can't serialize a two billion-element tuple:
http://www.python.org/dev/buildbot/all/builders/AMD64%20debian%20parallel%20custom/builds/6/steps/test/logs/stdio

So I propose to expose a private C API for the "accumulator" pattern introduced in 47176e8d7060 (with, e.g., the _PyAccumulator prefix), and to use that API from relevant code.

History
Date	User	Action	Args
2011-09-06 12:20:10	pitrou	set	recipients: + pitrou, rhettinger, mark.dickinson
2011-09-06 12:20:09	pitrou	set	messageid: <1315311609.99.0.705451675521.issue12911@psf.upfronthosting.co.za>
2011-09-06 12:20:09	pitrou	link	issue12911 messages
2011-09-06 12:20:09	pitrou	create