On a 8GB RAM box (more than 6GB free), serializing many small objects can eat all memory, while the end result would take around 600MB on an UCS2 build:

$ LANG=C time opt/python -c "import json; l = [1] * (100*1024*1024); encoded = json.dumps(l)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/antoine/cpython/opt/Lib/json/__init__.py", line 224, in dumps
    return _default_encoder.encode(obj)
  File "/home/antoine/cpython/opt/Lib/json/encoder.py", line 188, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/antoine/cpython/opt/Lib/json/encoder.py", line 246, in iterencode
    return _iterencode(o, 0)
MemoryError
Command exited with non-zero status 1
11.25user 2.43system 0:13.72elapsed 99%CPU (0avgtext+0avgdata 27820320maxresident)k
2920inputs+0outputs (12major+1261388minor)pagefaults 0swaps


I suppose the encoder internally builds a large list of very small unicode objects, and only joins them at the end. Probably we could join it by chunks so as to avoid this behaviour.

I think this is because dumps() uses the C encoder. Making the C encoder incremental (i.e. iterator-based) like the Python encoder would solve this.

I actually looked into doing this for issue #12134, but it didn't seem so simple; Since C has no yield, I think the iterator would need to maintain its own stack to keep track of where it is in the object tree it's encoding...

If there is interest though, I may be able to write a patch when I have some time off again...

This patch does the trick.

> I actually looked into doing this for issue #12134, but it didn't seem 
> so simple; Since C has no yield, I think the iterator would need to
> maintain its own stack to keep track of where it is in the object tree 
> it's encoding...

The encoder doesn't have to be turned into an iterator. It would just need to call a given callable (fp.write) at regular intervals and that would be enough to C-accelerate dump().

My patch actually provides a good foundation for this.

New changeset 47176e8d7060 by Antoine Pitrou in branch 'default':
Issue #12778: Reduce memory consumption when JSON-encoding a large container of many small objects.
http://hg.python.org/cpython/rev/47176e8d7060

> It would just need to call a given callable (fp.write) at regular intervals and that would be enough to C-accelerate dump().

True, but that would just special case dump(), just like dumps() is special-cased now. Ideally JSONEncoder.iterencode() would be accelerated, so you wouldn't need any special cases. Or deprecate iterencode() and replace it with a callback interface...

> > It would just need to call a given callable (fp.write) at regular
> intervals and that would be enough to C-accelerate dump().
> 
> True, but that would just special case dump(), just like dumps() is
> special-cased now. Ideally JSONEncoder.iterencode() would be
> accelerated, so you wouldn't need any special cases. Or deprecate
> iterencode() and replace it with a callback interface...

Is iterencode() used much? I would think dump() and dumps() see the most
use.

> Is iterencode() used much? I would think dump() and dumps() see the most use.

Of course. I'd just prefer an elegant & complete solution. But I agree accelerating just dump() would already be much better than the current situation.