This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author neologix
Recipients BreamoreBoy, amaury.forgeotdarc, anglocelt, elachuni, marcio, mcicogni, neologix, pitrou, r.david.murray, taukki, trogdorsey, vila
Date 2011-05-22.10:45:41
SpamBayes Score 0.0
Marked as misclassified No
Message-id <1306061142.36.0.515603172758.issue1441530@psf.upfronthosting.co.za>
In-reply-to
Content
Digging a little deeper, here's the conclusion:
- with py3k, fragmentation is less likely: the buffered reader returned by makefile() ensures that we can allocate only one result buffer for the total number of bytes read() (thanks to socket's readinto()). It's also faster. Finally, since buffered read() guarantees to return exactly the number of bytes asked, there's no need to call it repeatedly with min(size-read, 4096): it's slower, and actually makes fragmentation more likely.
So I've attached a patch removing the looping from py3k IMAP4 read() method, which leads to simpler, faster and fragmentation-less code.
- for 2.7, it's also better to wrap the SSL socket with makefile(): the leads to simpler code (no need to loop), and also I noticed that socket's _fileobject's (returned by makefile()) read() and readline() methods already use a StringIO to avoid fragmentation.
So I've attached a second patch wrapping 2.7 IMAP4_SSL socket with makefile.

While I can't reproduce this issue on my system, I've analyzed malloc/realloc/free calls using ltrace, and I'm pretty confident this should solve fragmentation issues for both 2.7 and py3k.

By the way, nice work with the I/O stack in py3k, the new layer allows much more efficient code (reduced allocations and copies)!
History
Date User Action Args
2011-05-22 10:45:42neologixsetrecipients: + neologix, mcicogni, amaury.forgeotdarc, pitrou, taukki, vila, elachuni, anglocelt, r.david.murray, marcio, trogdorsey, BreamoreBoy
2011-05-22 10:45:42neologixsetmessageid: <1306061142.36.0.515603172758.issue1441530@psf.upfronthosting.co.za>
2011-05-22 10:45:41neologixlinkissue1441530 messages
2011-05-22 10:45:41neologixcreate