Author serhiy.storchaka
Recipients Arfrever, eli.bendersky, jcon, meador.inge, ncoghlan, pitrou, serhiy.storchaka, tshepang
Date 2012-07-21.15:54:30
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <201207211854.14377.storchaka@gmail.com>
In-reply-to <1342865621.3492.1.camel@localhost.localdomain>
Content
> It's under 64-bit Linux, Intel Core i5 CPU. Are you sure you're testing
> in non-debug mode?

I use 32-bit Linux.

> That said, the numbers under Windows suggest me that Eli's original idea
> (append and then join at the end) would be more robust.

I agree, it would be more robust, but much more complex solution. I will try to implement this approach too. Victor Stinner in their experiments with formatting preferred 
overallocation approach (_PyUnicodeWriter), therefore, the solution probably will be a hybrid.

However, even in itself the patch deserves attention. It not only significant speeds up writing under Linux, but also speeds up the initialization, which leads to faster reading.

./python -m timeit -s "import io; d=(b'a'*99+b'\n')*10000"  "s=io.BytesIO(d); r=s.readline"  "while r(): pass"

Unpatched:  100 loops, best of 3: 5.92 msec per loop
Patched:  100 loops, best of 3: 3.95 msec per loop

I will try to preserve this advantage.

> By the way, here is Matt Mackall's take on realloc():

Note, that list also uses realloc() inside and therefore you get the same behavior with other constant factor. Actually, appending to a list less than sizeof(void*) bytes at a 
time you will run into this sooner. Perhaps this is the cause that append/join approach under Windows slower than under Linux.

Thank you for measurements, this shows the scale of the issue.
History
Date User Action Args
2012-07-21 15:54:31serhiy.storchakasetrecipients: + serhiy.storchaka, ncoghlan, pitrou, Arfrever, eli.bendersky, meador.inge, tshepang, jcon
2012-07-21 15:54:30serhiy.storchakalinkissue15381 messages
2012-07-21 15:54:30serhiy.storchakacreate