This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author lemburg
Recipients ajaksu2, amaury.forgeotdarc, collinwinter, eric.smith, ezio.melotti, gvanrossum, jafo, jimjjewett, lemburg, orivej, pitrou, rhettinger
Date 2009-06-04.10:43:55
SpamBayes Score 7.60503e-15
Marked as misclassified No
Message-id <4A27A56A.6030306@egenix.com>
In-reply-to <1244061829.5505.23.camel@localhost>
Content
Here's a new version of the unicode reference type, extended
to run in both Python 2.6 and 3.1:

http://downloads.egenix.com/python/unicoderef-0.0.2.tar.gz

I've also included a benchmark implemented in C which measures
Unicode/Bytes allocation performance at high resolution
and without any VM overhead.

These are the results on a 64-bit system using a regular UCS2
build of Python 2.6.1 and 3.1 on a single core AMD system:

 * Unicode allocation is always much slower than Bytes

 This depends a lot on how much data you have to handle. If the
 data fits into the pymalloc managed sizes, there's little
 difference.

 * CPU cache lines / memory boundaries obviously have some
   impact on the results

 If things are properly aligned, performance is better than
 in the unaligned case. Unfortunately, there's nothing much the
 CPython implementation can do to benefit from this, since the
 hardware layouts are too diverse.

 This explains why you see some unexpected figures in the
 results, e.g. Python 3.1, 4096 words, 32chars - Unicode
 being faster than Bytes.

 * Something must have improved in pymalloc

 The 4096 word tests perform better on 3.1 than on 2.6.

 Since such improvement have a great impact, more
 emphasis should be placed on these, e.g. by having pymalloc
 provide more space to fixed size objects such PyUnicodeObject,
 reducing the need to create new arenas.

 Dictionaries and other PyObjects such as instances objects
 would also benefit from such improvements.

Note that the benchmark scales the bytes word sizes to match
the internal allocation size of the Unicode objects, ie.
2chars maps to a length 4 bytes string on a UCS2 build.

Testing Unicode/Bytes Allocation Speed with Python 3.1rc1+
------------------------------------------------------------------------

=== 1024 words ============================================================

Variant 2chars, 1024 words, Unicode: 0.005503 seconds = 0.054 us per object
(best of 50 rounds)
Variant 2chars, 1024 words, Bytes  : 0.006838 seconds = 0.067 us per object
(best of 50 rounds)

Variant 9chars, 1024 words, Unicode: 0.008956 seconds = 0.087 us per object
(best of 50 rounds)
Variant 9chars, 1024 words, Bytes  : 0.007380 seconds = 0.072 us per object
(best of 50 rounds)

Variant 16chars, 1024 words, Unicode: 0.009257 seconds = 0.090 us per object
(best of 50 rounds)
Variant 16chars, 1024 words, Bytes  : 0.007291 seconds = 0.071 us per object
(best of 50 rounds)

Variant 32chars, 1024 words, Unicode: 0.009589 seconds = 0.094 us per object
(best of 50 rounds)
Variant 32chars, 1024 words, Bytes  : 0.007803 seconds = 0.076 us per object
(best of 50 rounds)

Variant pep100, 1024 words, Unicode: 0.010569 seconds = 0.103 us per object
(best of 50 rounds)
Variant pep100, 1024 words, Bytes  : 0.008198 seconds = 0.080 us per object
(best of 50 rounds)

=== 2048 words ============================================================

Variant 2chars, 2048 words, Unicode: 0.011624 seconds = 0.057 us per object
(best of 50 rounds)
Variant 2chars, 2048 words, Bytes  : 0.013941 seconds = 0.068 us per object
(best of 50 rounds)

Variant 9chars, 2048 words, Unicode: 0.018608 seconds = 0.091 us per object
(best of 50 rounds)
Variant 9chars, 2048 words, Bytes  : 0.014773 seconds = 0.072 us per object
(best of 50 rounds)

Variant 16chars, 2048 words, Unicode: 0.018556 seconds = 0.091 us per object
(best of 50 rounds)
Variant 16chars, 2048 words, Bytes  : 0.014550 seconds = 0.071 us per object
(best of 50 rounds)

Variant 32chars, 2048 words, Unicode: 0.018972 seconds = 0.093 us per object
(best of 50 rounds)
Variant 32chars, 2048 words, Bytes  : 0.016377 seconds = 0.080 us per object
(best of 50 rounds)

Variant pep100, 2048 words, Unicode: 0.021005 seconds = 0.103 us per object
(best of 50 rounds)
Variant pep100, 2048 words, Bytes  : 0.016636 seconds = 0.081 us per object
(best of 50 rounds)

=== 4096 words ============================================================

Variant 2chars, 4096 words, Unicode: 0.022950 seconds = 0.056 us per object
(best of 50 rounds)
Variant 2chars, 4096 words, Bytes  : 0.027813 seconds = 0.068 us per object
(best of 50 rounds)

Variant 9chars, 4096 words, Unicode: 0.037229 seconds = 0.091 us per object
(best of 50 rounds)
Variant 9chars, 4096 words, Bytes  : 0.031123 seconds = 0.076 us per object
(best of 50 rounds)

Variant 16chars, 4096 words, Unicode: 0.037118 seconds = 0.091 us per object
(best of 50 rounds)
Variant 16chars, 4096 words, Bytes  : 0.036433 seconds = 0.089 us per object
(best of 50 rounds)

Variant 32chars, 4096 words, Unicode: 0.040970 seconds = 0.100 us per object
(best of 50 rounds)
Variant 32chars, 4096 words, Bytes  : 0.051422 seconds = 0.126 us per object
(best of 50 rounds)

Variant pep100, 4096 words, Unicode: 0.049630 seconds = 0.121 us per object
(best of 50 rounds)
Variant pep100, 4096 words, Bytes  : 0.034551 seconds = 0.084 us per object
(best of 50 rounds)

Testing Unicode/Bytes Allocation Speed with Python 2.6.1
------------------------------------------------------------------------

=== 1024 words ============================================================

Variant 2chars, 1024 words, Unicode: 0.005810 seconds = 0.057 us per object
(best of 50 rounds)
Variant 2chars, 1024 words, Bytes  : 0.006918 seconds = 0.068 us per object
(best of 50 rounds)

Variant 9chars, 1024 words, Unicode: 0.008539 seconds = 0.083 us per object
(best of 50 rounds)
Variant 9chars, 1024 words, Bytes  : 0.007618 seconds = 0.074 us per object
(best of 50 rounds)

Variant 16chars, 1024 words, Unicode: 0.008478 seconds = 0.083 us per object
(best of 50 rounds)
Variant 16chars, 1024 words, Bytes  : 0.008097 seconds = 0.079 us per object
(best of 50 rounds)

Variant 32chars, 1024 words, Unicode: 0.008969 seconds = 0.088 us per object
(best of 50 rounds)
Variant 32chars, 1024 words, Bytes  : 0.008649 seconds = 0.084 us per object
(best of 50 rounds)

Variant pep100, 1024 words, Unicode: 0.009779 seconds = 0.096 us per object
(best of 50 rounds)
Variant pep100, 1024 words, Bytes  : 0.008318 seconds = 0.081 us per object
(best of 50 rounds)

=== 2048 words ============================================================

Variant 2chars, 2048 words, Unicode: 0.011745 seconds = 0.057 us per object
(best of 50 rounds)
Variant 2chars, 2048 words, Bytes  : 0.013978 seconds = 0.068 us per object
(best of 50 rounds)

Variant 9chars, 2048 words, Unicode: 0.017418 seconds = 0.085 us per object
(best of 50 rounds)
Variant 9chars, 2048 words, Bytes  : 0.014443 seconds = 0.071 us per object
(best of 50 rounds)

Variant 16chars, 2048 words, Unicode: 0.018138 seconds = 0.089 us per object
(best of 50 rounds)
Variant 16chars, 2048 words, Bytes  : 0.016200 seconds = 0.079 us per object
(best of 50 rounds)

Variant 32chars, 2048 words, Unicode: 0.021348 seconds = 0.104 us per object
(best of 50 rounds)
Variant 32chars, 2048 words, Bytes  : 0.018144 seconds = 0.089 us per object
(best of 50 rounds)

Variant pep100, 2048 words, Unicode: 0.019593 seconds = 0.096 us per object
(best of 50 rounds)
Variant pep100, 2048 words, Bytes  : 0.016692 seconds = 0.082 us per object
(best of 50 rounds)

=== 4096 words ============================================================

Variant 2chars, 4096 words, Unicode: 0.024347 seconds = 0.059 us per object
(best of 50 rounds)
Variant 2chars, 4096 words, Bytes  : 0.028411 seconds = 0.069 us per object
(best of 50 rounds)

Variant 9chars, 4096 words, Unicode: 0.041480 seconds = 0.101 us per object
(best of 50 rounds)
Variant 9chars, 4096 words, Bytes  : 0.031532 seconds = 0.077 us per object
(best of 50 rounds)

Variant 16chars, 4096 words, Unicode: 0.047986 seconds = 0.117 us per object
(best of 50 rounds)
Variant 16chars, 4096 words, Bytes  : 0.038006 seconds = 0.093 us per object
(best of 50 rounds)

Variant 32chars, 4096 words, Unicode: 0.062115 seconds = 0.152 us per object
(best of 50 rounds)
Variant 32chars, 4096 words, Bytes  : 0.049687 seconds = 0.121 us per object
(best of 50 rounds)

Variant pep100, 4096 words, Unicode: 0.044331 seconds = 0.108 us per object
(best of 50 rounds)
Variant pep100, 4096 words, Bytes  : 0.035939 seconds = 0.088 us per object
(best of 50 rounds)
History
Date User Action Args
2009-06-04 10:44:00lemburgsetrecipients: + lemburg, gvanrossum, collinwinter, rhettinger, jafo, jimjjewett, amaury.forgeotdarc, pitrou, eric.smith, ajaksu2, orivej, ezio.melotti
2009-06-04 10:43:58lemburglinkissue1943 messages
2009-06-04 10:43:55lemburgcreate