There are a couple of aliasing warnings in non-debug mode. For example:

http://www.python.org/dev/buildbot/all/builders/x86%20Gentoo%20Non-Debug%203.x/builds/1741


Objects/object.c:293: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
Objects/object.c:302: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result
Objects/unicodeobject.c:5533: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
Objects/unicodeobject.c:5533: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
Objects/unicodeobject.c:5533: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
Objects/unicodeobject.c:5523: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
Objects/unicodeobject.c:5523: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
cc1: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
cc1: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
cc1: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
Objects/unicodeobject.c:5523: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
Objects/unicodeobject.c:5523: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
Objects/unicodeobject.c:5533: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
Objects/unicodeobject.c:5533: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
Objects/unicodeobject.c:5533: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
cc1: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
cc1: warning: dereferencing pointer 'pblock' does break strict-aliasing rules
cc1: warning: dereferencing pointer 'pblock' does break strict-aliasing rules

gcc 4.5 doesn't warn for me. Is this a compiler bug in 4.4 or 4.5? That is, are these actual aliasing violations?

Benjamin Peterson <report@bugs.python.org> wrote:
> gcc 4.5 doesn't warn for me. Is this a compiler bug in 4.4 or 4.5?
> That is, are these actual aliasing violations?

I see this with 4.4 but also with 4.6 when using -Wstrict-aliasing=2.
However, nothing bad happens when I compile with -fstrict-aliasing.

I think, but I would have to consult the standard again to be sure, that
technically this might be an aliasing violation in C99, since 'pblock' does
not have the type of 'block'.

     unsigned long block = * (unsigned long *) _q;
     unsigned short *pblock = (unsigned short*)&block;

Attached patch uses an union to make the compiler warning quiet. It should not speed up Python because the function already ensures that the pointer is aligned to the size of a long. It may slow down the function, I don't know gcc enough to guess exactly the impact on performances.

An alternative is to use __attribute__((__may_alias__)), a GCC specific attribute. I don't know the impact on performances of this attribute.

What if add more hacking? If long integer already used for buffering and checking, let use it for swapping and splitting too.

With my patch (attached) codecs.utf_16_be_decode runs 5% faster (on 32-bit Linux, I was not tested 64-bit). And of cause no pointers -- no aliasing warnings.

> With my patch (attached) codecs.utf_16_be_decode runs 5% faster (on 32-bit Linux, I was not tested 64-bit). And of cause no pointers -- no aliasing warnings.

Your patch is wrong: you need to use & 0xffff to get lower 16 bits
when reading a UTF-16 unit. For example, (Py_UCS2)(block >> 32) should
be written (Py_UCS2)((block >> 32) & 0xffff).

Heh. This was in previous version of my patch. I have removed '& 0xFFFFu' and parents for simplicity. GCC produces same binaries for both sources. But you can return it back. It has effect only on plathforms with non-16-bit short, but now Python not supports they (Python is not well portable on exotic plathforms).

> It has effect only on plathforms with non-16-bit short

The problem is for 64-bit long: "long >> 32" returns the 32 higher
bits (32..64), not 16 bits (32..48).

"(unsigned short)(long >> 32)" returns the 16 bits (32..48) if short is 16-bit. I agree that this variant is more strict and reliable (and this was my original version) and if you do not find it verbose and redundant, so be it. The difference will be noticeable only on a very exotic platform (with a 9-bit chars, for example), where the original code also will not work. Frankly, in this straightforward patch hacking is less than in the original code.

I made a mistake with the microbenchmark. In fact, acceleration is not 5%, but 20-40%.

./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = (" " * 1000).encode("utf-16be")' 'd(x)'
./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = ("\u263A" * 1000).encode("utf-16be")' 'd(x)'

I'm sorry. Here is the corrected patch for big-endian plathform.

New changeset 2c514c382a2a by Victor Stinner in branch 'default':
Close #14249: Use an union instead of a long to short pointer to avoid aliasing
http://hg.python.org/cpython/rev/2c514c382a2a

Result of the benchmark before/after my commit. I prefer an unit over manually manipulate long as short or bytes, because I think that the compiler knows better how to optimize operations on integers.

unpatched:

$ ./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = (" " * 1000).encode("utf-16be")' 'd(x)'
100000 loops, best of 3: 4.64 usec per loop
$ ./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = ("\u263A" * 1000).encode("utf-16be")' 'd(x)'
100000 loops, best of 3: 5.87 usec per loop

patched:

$ ./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = (" " * 1000).encode("utf-16be")' 'd(x)'
100000 loops, best of 3: 3.53 usec per loop
$ ./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = ("\u263A" * 1000).encode("utf-16be")' 'd(x)'
100000 loops, best of 3: 4.85 usec per loop

What compiler are you using? With gcc 4.4 on 32-bit Linux netbook I get:

                         unpatched  union  shift
utf-16le  " "*10000         129      126    109
utf-16le  "\u263A"*10000    208      203    160
utf-16be  " "*10000         153      147    114
utf-16be  "\u263A"*10000    226      227    167

The difference is that for shift the compiler stores block in register, and for the union the compiler stores block in memory, so that it can get address. May be more recent compilers learned to do this more effectively?

Besides, shifts are more pronounced for CPython code, searching shows very few uses of union in the source code.

On 64-bit Linux with gcc-4.4 I get:

Unpatched:

$ ./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = (" " * 1000).encode("utf-16be")' 'd(x)'
100000 loops, best of 3: 4.1 usec per loop
$ ./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = ("\u263A" * 1000).encode("utf-16be")' 'd(x)'
100000 loops, best of 3: 5.87 usec per loop

2c514c382a2a:

$ ./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = (" " * 1000).encode("utf-16be")' 'd(x)'
100000 loops, best of 3: 3.68 usec per loop
$ ./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = ("\u263A" * 1000).encode("utf-16be")' 'd(x)'
100000 loops, best of 3: 4.72 usec per loop

utf16_decoder_shift_3.patch:

$ ./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = (" " * 1000).encode("utf-16be")' 'd(x)'
100000 loops, best of 3: 2.23 usec per loop
$ ./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = ("\u263A" * 1000).encode("utf-16be")' 'd(x)'
100000 loops, best of 3: 3.11 usec per loop

Linux, 64-bit, Intel Core i5 2500:

-> unpatched:

$ ./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = (" " * 1000).encode("utf-16be")' 'd(x)'
100000 loops, best of 3: 2.99 usec per loop

-> Victor's commit:

$ ./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = (" " * 1000).encode("utf-16be")' 'd(x)'
100000 loops, best of 3: 2.83 usec per loop

-> utf16_decoder_shift_3.patch:

$ ./python -m timeit -s 'import codecs; d = codecs.utf_16_be_decode; x = (" " * 1000).encode("utf-16be")' 'd(x)'
1000000 loops, best of 3: 1.88 usec per loop


It seems that the wrong patch was committed.

New changeset 489f252b1f8b by Victor Stinner in branch 'default':
Close #14249: Use bit shifts instead of an union, it's more efficient.
http://hg.python.org/cpython/rev/489f252b1f8b

Ok, benchmarks have spoken, amen. I applied Serhiy Storchaka's patch (version 3). I just replaced expressions in calls to Py_MAX by variables: Py_MAX is a macro and it may have to compute each expression twice. I didn't check if it's more or less efficient. Thanks Serhiy for your contribution! I added your name to Misc/ACKS.

We may change Py_MIN/Py_MAX to something more efficient using typeof():

     #define max(a,b) \
       ({ typeof (a) _a = (a); \
           typeof (b) _b = (b); \
         _a > _b ? _a : _b; })

I don't know which C compilers support it, but gcc does, at least.

> I just replaced expressions in calls to Py_MAX by variables: Py_MAX is a macro and it may have to compute each expression twice.

gcc computes those values only once. It even caches them for use in
PyUnicode_WRITE. But other compilers may not be so smart.

Instead of Py_MAX(a,b) here you can use a|b. In theory this should be
more efficient, but I couldn't see the difference even with microscope.

However, all this does not matter, soon I will submit complex patch,
which speeds up the utf-16 decoder in 2-5 times.