Amazingly faster UTF-8 decoding

[AntoinePitrou]

A couple of cosmetic comments, I hope someone else can look at the patch in
detail. The performance numbers are quite promising.

http://bugs.python.org/review/14738/diff/4836/17107
File Objects/stringlib/codecs.h (right):

http://bugs.python.org/review/14738/diff/4836/17107#newcode113
Objects/stringlib/codecs.h:113: ch = (ch << 6) + ch2 - 030200;
I don't think octal literals are a good idea here. How about using the hex
representation?

http://bugs.python.org/review/14738/diff/4836/17107#newcode152
Objects/stringlib/codecs.h:152: ch = (ch << 12) + (ch2 << 6) + ch3 - 03420200;
Same comment.

http://bugs.python.org/review/14738/diff/4836/17107#newcode190
Objects/stringlib/codecs.h:190: ch = (ch << 18) + (ch2 << 12) + (ch3 << 6) + ch4
- 0362020200;
And here too.

http://bugs.python.org/review/14738/diff/4836/17112
File Objects/unicodeobject.c (right):

http://bugs.python.org/review/14738/diff/4836/17112#newcode4654
Objects/unicodeobject.c:4654: Py_LOCAL_INLINE(Py_ssize_t)
I don't think inlining is desired here.

[storchaka]

http://bugs.python.org/review/14738/diff/4836/17107
File Objects/stringlib/codecs.h (right):

http://bugs.python.org/review/14738/diff/4836/17107#newcode113
Objects/stringlib/codecs.h:113: ch = (ch << 6) + ch2 - 030200;
On 2012/05/06 23:08:10, AntoinePitrou wrote:
> I don't think octal literals are a good idea here. How about using the hex
> representation?

Octal literals is more suitable here because of shifts on 6 bits. Well, I'll
write it in the form of expression: ((0xC0 << 6) + 0x80).

http://bugs.python.org/review/14738/diff/4836/17112
File Objects/unicodeobject.c (right):

http://bugs.python.org/review/14738/diff/4836/17112#newcode4654
Objects/unicodeobject.c:4654: Py_LOCAL_INLINE(Py_ssize_t)
On 2012/05/06 23:08:10, AntoinePitrou wrote:
> I don't think inlining is desired here.

I was afraid that it will affect the performance, but forgot to check.
Now I checked and remove inline.

[AntoinePitrou]

The patch looks ok overall. I have a couple of questions.

http://bugs.python.org/review/14738/diff/4837/17114
File Objects/stringlib/codecs.h (right):

http://bugs.python.org/review/14738/diff/4837/17114#newcode165
Objects/stringlib/codecs.h:165: /* \xF0\x90\x80\80-\xF4\x8F\xBF\xBF --
10000-10FFFF */
Typo in comment? It looks like it should be "\xF0\x80\x80\80", not
"\xF0\x90\x80\80".

http://bugs.python.org/review/14738/diff/4837/17119
File Objects/unicodeobject.c (right):

http://bugs.python.org/review/14738/diff/4837/17119#newcode4772
Objects/unicodeobject.c:4772: while (endinpos < size && (starts[endinpos] &
0xC0) == 0x80)
Why do you need this, instead of "endinpos = end"?

http://bugs.python.org/review/14738/diff/4837/17119#newcode4829
Objects/unicodeobject.c:4829: wchar_t *unicode;
I don't have a Mac, did you actually check this code compiles and works?

[storchaka]

http://bugs.python.org/review/14738/diff/4837/17114
File Objects/stringlib/codecs.h (right):

http://bugs.python.org/review/14738/diff/4837/17114#newcode165
Objects/stringlib/codecs.h:165: /* \xF0\x90\x80\80-\xF4\x8F\xBF\xBF --
10000-10FFFF */
On 2012/05/09 17:45:57, AntoinePitrou wrote:
> Typo in comment? It looks like it should be "\xF0\x80\x80\80", not
> "\xF0\x90\x80\80".

Yes, there's a typo, but not the one you think. Missing "x" in last "\x80".

See comment below -- range \xF0\x80\x80\x80-\xF0\x80\xBF\xBF is invalid.

http://bugs.python.org/review/14738/diff/4837/17114#newcode184
Objects/stringlib/codecs.h:184: \xF0\x80\x80\80-\xF0\x80\xBF\xBF -- fake
0000-FFFF */
Typo in comment -- missing "x" in last "\x80".

http://bugs.python.org/review/14738/diff/4837/17119
File Objects/unicodeobject.c (right):

http://bugs.python.org/review/14738/diff/4837/17119#newcode4772
Objects/unicodeobject.c:4772: while (endinpos < size && (starts[endinpos] &
0xC0) == 0x80)
On 2012/05/09 17:45:57, AntoinePitrou wrote:
> Why do you need this, instead of "endinpos = end"?

What if last three bytes are '\xF0\x90\x41'? \xF0 requires 4 bytes, '\xF0\x90'
is incompleted sequence (and can be ignored or replaced), but \x41 is valid
encoding of "A".

http://bugs.python.org/review/14738/diff/4837/17119#newcode4829
Objects/unicodeobject.c:4829: wchar_t *unicode;
On 2012/05/09 17:45:57, AntoinePitrou wrote:
> I don't have a Mac, did you actually check this code compiles and works?

At least it compiles (I checked, removing __APPLE__ guard). I also don't have a
Mac.