The patch fixes the output for profile and cProfile. Another patch from
Alexandre and me added additional calls to the UTF-8 codec.

There is no patch.  =)

Sure there is a patch ... well it's ... *uhm* ... it's hidden under your
bed. O:-)

The new patch does a far more better job. I had the idea after a
discussion with Alexandre on #python and a small debugging session.

The tests for profile, cProfile and doctest are failing on my Linux box
because Py_FileSystemDefaultEncoding is "UTF-8" but PyUnicode_Decode()
has a shortcut for "utf-8" only. I've added a tolower loop and shortcuts
for UTF-16 and UTF-32 to PyUnicode_Decode(). 

I've also disabled the tests on system with a fs encoding that isn't
built in.

I don't think it's possible to add shortcuts in PyUnicode_Decode for
UTF-16 and UTF-32 because the byte-order can be different depending of
the platform. So, these two need to pass through the codecs module.

I am sure if it's better, but I factored out the normalization routine
into its own function.

Alexandre Vassalotti wrote:
> I don't think it's possible to add shortcuts in PyUnicode_Decode for
> UTF-16 and UTF-32 because the byte-order can be different depending of
> the platform. So, these two need to pass through the codecs module.

utf-16 and utf-32 are the the names for the native codecs. The explicit
names are e.g. utf-16-be or utf-32-le. The last argument 0 also means
"native byte order".

I used a shorter algorithm to optimize the normalization for the special
cases of the strcmp() shortcuts. Your version is fine but takes several
CPU cycles longer. I don't think it has a large performance impact. ;)

Christian

I like Chris's patch better, but he should use (sizeof(lower)-1) instead
of 8, and he should use the TOLOWER macro from bytes_methods.h.

Please submit a new one and I'll check it in!

Why should I use (sizeof(lower)-1)? Do you mean
PyMem_Malloc(strlen(encoding) + 1)?

Changes since last patch:
* added #include "bytes_methods.h" in unicodeobject.c
* fix all is* to use the macros from bytes_methods.h
* use malloc/free the lower version instead of using a fixed buffer
* replace '_' with '-' in encoding

I still think that a fixed buffer of 12 chars (strlen(iso-8859-1) + 1 +
\0) is sufficient and faster.
* also check for iso-8859-1

Committed revision 58659.

I'm sorry I confused you; I was fine with the version that has char
lower[N] but I wanted you to not repeat N-1 later in the code. See what
I checked in. :-)

Guido van Rossum wrote:
> Guido van Rossum added the comment:
> 
> Committed revision 58659.
> 
> I'm sorry I confused you; I was fine with the version that has char
> lower[N] but I wanted you to not repeat N-1 later in the code. See what
> I checked in. :-)

I still don't understand why you are using (sizeof lower) - 2 and what
&lower[(sizeof lower) - 2] returns. Is it the memory address of lower[17]?

> I still don't understand why you are using (sizeof lower) - 2 

It is simply to avoid duplicating the constant (a.k.a. the Don't Repeat
Yourself (DRY) rule).

> and what &lower[(sizeof lower) - 2] returns. Is it the memory address
> of lower[17]?

It the address of lower[18] to be exact. (l < &lower[(sizeof lower) -
2]) is simply tricky notation to check the bound of the array.
Personally, I used like to subtract pointer, ((lower - l + 1) < (sizeof
lower)) to get the bound. But now, I find Guido's trick more cute (and
less error-prone). :)

> Personally, I used like to subtract pointer, ((lower - l + 1) < 
> (sizeof lower)) to get the bound.

Doh. I got it backward. It's (l - lower + 1), not (lower - l + 1).

> But now, I find Guido's trick more cute (and less error-prone).

At least, I got that right. :)

Alexandre Vassalotti wrote:
> It the address of lower[18] to be exact. (l < &lower[(sizeof lower) -
> 2]) is simply tricky notation to check the bound of the array.
> Personally, I used like to subtract pointer, ((lower - l + 1) < (sizeof
> lower)) to get the bound. But now, I find Guido's trick more cute (and
> less error-prone). :)

Wow, that's a cool trick. The old wizard has still some astonishing
tricks in the sleeves of his robe. :]