It seems that Tools/unicode/mkstringprep.py has not been used for many years. Now it is not valid Python3 code nor Python2 code. The proposed patch fixes all porting errors.

Apparently, Lib/stringprep.py would have regenerated. Tools/unicode/mkstringprep.py output is a bit different from Lib/stringprep.py (in b3_exceptions).

Indeed, the code hasn't been run, and really shouldn't have to. If it produces different output today, we should investigate why.

>  If it produces different output today, we should investigate why.

Lib/stringprep.py differs from updated Tools/unicode/mkstringprep.py output only by additional entity 0x130:'i\u0307' in b3_exceptions.

In 3.2 and lower '\u0130'.lower() == '\u0069', in 3.3 '\u0130'.lower() == '\u0069\u0307'.

The difference is in the reverse direction, right? I.e. the special case for U+0130 is removed.

This is harmless: it just means that we don't need to special-case that anymore.

I'm puzzled though that b3_exceptions doesn't become empty in 3.3. Supposedly, B.3 is CaseFolding.txt, and supposedly, Python 3.3 supports CaseFolding.txt. For example, CaseFolding maps U+00B5 to U+03BC in lower case, yet Python 3.3 maps it to U+00B5.

To protect themselves from the surprises we need working mkstringprep.py. Martin, what do you say about a patch?

I don't consider it relevant for 3.3; I may have time to review only post-3.3.

Martin, are you have time to review now?

Georg, now Tools/unicode/mkstringprep.py is broken in all Python 3 branches (it not usable at all, under 3.2 it even raises SyntaxError). Do you object against fixing it in 3.2 and 3.3?

No need to hurry; no one apparently needs it anyway, and if there is a change let Martin review it before commit.

Is there a chance, Martin?

I find the patch too large to review, and it appears to contain unrelated changes. Can you kindly split it up into two patches, namely

A) changes that are really absolutely necessary to make it work again
B) patches that are purely cosmetic, and do not affect the behavior at all (such as rewriting .startswith, or replacing %s with %r)

Possibly, there is also

C) changes that do change the behavior, but in a way unrelated to the issue at hand; such changes are best left out.

Sorry it took so long to react.

Here is a minimal patch.

Here is a pure cosmetic patch.

And here is a patch which changes behavior. It adds check for validating end of table detection.

Ok, these patches all look fine. Thanks for your effort.

Thank you for review. Should we regenerate Lib/stringprep.py now?

New changeset 8f95d77443da by Serhiy Storchaka in branch '3.3':
Issue #15239: Make mkstringprep.py work again on Python 3.
http://hg.python.org/cpython/rev/8f95d77443da

New changeset 4abe61a412be by Serhiy Storchaka in branch 'default':
Issue #15239: Make mkstringprep.py work again on Python 3.
http://hg.python.org/cpython/rev/4abe61a412be