This patch changes the functions _PyUnicode_ToNumeric,
_PyUnicode_IsLinebreak and _PyUnicode_IsWhitespace from
having to be manually updated into being generated from
data in the unicode database.

It will allso read numeric values for characters whos
numeric type is defined in the Unihan.txt file and not
in the UnicodeData.txt file.

The patch should work for both the release25-maint
branch as well as the trunk.

The patch is so big i had to split it into two files
for sourcefore to accept it.

Logged In: YES 
user_id=38388

Instead of attaching the patch with the generated code,
could you please just attach the script that generates the
files and/or any patch needed to support the new generation
of the above three functions ?

That makes reviewing this a lot easier.

Thanks.

Logged In: YES 
user_id=621306

Here is a patch without the generated files.

I believe this one is out of date, but without a sample test to check
verifying is harder...

Adding Python 2.6 to the list of affected versions - as that is where I
found the bug reported in issue 6383 (now superseded by this one.)

Here is a refreshed version of the patch, without the generated files.
The patch combines several changes which are fairly independent from 
each other:

- Using the unicode database to generate the functions adds 143 new 
codepoints to PyUnicode_ToNumeric, and one codepoint to 
PyUnicode_IsWhitespace.

- In addition, PyUnicode_ToNumeric now contains code for all numerics; 
previously those which are also digits fell in the 'default:' case and 
were converted with PyUnicode_ToDigit(). This adds 468 new codepoints, 
but removes the need to call PyUnicode_ToDigit()

- The Unihan.txt files (two files to download, 25Mb each) are now 
parsed, and this adds 73 more codepoints to PyUnicode_ToNumeric. (There 
are now 1009 entries in this function.)
The 3.2.0 version of this file contains two huge numbers: 1e16 and 1e20, 
I had to widen the type of 'change_record.numeric_changed' from 'int' to 
'double'.  It is possible that these were removed from the Unicode 
database between versions 4.1 and 5.1.

- the database has a new flag, NUMERIC_MASK, used by 
PyUnicode_IsNumeric.  This adds ~350 lines in the arrays of numbers in 
unicodetype_db.h

If this patch is accepted, the md5 checksum in test_unicodedata.py will 
need to change.

Marc-Andre, could you comment on this patch?
The comments above were made by inspecting the generated code, comparing
with the previous version.
IMO the only drawback is the increased memory usage.

Amaury Forgeot d'Arc wrote:
> 
> Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:
> 
> Marc-Andre, could you comment on this patch?
> The comments above were made by inspecting the generated code, comparing
> with the previous version.
> IMO the only drawback is the increased memory usage.

I haven't tried applying the patch, but from reading it, it looks
good.

Patch applied with r75272.
Merged to py3k, adapted and regenerated files with r75274.