Alexander Belopolsky wrote:
> 
> Alexander Belopolsky <belopolsky@users.sourceforge.net> added the comment:
> 
> I am submitting a patch (issue10557b.diff) for commit review.  As Marc suggested, decimal conversion is now performed on Py_UNICODE characters. For this purpose, I introduced _PyUnicode_NormalizeDecimal() function that takes Py_UNICODE and returns a PyUnicode object with whitespace stripped and non-ASCII digits converted to ASCII equivalents.  The PyUnicode_EncodeDecimal() function is no longer used and I added a comment recommending that _PyUnicode_NormalizeDecimal() be used instead. I would like to eventually remove PyUnicode_EncodeDecimal(), but I amd not sure about the proper deprecation procedures for undocumented C APIs.
> 
> As a result, int(), float(), etc will no longer raise UnicodeDecodeError unless given a string with lone surrogates.  (This error comes from UTF-8 codec that is applied after digit normalization.)
> 
> A few error cases such as embedded '\0' and non-digit characters with ord(c) > 255 will now raise ValueError instead of UnicodeDecodeError.  Since UnicodeDecodeError is a subclass of ValueError, it is unlikely that existing code would attempt to differentiate between the two.  It is possible to achieve complete compatibility, but it is hard to justify reporting different error types on non-digit characters below and above code point 255.
> 
> The patch contains tests for error messages that I tried to make robust by only requiring that s.strip() be found somewhere in the error message from int(s).  Note that since in this patch whitespace is stripped before the string is passed to the parser, the parser errors do not contain the whitespace.  This may actually be desirable because it helps the user to see the source of the error without being distracted by irrelevant white space.

Thanks for the patch. I've had a quick look...

Some comments:

 * Please change the API _PyUnicode_NormalizeDecimal() to
   PyUnicode_ConvertToASCIIDecimal() - that's closer to what
   it does.

 * Don't have the API remove any whitespace. It should just
   work on decimal digit code points (chainging the length
   of the Unicode string is a bad idea).

 * Please remove the note "This function is no longer used.
   Use _PyUnicode_NormalizeDecimal instead." from the
   PyUnicode_EncodeDecimal() API description in the
   header file. The API won't go away (it does have its
   use and is being used in 3rd party extensions) and
   you cannot guide people to use a Python private API.

 * Please double check the ref counts. I think you have a leak
   in PyLong_FromUnicode() (for norm) and possible in other
   functions as well.