classification
Title: Documentation v/s behaviour mismatch wrt integer literals containing non-ASCII characters
Type: behavior Stage: needs patch
Components: Documentation, Interpreter Core, Unicode Versions: Python 3.6, Python 3.4, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, ezio.melotti, martin.panter, r.david.murray, shreevatsa, vstinner
Priority: normal Keywords:

Created on 2015-09-30 05:19 by shreevatsa, last changed 2015-10-01 04:07 by martin.panter.

Messages (8)
msg251915 - (view) Author: Shreevatsa R (shreevatsa) Date: 2015-09-30 05:19
Summary: This is about int(u'१२३४') == 1234.

At https://docs.python.org/2/library/functions.html and also https://docs.python.org/3/library/functions.html the documentation for 

     class int(x=0)
     class int(x, base=10)

says (respectively):

> If x is not a number or if base is given, then x must be a string or Unicode object representing an integer literal in radix base.

> If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in radix base.

If you follow the definition of "integer literal" into the reference (https://docs.python.org/2/reference/lexical_analysis.html#integers and https://docs.python.org/3/reference/lexical_analysis.html#integers respectively), the definitions ultimately involve

     nonzerodigit   ::=  "1"..."9"
     octdigit       ::=  "0"..."7"
     bindigit       ::=  "0" | "1"
     digit          ::=  "0"..."9"

So it looks like whether the behaviour of int() conforms to its documentation hinges on what "representing" means. Apparently it is some definition under which u'१२३४' represents the integer literal 1234, but it would be great to either clarify the documentation of int() or change its behaviour.
msg251930 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-09-30 13:16
Apparently that documentation is simply wrong.  The actual definition of what 'int' handles is *different* from what the parser handles.  I think that difference must constitute a bug (not just a doc bug), but I'm not sure if it is something that we want to fix (changing the parser).

I think the *operational* definition of int conversion for both is the same as for isdigit in python3 (https://docs.python.org/3/library/stdtypes.html#str.isdigit).  (The python2 docs just say '8 bit strings may be locale dependent', which means the same thing but is less precise).

>>> १२३४
  File "<stdin>", line 1
    १२३४
       ^
SyntaxError: invalid character in identifier
>>> int('१२३४')
1234
>>> '१२३४'.isdigit()
True

The above behavior discrepancy doesn't apply to python2, since in python2 you can't use unicode in integer literals.

So, this is a bit of a mess :(.

The doc fix is simple: just replace the mention of integer literal with a link to isdigit, and fix the python2 isdigit docs to match python3's.
msg251932 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-09-30 13:18
I mean, in python2 you can't use unicode in python code, only in strings, as opposed to python3 where unicode is valid in identifiers (but not integer literals, obviously).
msg251965 - (view) Author: Shreevatsa R (shreevatsa) Date: 2015-09-30 20:45
Minor difference, but the relevant function for int() is not quite isdigit(), e.g.:

    >>> import unicodedata
    >>> s = u'\u2460'
    >>> unicodedata.name(s)
    'CIRCLED DIGIT ONE'
    >>> print s
    ①
    >>> s.isdigit()
    True
    >>> s.isdecimal()
    False
    >>> int(s)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    UnicodeEncodeError: 'decimal' codec can't encode character u'\u2460' in position 0: invalid decimal Unicode string

It seems to be isdecimal(), plus if there are other digits in the string then many leading and trailing space-like characters are also allowed (e.g. 5760 OGHAM SPACE MARK or 8195 EM SPACE or 12288 IDEOGRAPHIC SPACE:

    >>> 987 == int(u'\u3000\n 987\u1680\t')
    True
msg251966 - (view) Author: Shreevatsa R (shreevatsa) Date: 2015-09-30 20:48
About the mismatch: of course it's probably not a good idea to change the parser (so that simply typing १२३४ in Python 3 code is like typing 1234), but how about changing the behaviour of int()? Not sure whether anyone should be relying on int(u'१२३४') being 1234, given that it is not documented as such.
msg251967 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-09-30 20:54
Good catch.  Yes, it is already documented that Int ignores leading and trailing whitespace.

But, even that isn't quite correct:

>>> 'A'.isdecimal()
False
>>> int('A', 16)
10

I seem to vaguely recall a discussion somewhere in this tracker about what "should" count as digits for larger-than-decimal radii, but I don't remember the outcome.
msg251968 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-09-30 20:56
No, we can't make it stop working for int, that would be a backward compatibility break.  Doing so was discussed at one point and rejected (another issue somewhere in this tracker :)
msg251991 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-10-01 04:07
Related discussion and background in Issue 10581, although that report seems to be geared at extending the Unicode support even further (disallowing mixed scripts, allowing proper minus signs, full-width characters, Roman numerals, etc).

The existing support is actually documented if you know where to look; see the sixth note under the table at <https://docs.python.org/dev/library/stdtypes.html#typesnumeric>. I agree that the references for each constructor should also document this as well.
History
Date User Action Args
2015-10-01 04:07:07martin.pantersetnosy: + martin.panter
messages: + msg251991
2015-09-30 20:56:09r.david.murraysetmessages: + msg251968
2015-09-30 20:54:42r.david.murraysetmessages: + msg251967
2015-09-30 20:48:47shreevatsasetmessages: + msg251966
2015-09-30 20:45:10shreevatsasetmessages: + msg251965
2015-09-30 13:18:58r.david.murraysetmessages: - msg251931
2015-09-30 13:18:48r.david.murraysetmessages: + msg251932
2015-09-30 13:18:15r.david.murraysetmessages: + msg251931
2015-09-30 13:16:34r.david.murraysetversions: + Python 2.7, Python 3.4, Python 3.5, Python 3.6
nosy: + r.david.murray

messages: + msg251930

type: behavior
stage: needs patch
2015-09-30 05:19:45shreevatsacreate