classification
Title: uses of locale-dependent ctype functions
Type: behavior Stage: resolved
Components: Extension Modules, Interpreter Core Versions: Python 3.4, Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: mark.dickinson, pitrou, python-dev, rhettinger, serhiy.storchaka, skrah
Priority: normal Keywords:

Created on 2013-02-09 20:49 by pitrou, last changed 2013-02-09 22:35 by pitrou. This issue is now closed.

Messages (5)
msg181759 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-02-09 20:49
Grepping through the code reveals we are still using a number of locale-dependent C library functions:

Python/mystrtoul.c:102:    while (*str && isspace(Py_CHARMASK(*str)))
Python/mystrtoul.c:141:                while (isspace(Py_CHARMASK(*str)))
Python/mystrtoul.c:269:    while (*str && isspace(Py_CHARMASK(*str)))
Python/formatter_unicode.c:404:    while (pos<end && isdigit(PyUnicode_READ_CHAR(s, pos)))
Python/ast.c:3764:    if (isalpha(quote)) {
Python/dynload_aix.c:144:        while (isdigit(Py_CHARMASK(*message[i]))) message[i]++ ;
Objects/longobject.c:2011:    while (*str != '\0' && isspace(Py_CHARMASK(*str)))
Objects/longobject.c:2255:    while (*str && isspace(Py_CHARMASK(*str)))
Modules/getaddrinfo.c:228:        if (! isdigit(*q))
Modules/_sre.c:153:#define SRE_LOC_IS_DIGIT(ch) (!((ch) & ~255) ? isdigit((ch)) : 0)
Modules/_sre.c:154:#define SRE_LOC_IS_SPACE(ch) (!((ch) & ~255) ? isspace((ch)) : 0)
Modules/_sre.c:156:#define SRE_LOC_IS_ALNUM(ch) (!((ch) & ~255) ? isalnum((ch)) : 0)
Modules/binascii.c:1138:    if (isdigit(c))
Modules/_decimal/libmpdec/io.c:153:            if (!isdigit((uchar)*s))
Modules/_decimal/libmpdec/io.c:157:                    if (!isdigit((uchar)*(s+1)))
Modules/_decimal/libmpdec/io.c:159:                              isdigit((uchar)*(s+2))))
Modules/_decimal/libmpdec/io.c:185:    while (isdigit((uchar)*s))
Modules/_decimal/libmpdec/io.c:825:    if (isdigit((uchar)*cp)) {
Modules/_decimal/libmpdec/io.c:847:        if (!isdigit((uchar)*cp)) {
Modules/_decimal/libmpdec/io.c:1087:    assert(isdigit((uchar)*dp));
Modules/_decimal/libmpdec/io.c:1089:    while (isdigit((uchar)*dp)) {
Modules/_decimal/libmpdec/io.c:1244:    if (isupper((uchar)type)) {
Modules/_struct.c:1273:        if (isspace(Py_CHARMASK(c)))
Modules/_struct.c:1338:        if (isspace(Py_CHARMASK(c)))


Include/pyctype.h has locale-agnostic counterpart (a bit faster probably, too).
msg181760 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2013-02-09 20:57
+1 for fixing this everywhere.
msg181763 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2013-02-09 21:41
I'm not sure if I'll use pyctype.h in libmpdec: It's still going to be an
external project that should be completely identical to the version in the
Python tree.

libmpdec/io.c is specified to be ASCII only (while handling the Turkish 'I')
and is used accordingly in _decimal.c.

I think it is impossible to trigger any misbehavior just by using the
decimal module.
msg181765 - (view) Author: Roundup Robot (python-dev) Date: 2013-02-09 22:29
New changeset 38830281d43b by Antoine Pitrou in branch '3.2':
Issue #17173: Remove uses of locale-dependent C functions (isalpha() etc.) in the interpreter.
http://hg.python.org/cpython/rev/38830281d43b

New changeset c08bcf5302ec by Antoine Pitrou in branch '3.3':
Issue #17173: Remove uses of locale-dependent C functions (isalpha() etc.) in the interpreter.
http://hg.python.org/cpython/rev/c08bcf5302ec

New changeset 10e59553a8de by Antoine Pitrou in branch 'default':
Issue #17173: Remove uses of locale-dependent C functions (isalpha() etc.) in the interpreter.
http://hg.python.org/cpython/rev/10e59553a8de
msg181766 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-02-09 22:31
Done. I haven't touched _decimal, sre, getaddrinfo.c and zlib.
History
Date User Action Args
2013-02-09 22:35:29pitrousetstatus: pending -> closed
2013-02-09 22:31:21pitrousetstatus: open -> pending
resolution: fixed
messages: + msg181766

stage: resolved
2013-02-09 22:29:06python-devsetnosy: + python-dev
messages: + msg181765
2013-02-09 21:41:44skrahsetmessages: + msg181763
2013-02-09 21:39:31skrahlinkissue9067 superseder
2013-02-09 20:57:28rhettingersetnosy: + rhettinger
messages: + msg181760
2013-02-09 20:49:59pitroucreate