classification
Title: Incorrect charset range handling with ignore case flag?
Type: behavior Stage:
Components: Regular Expressions Versions: Python 3.0, Python 2.6, Python 2.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder: IGNORECASE breaks unicode literal range matching
View: 17381
Assigned To: Nosy List: ezio.melotti, mrabarnett, pitrou, serhiy.storchaka, terry.reedy, timehorse
Priority: normal Keywords:

Created on 2008-08-06 19:41 by mrabarnett, last changed 2014-11-08 12:14 by serhiy.storchaka. This issue is now closed.

Messages (7)
msg70799 - (view) Author: Matthew Barnett (mrabarnett) * Date: 2008-08-06 19:41
While working on the regex code in sre_compile.py I came across the
following code in the handling of charset ranges in _optimize_charset:

    for i in range(fixup(av[0]), fixup(av[1])+1):
        charmap[i] = 1

The function fixup converts the ends of the range to lower case if the
ignore-case flag is present. The problem with this approach is
illustrated below:

>>> import re
>>> print re.match(r'[9-A]', 'A')
<_sre.SRE_Match object at 0x00A78058>
>>> print re.match(r'[9-A]', 'a')
None
>>> print re.match(r'[9-A]', '_')
None
>>> print re.match(r'[9-A]', 'A', re.IGNORECASE)
<_sre.SRE_Match object at 0x00D0BFA8>
>>> print re.match(r'[9-A]', 'a', re.IGNORECASE)
<_sre.SRE_Match object at 0x00A78058>
>>> print re.match(r'[9-A]', '_', re.IGNORECASE)
<_sre.SRE_Match object at 0x00D0BFA8>
>>> 

'_' doesn't lie between '9' and 'A', but it does lie between '9' and 'a'.

Surely the ignore-case flag should not affect whether non-letters are
matched or not?
msg73711 - (view) Author: Jeffrey C. Jacobs (timehorse) Date: 2008-09-24 13:15
I think this is even more complicated when you consider that
localization my be an issue.  Consider "Á": is this grammatically before
 "A" or after "a"?  From a character set point of view, it is typically
after "a" but when Locale is taken into account, all that is done is
there is a change to relative ordering, so Á appears somewhere before A
and B.  But when this is done, does that mean that [9-Á] is going to
cover ALL uppercase and ALL lowercase and ALL characters with ord from
91 to 96 and 123 to 127 and all kinds of other UNICODE symbols?  And how
will this effect case-insensitivity.

In a sense, I think it may only be safe to say that character class
ranges are ONLY appropriate over Alphabetic character ranges or numeric
character ranges, since the order of the ASCII symbols between 0 and 47,
56 and 64, 91 adn 96 and 123 and 127, though well-defined, are none the
less implementation dependent.  When we bring UNICODE into this, things
get even more befuddled with some Latin characters in Latin-1, some in
Latin-2, Cyrillic, Hebrew, Arabic, Chinese, Japanese and Korean
character sets just to name a few of the most common!  And how does a
total ordering of characters apply to them?

In the end, I think it's just dangerous to define character group ranges
that span the gap BETWEEN numbers and alphabetics.  Instead, I think a
better solution is simply to implement Emacs / Perl style named
character classes as in issue 2636 sub-item 8.

I do agree this is a problem, but as I see it, the solution may not be
that simple, especially in a UNICODE world.
msg82515 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2009-02-20 06:46
I'd close this as "won't fix", because (IMHO) ranges like [9-A]
shouldn't be used at all, so I won't expect it to work properly.

FWIW Perl doesn't seem to match the '_', even with the 'i' flag. Tested
with: perl -e '$s = ("_" =~ /[9-A]/); print $s' and perl -e '$s = ("_"
=~ /[9-A]/i); print $s'. It matches ":" with [9-A] and "_" with [9-a]
though (both without the 'i' flag).
msg82537 - (view) Author: Matthew Barnett (mrabarnett) * Date: 2009-02-20 16:31
"[9-A]" is equivalent to "[9:;<=>?@A]", or should be.

It'll be fixed in issue #2636.
msg82541 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2009-02-20 19:05
If there's already a patch, then it's fine (and useful for ranges of
Unicode chars), but I wouldn't like to find a range like [9-A] around ;)
msg112652 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-08-03 18:56
EM and MB seemed to agree on closing this.
msg230848 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-11-08 12:14
Fixed in issue17381 (which has more realistic example than [9-A]).
History
Date User Action Args
2014-11-08 12:14:26serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg230848
resolution: wont fix -> duplicate

superseder: IGNORECASE breaks unicode literal range matching
2010-08-03 18:56:03terry.reedysetstatus: open -> closed

nosy: + terry.reedy
messages: + msg112652

resolution: wont fix
2009-02-20 19:05:29ezio.melottisetmessages: + msg82541
2009-02-20 16:31:11mrabarnettsetmessages: + msg82537
2009-02-20 06:46:22ezio.melottisetnosy: + ezio.melotti
messages: + msg82515
2008-09-24 13:15:03timehorsesetmessages: + msg73711
2008-09-24 12:58:36timehorsesetnosy: + timehorse
2008-08-06 19:48:38pitrousetpriority: normal
nosy: + pitrou
versions: + Python 2.6, Python 3.0
2008-08-06 19:41:11mrabarnettcreate