Author stiv
Recipients stiv
Date 2010-12-14.17:41:56
SpamBayes Score 3.14445e-09
Marked as misclassified No
Message-id <1292348517.73.0.298960721159.issue10703@psf.upfronthosting.co.za>
In-reply-to
Content
The regex package doesn't seem to correctly implement the single grapheme match "\X" (\P{M}\p{M}*) for pre-Python 3. I'm using the string "íi-te" (i, U+0301, i, -, t, e -- where U+0301 is Unicode COMBINING ACUTE ACCENT), reading it in from a file to bypass Unicode c&p issues in the older IDLEs). 


stiv@x$ python3.1
Python 3.1.2 (r312:79147, May 19 2010, 11:50:28) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import regex
>>> file = open("test_data", "rt", encoding="utf-8")
>>> s = file.readline()
>>> print (s)
íi-te
>>> print (g.findall(s))
['í', 'i', '-', 't', 'e']

* Correct in 3.1 - i+U+0301 considered one grapheme.

stiv@x$ python2.7
Python 2.7 (r27:82500, Oct  4 2010, 14:49:53) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import codecs                                
>>> import regex
>>> file = codecs.open("test_data", "r", "utf-8")
>>> g = regex.compile("\X")
>>> s = file.readline()
>>> s
u'i\u0301i-te'
>>> print s.encode("utf-8")
íi-te
>>> print g.findall(s)
[u'i', u'\u0301', u'i', u'-', u't', u'e']

*Not correct -- accent is treated as a separate character.

Thanks.
History
Date User Action Args
2010-12-14 17:41:57stivsetrecipients: + stiv
2010-12-14 17:41:57stivsetmessageid: <1292348517.73.0.298960721159.issue10703@psf.upfronthosting.co.za>
2010-12-14 17:41:57stivlinkissue10703 messages
2010-12-14 17:41:56stivcreate