On Py3 strptime("２００９", "%Y") fails:
>>> strptime("２００９", "%Y")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.0/_strptime.py", line 454, in _strptime_time
    return _strptime(data_string, format)[0]
  File "/usr/local/lib/python3.0/_strptime.py", line 325, in _strptime
    (data_string, format))
ValueError: time data '２００９' does not match format '%Y'

but non-ascii numbers are supported elsewhere:
>>> int("２００９")
2009
>>> re.match("^\d{4}$", "２００９").group()
'２００９'

The problem seems to be at the line 265 of _strptime.py:
        return re_compile(self.pattern(format), IGNORECASE | ASCII)
The ASCII flag prevent the regex to work properly with '２００９':
>>> re.match("^\d{4}$", "２００９", re.ASCII)
>>>

I tried to remove the ASCII flag and it worked fine.

On Py2.x the problem is the same:
>>> strptime(u"２００９", "%Y")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/_strptime.py", line 330, in strptime
    (data_string, format))
ValueError>>>
>>> int(u"２００９")
2009
>>> re.match("^\d{4}$", u"２００９")

Here there's probably to add the re.UNICODE flag at the line 265 (untested):
        return re_compile(self.pattern(format), IGNORECASE | UNICODE)
in order to make it work:
>>> re.match("^\d{4}$", u"２００９", re.U).group()
u'\uff12\uff10\uff10\uff19'

This patch comes from bpo-5240. I think testcase is needed. I'll try if
I can.

Hmm, this fails on python2 too. Maybe re.ASCII is added for backward
compatibility? Again, I'm not familiar with unicode, so I won't call
remove_ascii_flag.patch as *fix*.

Hmm, this fails on python2 too. Maybe re.ASCII is added for backward
compatibility? Again, I'm not familiar with unicode, so I won't call
remove_ascii_flag.patch as *fix*.

re.ASCII was added to many stdlib modules because I wanted to minimize
the potential for breakage when I converted the re library to use
unicode matching by default.

If it is desireable for strptime() and friends to match unicode digits
as well as pure-ASCII digits (which sounds like a reasonable request to
me), then re.ASCII can probably be dropped without any regret.

(py3k doesn't have to be 100% compatible with python2 :-))

I think Py3 with re.ASCII is the same as Py2 without re.UNICODE (and Py3
without re.ASCII is the same as Py2 with re.UNICODE).

It's probably a good idea to have a coherent behavior between Py2 and
Py3, so if we remove re.ASCII from Py3 we should add re.UNICODE to Py2.

Le vendredi 13 février 2009 à 14:44 +0000, Ezio Melotti a écrit :

It's probably a good idea to have a coherent behavior between Py2 and
Py3, so if we remove re.ASCII from Py3 we should add re.UNICODE to Py2.

Removing re.ASCII in py3k is a no-brainer, because unicode is how
strings work by default.
On the other hand, strings in 2.x are 8-bit, so it would probably be
better to keep strptime as is.
As I said, py3k doesn't have to be compatible with 2.x, that's even the
whole point of it.

Removing re.ASCII in py3k is a no-brainer, because unicode is how
strings work by default.

I meant from the line 265 of _strptime.py, not from Python :P

> Removing re.ASCII in py3k is a no-brainer, because unicode is how
> strings work by default.

I meant from the line 265 of _strptime.py, not from Python :P

That's what I understood.

Sorry, I misunderstood the meaning of "no-brainer".

If we add re.UNICODE on Py2, strptime should work fine with unicode
strings, but it could fail somehow with normal strings. Is it more
important to provide a way to use Unicode chars that works only with
unicode strings or to have a coherent behavior between str and unicode?

I don't think that adding re.UNICODE will break any existing code, but
it may cause problems if someone tries to use encoded str instead of
unicode (but shouldn't work already).

Also note that encoded strings should be a problem only if they have to
match a strptime directive (e.g. %Y), the other chars should be compared
as they are, so it should work with str and unicode as long as they are
not mixed (I think that whitespaces are treated differently though).

I'll try to add re.UNICODE and see what happens.

I added test. But this requires bpo-5249 fix to be passed on windows.

(I used "\u3000" instead of "\xa0" because "\xa0" cannot be decoded on
windows mbcs)

If we add re.UNICODE on Py2, strptime should work fine with unicode
strings, but it could fail somehow with normal strings. Is it more
important to provide a way to use Unicode chars that works only with
unicode strings or to have a coherent behavior between str and unicode?

I'd say the latter, since str and unicode are often interchangeable in
2.x.

This issue seems to be fixed on py3k by r70755. (bpo-5236)

As Hirokazu pointed out, this was fixed.