The expectation would be that the re.A (or re.ASCII) flag should not impact the matching behavior of a regular expression on strings consisting only of ASCII characters.  However, for the characters 0x1c till 0x1f, the classes \s and \S differ. For ASCII theses characters are not considered space characters while for unicode they are. 

Note that python strings do consider these characters spaces as '\xc1'.isspace() gives True. 

All other classes and characters stay the same for unicode and ASCII matching.

Small addition, the sre categories CATEGORY_LINEBREAK and CATEGORY_UNI_LINEBREAK also do not agree on ASCII characters.
The first is only '\n' while the second also includes for example '\r' and some others. These do not seem to correspond to anything however and are never used in sre_parse.py or sre_compile.py.

For comparison, the regex module says that 0x1C..0x1F aren't whitespace, and the Unicode property White_Space ("\p{White_Space}" in a pattern, where supported) also says that they aren't whitespace.

Hi Joran,

I'm not sure why you think that /s should agree between ASCII and Unicode. That seems like an unjustified assumption to me.

You say: "The expectation would be that the re.A (or re.ASCII) flag should not impact the matching behavior of a regular expression on strings consisting only of ASCII characters."

But I'm not sure why you have that expectation. Is it documented somewhere? The docs clearly say that for character classes, "the characters they match depends on whether ASCII or LOCALE mode is in force." I am unable to find anything that says that the differences are limited only to non-ASCII code points.

I don't think there is any standard definition of "whitespace" in either the ASCII standard, or the very many different regex engines (Perl, dot-Net, Java, ECMA, etc).

Unicode does have an official whitespace character property, and as far as I can see '\x1c' through '\x1f' (File Separator, Group Separator, Record Separator and Unit Separator) are not considered whitespace:

https://en.wikipedia.org/wiki/Unicode_character_property#Whitespace

But the str.isspace() method does consider them as whitespace, while bytes.isspace() does not.


>>> '\x1c'.isspace()
True
>>> b'\x1c'.isspace()
False

In any case, any change to this would have to be limited to Python 3.11. It is not clearly a bug, so this would be an enhancement.

Hi,

I was not suggesting that the documentation literally says they should be the same but it might be unexpected for users if ASCCI characters change properties depending on whether they are considered in a unicode or pure ASCII setting. 

The documentation says about re.A: "Make \w, \W, \b, \B, \d, \D, \s and \S perform ASCII-only matching instead of full Unicode matching. ". The problem might be that there is no clear notion of "ASCII-only matching". I assumed this mean matching ASCII characters only, i.e., the character classes are simply limited to codes below 128. 

About \s the documentation says:
"Matches Unicode whitespace characters (which includes [ \t\n\r\f\v], and also many other characters, for example the non-breaking spaces mandated by typography rules in many languages). If the ASCII flag is used, only [ \t\n\r\f\v] is matched.". This heavily implies that there are non-ASCII characters in Unicode that might be considered spaces, but that the ASCII characters are [ \t\n\r\f\v], although again, not stated literally. 

There might be valid reasons to change the definition (even for ASCII characters) depending on re.A, but should it then not follow the unicode standard for white space in the unicode case? (which would coincide with the current ASCII case). There seem to be many different places where python is opinionated about what a space is, but not much consistency behind it.

I am a bit worried about the undocumented nature of the precise definitions of the regex classes in general. How is a user supposed to know that the default behavior of \s, when no flag is passed, is to also match other ASCII characters then those mentioned for the ASCII case? In contrast to this, the \d class is directly defined as the unicode category [Nd]. 

It is likely to hard to change and to many things depend on it but the following definitions would make more sense to me, and hopefully others:
- Character classes are defined as a set of unicode properties/categories, following the same definitions as elsewhere in python.
- If re.A is passed, they are this same set but limited to codes below 128. 

After some digging in the code I traced the current definitions as follows:
 - For unicode Py_UNICODE_ISSPACE is called, which either does a lookup in the constant table _Py_ascii_whitespace or calls _PyUnicode_IsWhitespace for non ASCII characters. Both of these define a space as "Unicode characters having the bidirectional type 'WS', 'B' or 'S' or the category 'Zs'", i.e., this is simply the unicode string isspace() definition. 
 - For ASCII Py_ISSPACE is called which does a lookup in _Py_ctype_table. It is unclear to me how this table was made.

So sre just follows the other python definitions.
In searching around i found issue  #18236 , which also considers how the python definition differs from the unicode one.