Title: re.sub, re.Match.expand, etc doesn't allow x, u, U, or N escapes in the template
Type: behavior Stage:
Components: Versions:
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: bup, steven.daprano
Priority: normal Keywords:

Created on 2022-01-11 22:10 by bup, last changed 2022-01-11 22:21 by steven.daprano.

Messages (2)
msg410337 - (view) Author: Dan Snider (bup) * Date: 2022-01-11 22:10
The docs use the phrase "unknown escapes of ASCII letters are reserved for future use and treated as errors". That seems ambiguous enough to question why "\x", "\u", "\U", and "\N{}" escapes aren't expanded in the template parameter like they are in patterns. 

Since I didn't get a response to the security report I submitted a few weeks ago about \N{} escapes, I'm cautiously assuming it's safe to bring it up here that the "unicode-escape" encoding and re and probably everything else that uses it ignores two obvious clues that a name lookup will fail: length and the presence of invalid characters. I didn't look very hard for a  definite length cap in the spec, but 255 seems more than sufficient, based on longest name at present with its 82 characters. Even something as absurd as 65535 would be preferable to the current implementations, which will keep going to the end as in:

    >>> r"\N{%s}" % ("\ufb03"*2**30)

searching or a terminating "}" and still perform a lookup of the 2**30 character name.

Another tangentially related "bug" (which probably deserves its own issue) is the inconsistency between group names and standard Python identifiers. The following example shows how the python compiler decomposes a ligature 'ffi' in source code to the ASCII string "ffi", while re merely checks if it could be converted to an identifier:

    >>> ffi ="(?P<ffi>.)", "xxx")
    >>> ffi.groupdict()
    {'ffi': 'x'}
    >>> "\ufb03" in vars(), "\ufb03" in _
    (False, True)
msg410340 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2022-01-11 22:21
It is better to raise each issue in its own ticket.

You seem to have three distinct issues here:

- The issue listed in the title, which I don't understand. A demonstration of the issue would be helpful.

- The unrelated(?) issue of bad \N{} escapes, which appears to have nothing to do with regexes, but maybe I'm wrong? In any case, whether this is regular expressions or plain old strings, it seems reasonable to me to limit \N{} to ASCII-only (Unicode guarantees that code point names are ASCII) and some reasonable length. I can't imagine the Unicode consortium ever deciding on a name > 255 characters, it wouldn't be practical.

- The difference in normalisation between group names and identifiers.
Date User Action Args
2022-01-11 22:21:30steven.dapranosetnosy: + steven.daprano
messages: + msg410340
2022-01-11 22:10:12bupcreate