Currently, strings.Template uses re.IGNORECASE without re.ASCII:

    idpattern = r'[_a-z][_a-z0-9]*'
    flags = _re.IGNORECASE

[a-z] matches against 'ı' (0x131, DOTLESS I) and 'ſ' (0x17f, LONG S).
It is not intentional, and it makes re.compile slower.

Technically it *is* an API change since `flags` is a part of the public API.  The documentation says:

    $identifier names a substitution placeholder matching a mapping key of 
    "identifier". By default, "identifier" is restricted to any case-
    insensitive ASCII alphanumeric string (including underscores) that starts 
    with an underscore or ASCII letter. The first non-identifier character 
    after the $ character terminates this placeholder specification.

This means if someone does subclass string.Template and changes the pattern to accept Unicode identifiers, then with this change they will also have to modify flags, whereas before they didn't.

It really wasn't ever the intention to allow non-ASCII identifiers, so this is probably safe in practice.  OTOH, making the change for performance reasons might be questionable, given that the regular expressions are compiled by the Template's metaclass, so unlikely to contribute significantly to overall performance wins.

> This means if someone does subclass string.Template and changes the pattern to accept Unicode identifiers, then with this change they will also have to modify flags, whereas before they didn't.

Thank you for pointing it out.
I removed re.A flag after compilation.

> OTOH, making the change for performance reasons might be questionable, given that the regular expressions are compiled by the Template's metaclass, so unlikely to contribute significantly to overall performance wins.

original: import time:      2310 |       9589 | string
patched:  import time:      1420 |       8702 | string

We can save about 900 us.

When optimizing, please don't make API changes.

Yet one way -- make re.ASCII a local flag. Than we could just change the idpattern attribute to r'(?a:[_a-z][_a-z0-9]*)', without touching the flags attribute.

> When optimizing, please don't make API changes.

This is not only optimization, but bugfix.

Document of string.Template says:

> By default, "identifier" is restricted to any case-insensitive ASCII  alphanumeric string (including underscores) that starts with an underscore or ASCII letter.

So, missing re.ASCII flag is bug because non-ASCII alphabet can be matched.

> Yet one way -- make re.ASCII a local flag. Than we could just change the idpattern attribute to r'(?a:[_a-z][_a-z0-9]*)', without touching the flags attribute.

https://docs.python.org/3.6/library/re.html#regular-expression-syntax says
`(?imsx-imsx:...)`

Anyway, Template.idpattern is documented public API too.

Current pull request override `Template.flags = re.I` after class creation for backward compatibility, without any API change.

But I'm not sure it's right approach.
How many people who subclass string.Template expect non-ASCII match?
If this change is bugfix for 99% of subclasses, why should we keep pitfall?

On Oct 4, 2017, at 02:29, INADA Naoki <report@bugs.python.org> wrote:
> 
> INADA Naoki <songofacandy@gmail.com> added the comment:
> 
>> Yet one way -- make re.ASCII a local flag. Than we could just change the idpattern attribute to r'(?a:[_a-z][_a-z0-9]*)', without touching the flags attribute.
> 
> https://docs.python.org/3.6/library/re.html#regular-expression-syntax says
> `(?imsx-imsx:...)`
> 
> Anyway, Template.idpattern is documented public API too.

Too bad, because I like that approach.  How hard would it be to add support for ‘a’ as a local flag?  (I’m kind of surprised that it isn’t already supported - it seems like it would be useful.)

I would like this better than hacking Template.flags after the fact.  It seems like the right way to align the code with the documentation (i.e. fix a bug).

See issue31690. But this solution can be used only in 3.7.

On Oct 4, 2017, at 10:05, Serhiy Storchaka <report@bugs.python.org> wrote:
> 
> See issue31690. But this solution can be used only in 3.7.

That’s fine.  I don’t think this is important enough to backport.

Another solution (works in 3.6 too): set idpattern to r'(?-i:[_a-zA-Z][_a-zA-Z0-9]*)'.

This looks pretty weird, setting the re.IGNORECASE flag and then unsetting it. But it works, and don't break the user code that changes idpattern without changing flags.

On Oct 6, 2017, at 15:04, Serhiy Storchaka <report@bugs.python.org> wrote:
> Serhiy Storchaka <storchaka+cpython@gmail.com> added the comment:
> 
> Another solution (works in 3.6 too): set idpattern to r'(?-i:[_a-zA-Z][_a-zA-Z0-9]*)'.
> 
> This looks pretty weird, setting the re.IGNORECASE flag and then unsetting it. But it works, and don't break the user code that changes idpattern without changing flags.

Oh, I think I like that :)

With some comments to clarify of course.

New changeset b22273ec5d1992b0cbe078b887427ae9977dfb78 by INADA Naoki in branch 'master':
bpo-31672: Fix string.Template accidentally matched non-ASCII identifiers (GH-3872)
https://github.com/python/cpython/commit/b22273ec5d1992b0cbe078b887427ae9977dfb78

New changeset 7060380d577690a40ebc201c0725076349e977cd by INADA Naoki in branch '3.6':
bpo-31672: Fix string.Template accidentally matched non-ASCII identifiers (GH-3872)
https://github.com/python/cpython/commit/7060380d577690a40ebc201c0725076349e977cd

New changeset 073150db39408c1800e4b9e895ad0b0e195f1056 by INADA Naoki in branch 'master':
bpo-31672: doc: Remove one sentence from library/string.rst (GH-3990)
https://github.com/python/cpython/commit/073150db39408c1800e4b9e895ad0b0e195f1056

Is something left to do here?

In 3.7 r'(?-i:[_a-zA-Z][_a-zA-Z0-9]*)' can be replaced with r'(?a:[_a-z][_a-z0-9]*)' if it will add clarity.

@inada.naoki - I think changing this to ?a will make things more clear.  I submitted a PR for that, and then once it lands, we can close this issue.

New changeset e256b408889eba867e1d90e5e1a0904843256255 by Barry Warsaw in branch 'master':
bpo-31672 - Add one last minor clarification for idpattern (#4483)
https://github.com/python/cpython/commit/e256b408889eba867e1d90e5e1a0904843256255

PR 5099 reverts in 3.7 a subtle behavior change made by the fix for this issue. Before merging the fix setting flags to 0 in a Template subclass made the default pattern matching only lower case letters. Currently setting flags to 0 doesn't have any effect.

New changeset 87be28f4a1c5b76926c71a3d9f92503f9eb82d51 by Serhiy Storchaka in branch 'master':
bpo-31672: Restore the former behavior when override flags in Template. (#5099)
https://github.com/python/cpython/commit/87be28f4a1c5b76926c71a3d9f92503f9eb82d51

Results.

There are 4 cases:

1) default idpattern and flags
2) overridden only idpattern
3) overridden only flags
4) overridden both idpattern and flags

The case 1 was the one that was broken when this issue was opened. The initial Inada's version fixed the case 1, but broke the case 2. His final version (applied also to 3.6) fixed the case 1, but broke the case 3. This is a win, because cases 1 and 2 look much more common than the case 3. And finally PR 5099 has fixed also the case 3. The case 4 obviously is not affected by any changes of default values.

Now all four cases are correct in 3.7 and the only broken case in 3.6 is the uncommon case 3.