After thinking about it more, I guess I misunderstood what \w was doing compared to isidentifier. Since Python just relies on the Unicode database, there's not much to be done anyway. Closing this.

For anyone interested, we ended up with a hybrid approach for lexing identifiers: build a regex group that includes all valid ranges not matched by \w, then validate with isidentifier later. https://github.com/pallets/jinja/pull/731/files