Message270946
I still think the easiest thing to do would be to make all non-ASCII characters instances of "invalid_character_token", self-delimiting in the same way that operators are. That would automatically point to exactly the right place in the token stream, and requires zero changes to the error handling code.
I don't have time to look at the code, but I suspect that you could handle this exactly the same way that ? and $ are handled, and maybe even use the same token type. |
|
Date |
User |
Action |
Args |
2016-07-21 17:15:04 | sjt | set | recipients:
+ sjt, ncoghlan, Rosuav, berker.peksag |
2016-07-21 17:15:04 | sjt | set | messageid: <1469121304.17.0.528382017648.issue27582@psf.upfronthosting.co.za> |
2016-07-21 17:15:04 | sjt | link | issue27582 messages |
2016-07-21 17:15:04 | sjt | create | |
|