> Py_CHARMASK should return a non-negative integer.

And it does, also on AIX. Do we have proof to the contrary?

> tokenizer.c:tok_get around line 1368:
>
>          while (Py_ISALNUM(c) || c == '_') {
>              c = tok_nextc(tok);
>          }
>
>
> 1) tok_nextc(tok) returns EOF (correct).
>
> 2) c is an int.
>
> 3) c == -1 gets passed to Py_ISALNUM(c):
>
>     #define Py_ISALNUM(c)  (_Py_ctype_table[Py_CHARMASK(c)]&  PY_CTF_ALNUM)
>
>
>
> So either it should be enforced that only chars are passed to
> Py_CHARMASK, or a cast for the EOF case is needed (but it should
> be to (unsigned char)).

Why do you say that? If c is -1, then Py_CHARMASK(c) is 255, which is a
positive integer. Passing -1, or any other integer, to Py_CHARMASK is 
perfectly fine.

There seems to be a minor bug in the loop above, which doesn't actually 
break at EOF. Instead, it implicitly trusts that Py_ISALNUM(EOF) is 
false (because 255 is not alpha-numeric); this is a flaw - but that
shouldn't cause breakage on AIX.