> but the thing I don't quite get is the use case.

And if that went unanswered: the usecase, how I'd formulate it, is to not expose CPython historical implementation detail of "tokenize" being disconnected from the rest of the language processing infrastructure, and make them learn tricks like needing to go back to character program form if they ever start to use "tokenize", but let it all integrate well into single processing pipeline.