The thing is that the parser itself does not get a stream of tokens as input. It only accepts either a file or a string and it lazily converts its input to tokens.

As for the PR attached to this patch, I'm -1 on that. I don't think the usecase is common enough for us to have another public function, that we need to maintain and keep backwards-compatible. I concur with Batuhan that if people need this, they can use ast.parse with tokenize.untokenize.