the Tokenize module has the following code:

# An undocumented, backwards compatible, API for all the places in the standard
# library that expect to be able to use tokenize with strings
def generate_tokens(readline):
    return _tokenize(readline, None)

So I'm going to assume it is Deprecated... (since 3.0 AFAICT). 

If it is expected from Python developers to not use it, may I suggest:

1) Marking it as deprecated in the docstring/
2) Raise a deprecation warning. 

Indeed not everyone code by looking at the documentation and it is relatively easy to have your IDE/editor/REPL to complete it. Even tools that grab the source (IPython double question mark for example) will not show the comment above which make it kinda pointless.