the Tokenize module has the following code:

# An undocumented, backwards compatible, API for all the places in the standard
# library that expect to be able to use tokenize with strings
def generate_tokens(readline):
    return _tokenize(readline, None)

So I'm going to assume it is Deprecated... (since 3.0 AFAICT). 

If it is expected from Python developers to not use it, may I suggest:

1) Marking it as deprecated in the docstring/
2) Raise a deprecation warning. 

Indeed not everyone code by looking at the documentation and it is relatively easy to have your IDE/editor/REPL to complete it. Even tools that grab the source (IPython double question mark for example) will not show the comment above which make it kinda pointless.

There is related discussion in Issue 12486 about supporting unencoded text input. The current patch there actually already raises a warning and removes call sites from the Python library, though it does not add a doc string.

I've opened a PR moving in the other direction (making this public rather than deprecating it):

https://github.com/python/cpython/pull/6957

Thanks Thomas.  I've been a long term user of tokenize.generate_tokens() and would be sad to see it go.  

The underlying _tokenize() functionality is used else within the module, so there the no benefit to removing the API.

I just stumbled upon this issue while triaging. With the related issue https://bugs.python.org/issue12486 closed making generate_tokens public can this issue be closed since the original issue is not valid?

Thanks

Yes, I think this can be closed now.

Since generate_tokens was made public and documented with issue12486 I am closing this issue as won't fix as part of triaging. Feel free to reopen this issue if needed.

Thanks