Title: Raise Deprecation warning for tokenize.generate_tokens
Type: Stage: resolved
Components: Documentation Versions: Python 3.7
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, martin.panter, mbussonn, rhettinger, takluyver, xtreak
Priority: normal Keywords:

Created on 2016-10-11 19:04 by mbussonn, last changed 2018-09-24 15:15 by xtreak. This issue is now closed.

Messages (7)
msg278500 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2016-10-11 19:04
the Tokenize module has the following code:

# An undocumented, backwards compatible, API for all the places in the standard
# library that expect to be able to use tokenize with strings
def generate_tokens(readline):
    return _tokenize(readline, None)

So I'm going to assume it is Deprecated... (since 3.0 AFAICT). 

If it is expected from Python developers to not use it, may I suggest:

1) Marking it as deprecated in the docstring/
2) Raise a deprecation warning. 

Indeed not everyone code by looking at the documentation and it is relatively easy to have your IDE/editor/REPL to complete it. Even tools that grab the source (IPython double question mark for example) will not show the comment above which make it kinda pointless.
msg278503 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-10-11 20:26
There is related discussion in Issue 12486 about supporting unencoded text input. The current patch there actually already raises a warning and removes call sites from the Python library, though it does not add a doc string.
msg316984 - (view) Author: Thomas Kluyver (takluyver) * Date: 2018-05-17 20:49
I've opened a PR moving in the other direction (making this public rather than deprecating it):
msg316997 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-05-18 01:23
Thanks Thomas.  I've been a long term user of tokenize.generate_tokens() and would be sad to see it go.  

The underlying _tokenize() functionality is used else within the module, so there the no benefit to removing the API.
msg326155 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-09-23 14:34
I just stumbled upon this issue while triaging. With the related issue closed making generate_tokens public can this issue be closed since the original issue is not valid?

msg326166 - (view) Author: Thomas Kluyver (takluyver) * Date: 2018-09-23 16:53
Yes, I think this can be closed now.
msg326246 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-09-24 15:15
Since generate_tokens was made public and documented with issue12486 I am closing this issue as won't fix as part of triaging. Feel free to reopen this issue if needed.

Date User Action Args
2018-09-24 15:15:28xtreaksetstatus: open -> closed
resolution: wont fix
messages: + msg326246

stage: resolved
2018-09-23 16:53:12takluyversetmessages: + msg326166
2018-09-23 14:34:27xtreaksetnosy: + xtreak
messages: + msg326155
2018-05-18 01:23:00rhettingersetnosy: + rhettinger
messages: + msg316997
2018-05-17 20:49:23takluyversetnosy: + takluyver
messages: + msg316984
2016-10-11 20:26:46martin.pantersetnosy: + martin.panter
messages: + msg278503
2016-10-11 19:04:57mbussonncreate