classification
Title: Add example to tokenize.tokenize
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Windson Yang, berker.peksag, docs@python, lisroach, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2019-04-18 03:43 by Windson Yang, last changed 2020-01-25 19:40 by berker.peksag. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 12947 merged Windson Yang, 2019-04-25 05:20
PR 18187 merged miss-islington, 2020-01-25 19:23
PR 18188 merged miss-islington, 2020-01-25 19:23
Messages (8)
msg340467 - (view) Author: Windson Yang (Windson Yang) * Date: 2019-04-18 03:43
> The tokenize() generator requires one argument, readline, which must be a callable object which provides the same interface as the io.IOBase.readline() method of file objects. Each call to the function should return one line of input as bytes.

Add an example like this should be easier to understand:

# example.py
def foo:
    pass

# tokenize_example.py
import tokenize
f = open('example.py', 'rb')
token_gen = tokenize.tokenize(f.readline)

for token in token_gen:
    # Something like this
    # TokenInfo(type=1 (NAME), string='class', start=(1, 0), end=(1, 5), line='class Foo:\n')
    # TokenInfo(type=1 (NAME), string='Foo', start=(1, 6), end=(1, 9), line='class Foo:\n')
    # TokenInfo(type=53 (OP), string=':', start=(1, 9), end=(1, 10), line='class Foo:\n')
    print(token)
msg340799 - (view) Author: Lisa Roach (lisroach) * (Python committer) Date: 2019-04-24 18:16
This could be added to the examples section of the tokenize doc, would you want to make the PR Windson?
msg340822 - (view) Author: Windson Yang (Windson Yang) * Date: 2019-04-25 04:21
Yes, I can make a PR for it.
msg341449 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-05-05 13:58
I do not think a new example is needed. The existing example already demonstrates the use of file's readline method.

If you need an example for opening a file, the tokenize module documentation is not an appropriate place for this.
msg360700 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2020-01-25 19:23
New changeset 4b09dc79f4d08d85f2cc945563e9c8ef1e531d7b by Berker Peksag (Windson yang) in branch 'master':
bpo-36654: Add examples for using tokenize module programmically (#12947)
https://github.com/python/cpython/commit/4b09dc79f4d08d85f2cc945563e9c8ef1e531d7b
msg360701 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2020-01-25 19:34
New changeset 1cf0df4f1bcc38dfd70a152af20cf584de531ea7 by Berker Peksag (Miss Islington (bot)) in branch '3.8':
bpo-36654: Add examples for using tokenize module programmatically (GH-18187)
https://github.com/python/cpython/commit/1cf0df4f1bcc38dfd70a152af20cf584de531ea7
msg360702 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2020-01-25 19:36
New changeset 6dbd843dedc9e05c0e3f4714294837f0a83deebe by Berker Peksag (Miss Islington (bot)) in branch '3.7':
bpo-36654: Add examples for using tokenize module programmatically (GH-12947)
https://github.com/python/cpython/commit/6dbd843dedc9e05c0e3f4714294837f0a83deebe
msg360704 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2020-01-25 19:40
Wow, I managed to make typos in all three commits!

PR 12947 has some discussion about why adding these examples would be a good idea as we now have two different APIs for unicode and bytes input.

Thanks for the PR, Windson.
History
Date User Action Args
2020-01-25 19:40:04berker.peksagsetstatus: open -> closed
versions: + Python 3.9, - Python 3.5, Python 3.6
messages: + msg360704

resolution: fixed
stage: patch review -> resolved
2020-01-25 19:36:07berker.peksagsetmessages: + msg360702
2020-01-25 19:34:40berker.peksagsetmessages: + msg360701
2020-01-25 19:23:19miss-islingtonsetpull_requests: + pull_request17571
2020-01-25 19:23:12miss-islingtonsetpull_requests: + pull_request17570
2020-01-25 19:23:03berker.peksagsetnosy: + berker.peksag
messages: + msg360700
2019-05-05 13:58:21serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg341449
2019-04-25 05:20:07Windson Yangsetkeywords: + patch
stage: patch review
pull_requests: + pull_request12871
2019-04-25 04:21:30Windson Yangsetmessages: + msg340822
2019-04-24 18:16:19lisroachsetnosy: + lisroach
messages: + msg340799
2019-04-18 03:43:56Windson Yangcreate