Author Anthony Sottile
Recipients Anthony Sottile
Date 2021-01-24.08:34:14
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
I did some profiling (attached a few files here with svgs) of running this script:

import io
import tokenize

# picked as the second longest file in cpython
with open('Lib/test/', 'rb') as f:
    bio = io.BytesIO(

def main():
    for _ in range(10):
        for _ in tokenize.tokenize(bio.readline):

if __name__ == '__main__':

the first profile is before the optimization, the second is after the optimization

The optimization takes the execution from ~6300ms to ~4500ms on my machine (representing a 28% - 39% improvement depending on how you calculate it)

(I'll attach the pstats and svgs after creation, seems I can only attach one file at once)
Date User Action Args
2021-01-24 08:34:14Anthony Sottilesetrecipients: + Anthony Sottile
2021-01-24 08:34:14Anthony Sottilesetmessageid: <>
2021-01-24 08:34:14Anthony Sottilelinkissue43014 messages
2021-01-24 08:34:14Anthony Sottilecreate