This post
https://groups.google.com/d/msg/leo-editor/DpZ2cMS03WE/VPqtB9lTEAAJ
discusses a complete rewrite of tokenizer.untokenize.

To quote from the post:

I have "discovered" a spectacular replacement for Untokenizer.untokenize in python's tokenize library module. The wretched, buggy, and impossible-to-fix add_whitespace method is gone. The new code has no significant 'if' statements, and knows almost nothing about tokens!  This is the way untokenize is written in The Book.

The new code should put an end to a long series of issues against untokenize code in python's tokenize library module.  Some closed issues were blunders arising from dumbing-down the TestRoundtrip.check_roundtrip method in test_tokenize.py. 

Imo, the way is now clear for proper unit testing of python's Untokenize class.