Zac, thank you for Hypothesmith.  I am thinking about how to maybe use it to test certain IDLE functions.  But your 2nd example, as posted, does not compile, even with 3.8.  Typo?

Thank you also for the two failure examples.  I worked on untokenize in 2013 and an not surprised that there are still bugs.  The test assert matches the doc claim that the untokenize return "is guaranteed to tokenize back to match the input" of untokenize as far as type and string.  To get output that could be used to fix the bugs, I converted to unittest (and ran with 3.10).

from io import StringIO as SIO
import tokenize
import unittest

class RoundtripTest(unittest.TestCase):
    def test_examples(self):
        examples = ("#", "\n\\\n", "#\n\x0cpass#\n",)
        for code in examples:
            with self.subTest(code=code):
                tokens = list(tokenize.generate_tokens(SIO(code).readline))
                print(tokens)
                outstring = tokenize.untokenize(tokens)  # may change whitespace from source
                print(outstring)
                output = tokenize.generate_tokens(SIO(outstring).readline)
                self.assertEqual([(t.type, t.string) for t in tokens],
                                 [(t.type, t.string) for t in output])

unittest.main()

"#" compiles: untokenize calls add_whitespace, which failed on line 173 with
 ValueError: start (1,1) precedes previous end (2,0)
tokens = [
TokenInfo(type=60 (COMMENT), string='#', start=(1, 0), end=(1, 1), line='#'),
TokenInfo(type=61 (NL), string='', start=(1, 1), end=(1, 1), line='#'),
TokenInfo(type=4 (NEWLINE), string='', start=(1, 1), end=(1, 2), line=''),
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]

The doc for NL, a tokenize-only token, says "Token value used to indicate a non-terminating newline. The NEWLINE token indicates the end of a logical line of Python code; NL tokens are generated when a logical line of code is continued over multiple physical lines."  The NL token seems to be a mistake here.

Calling add_whitespace also seems like a mistake.  In any case, raising on a valid token stream is obviously bad.


"\n\\\n" does not compile in 3.8 or 3.10.
>>> compile("\n\\\n", '', 'exec')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "", line 2
    \
     ^
SyntaxError: unexpected EOF while parsing

generate_tokens calls _tokenize, which failed on line 521, as it should, with
tokenize.TokenError: ('EOF in multi-line statement', (3, 0))
Nothing to fix here.


"#\n\x0cpass#\n": outstring is '#\n pass#\n', which fails compile with IndentationError.
[(60, '#'),(61, '\n'),         (1, 'pass'),(60, '#'),(4, '\n'),        (0, '')] != 
[(60, '#'),(61, '\n'),(5, ' '),(1, 'pass'),(60, '#'),(4, '\n'),(6, ''),(0, '')]

test_tokenize tests the roundtrip with various modules and test strings, but perhaps none with formfeed.  I think the bug is tokenizing 'pass' as starting in column 1 instead of column 0.
TokenInfo(type=1 (NAME), string='pass', start=(2, 1), end=(2, 5), line='\x0cpass#\n')

Formfeed = '\f' = '\x0c' is legal here precisely because it is non-space 'whitespace' that does not advance the column counter.  In _tokenize, \f sets the column counter for indentation to 0.  Otherwise, there would be an IndentationError, as there is with the outstring.  But the string position ignores the indentation counter.  Either the string position must be adjusted, so \f is replaced with nothing, or a token for \f must be emitted so it is not replaced.

Tokens 5 and 6 are INDENT and DEDENT, so the latter will go away with the former.

What is a bit silly about untokenize is that it ignores the physical line of 5-tuples even when present.  Another issue, along with the dreadful API.


A note could be added to https://docs.python.org/3/reference/lexical_analysis.html#whitespace-between-tokens when _tokenize is patched.
---

BPO is aimed at facilitating patches.  Other discussions are best done elsewhere.  But I have a quick question and comment.

Can hypothesis be integrated as is with unittest?  Does it work to decorate test_xyz and get a sensible report of multiple failures?  Is there now or possibly in the future an iterator interface, so one could write "for testcase in testcases:   with subTest...."

About properties: your blog post https://github.com/Zac-HD/stdlib-property-tests pointed me to metamorphic testing https://www.hillelwayne.com/post/metamorphic-testing/ which lead me to the PDF version of "“Metamorphic Testing: A Review of Challenges and Opportunities”.  Most properties I have seen correspond to metamorphic relations.  The 'metamorphic' is broader than may be immediately obvious.  I would like to discuss this more on a better channel.