Message328876
The behavior change introduced in 3.6.7 and 3.7.1 via https://bugs.python.org/issue33899 has further consequences:
```python
>>> tokenize.untokenize(tokenize.generate_tokens(io.StringIO('#').readline))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../cpython/cpython-upstream/Lib/tokenize.py", line 332, in untokenize
out = ut.untokenize(iterable)
File ".../cpython/cpython-upstream/Lib/tokenize.py", line 266, in untokenize
self.add_whitespace(start)
File ".../cpython/cpython-upstream/Lib/tokenize.py", line 227, in add_whitespace
raise ValueError("start ({},{}) precedes previous end ({},{})"
ValueError: start (1,1) precedes previous end (2,0)
```
The same goes for using the documented tokenize API (`generate_tokens` is not documented):
```
tokenize.untokenize(tokenize.tokenize(io.BytesIO(b'#').readline))
...
ValueError: start (1,1) precedes previous end (2,0)
```
`untokenize()` is no longer able to work on output of `generate_tokens()` if the input to generate_tokens() did not end in a newline.
Today's workaround: Always append a newline if one is missing to the line that the readline callable passed to tokenize or generate_tokens returns. Very annoying to implement. |
|
Date |
User |
Action |
Args |
2018-10-29 22:26:38 | gregory.p.smith | set | recipients:
+ gregory.p.smith, terry.reedy, taleinat, meador.inge, ammar2 |
2018-10-29 22:26:38 | gregory.p.smith | set | messageid: <1540851998.41.0.788709270274.issue35107@psf.upfronthosting.co.za> |
2018-10-29 22:26:38 | gregory.p.smith | link | issue35107 messages |
2018-10-29 22:26:38 | gregory.p.smith | create | |
|