This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Roundtrip-test tokenize.untokenize(iterable_of_5_tuples)
Type: behavior Stage: needs patch
Components: Library (Lib), Tests Versions: Python 3.11
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: iritkatriel, python-dev, terry.reedy
Priority: normal Keywords:

Created on 2014-02-23 22:07 by terry.reedy, last changed 2022-04-11 14:57 by admin.

Messages (4)
msg212034 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-02-23 22:07
Currently, test_tokenize.roundtrip only tests the 2-tuple mode of untokenize. The currently buggy 5-tuple mode needs to be tested also, as far as possible, to reduce the chance of introducing more bugs when fixing current bugs. When this is done, all the constructed tests pass. Good so far.

The doctest normally tests 10 files selected from test/test_*.py and a few tries showed no problem. However, when testing all with -ucpu. 
F:\Python\dev> 3\py33\pcbuild\python_d -m test -ucpu test_tokenize
7 failed. So we must remove these from the list of candidates until the failure cause is determined and either fixed or determined not fixable (as with

As suggested in the revised docstring, finding the failure items for these 7 would be much easier if the file roundtrip test were removed from the doctest and made into a unittest with assertEqual.
msg212040 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-02-23 23:01
New changeset 8d6dd02a973f by Terry Jan Reedy in branch '3.3':
Issue #20750, Enable roundtrip tests for new 5-tuple untokenize. The

New changeset 73aa6d672b81 by Terry Jan Reedy in branch 'default':
Merge with 3.3, #20750
msg212043 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-02-23 23:06
The 2.7 tokenize tests do not have roundtrip tests.
Excluding the 7 files is a temporary hack. The issue should stay open until each either passes and is removed from the exclusion list or is determined to be a permanent exclusion, like test_pep3131.
msg407578 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-03 11:39
The 7 tests are still removed and need to be checked:
Date User Action Args
2022-04-11 14:57:59adminsetgithub: 64949
2021-12-03 11:39:03iritkatrielsetnosy: + iritkatriel

messages: + msg407578
versions: + Python 3.11, - Python 3.3, Python 3.4
2014-02-23 23:06:29terry.reedysetmessages: + msg212043
2014-02-23 23:01:27python-devsetnosy: + python-dev
messages: + msg212040
2014-02-23 22:07:06terry.reedycreate