classification
Title: tab indentation breaks in tokenize.untokenize
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4
process
Status: closed Resolution: duplicate
Dependencies: Superseder: tokenize/untokenize roundtrip fails with tabs
View: 20387
Assigned To: Nosy List: gumblex
Priority: normal Keywords:

Created on 2015-06-14 03:33 by gumblex, last changed 2015-06-19 19:32 by terry.reedy. This issue is now closed.

Files
File name Uploaded Description Edit
tab.py gumblex, 2015-06-14 03:33 example test file
Messages (1)
msg245333 - (view) Author: Dingyuan Wang (gumblex) * Date: 2015-06-14 03:33
If a script uses tabs for indentation, tokenize.untokenize won't restore original indentation correctly from the second line of the indentation level, and thus breaks the file.

This affects all Python versions.

Test code:

python2 -c 'import sys, tokenize; sys.stdout.write(tokenize.untokenize(tokenize.generate_tokens(sys.stdin.readline)))' < tab.py

python3 -c 'import sys, tokenize; sys.stdout.buffer.write(tokenize.untokenize(tokenize.tokenize(sys.stdin.buffer.readline)))' < tab.py

Out:

def foo():
	pass
 pass
 if 1:
		pass
  pass
History
Date User Action Args
2015-06-19 19:32:43terry.reedysetstatus: open -> closed
superseder: tokenize/untokenize roundtrip fails with tabs
resolution: duplicate
stage: resolved
2015-06-14 03:33:14gumblexcreate