Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tab indentation breaks in tokenize.untokenize #68635

Closed
gumblex mannequin opened this issue Jun 14, 2015 · 1 comment
Closed

tab indentation breaks in tokenize.untokenize #68635

gumblex mannequin opened this issue Jun 14, 2015 · 1 comment
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@gumblex
Copy link
Mannequin

gumblex mannequin commented Jun 14, 2015

BPO 24447
Superseder
  • bpo-20387: tokenize/untokenize roundtrip fails with tabs
    		•
    Files
  • tab.py: example test file
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2015-06-19.19:32:43.881>
created_at = <Date 2015-06-14.03:33:14.093>
labels = ['type-bug', 'library']
title = 'tab indentation breaks in tokenize.untokenize'
updated_at = <Date 2015-06-19.19:32:43.880>
user = 'https://bugs.python.org/gumblex'

    bugs.python.org fields:

    activity = <Date 2015-06-19.19:32:43.880>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2015-06-19.19:32:43.881>
closer = 'terry.reedy'
components = ['Library (Lib)']
creation = <Date 2015-06-14.03:33:14.093>
creator = 'gumblex'
dependencies = []
files = ['39704']
hgrepos = []
issue_num = 24447
keywords = []
message_count = 1.0
messages = ['245333']
nosy_count = 1.0
nosy_names = ['gumblex']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '20387'
type = 'behavior'
url = 'https://bugs.python.org/issue24447'
versions = ['Python 3.4']
    @gumblex
    Copy link
    Mannequin Author

    gumblex mannequin commented Jun 14, 2015

    If a script uses tabs for indentation, tokenize.untokenize won't restore original indentation correctly from the second line of the indentation level, and thus breaks the file.

    This affects all Python versions.

    Test code:

    python2 -c 'import sys, tokenize; sys.stdout.write(tokenize.untokenize(tokenize.generate_tokens(sys.stdin.readline)))' < tab.py

    python3 -c 'import sys, tokenize; sys.stdout.buffer.write(tokenize.untokenize(tokenize.tokenize(sys.stdin.buffer.readline)))' < tab.py

    Out:

    def foo():
	pass
 pass
 if 1:
		pass
  pass

    @gumblex gumblex mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Jun 14, 2015
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @terryjreedy