Rietveld Code Review Tool
Help | Bug tracker | Discussion group | Source code | Sign in
(1)

Side by Side Diff: Lib/lib2to3/pgen2/tokenize.py

Issue 26581: Double coding cookie
Patch Set: Created 4 years, 2 months ago
Left:
Right:
Use n/p to move between diff chunks; N/P to move between comments. Please Sign in to add in-line comments.
Jump to:
View unified diff | Download patch
« no previous file with comments | « Lib/idlelib/IOBinding.py ('k') | Lib/test/test_importlib/source/test_source_encoding.py » ('j') | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 # Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006 Python Software Foundation. 1 # Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006 Python Software Foundation.
2 # All rights reserved. 2 # All rights reserved.
3 3
4 """Tokenization help for Python programs. 4 """Tokenization help for Python programs.
5 5
6 generate_tokens(readline) is a generator that breaks a stream of 6 generate_tokens(readline) is a generator that breaks a stream of
7 text into Python tokens. It accepts a readline-like method which is called 7 text into Python tokens. It accepts a readline-like method which is called
8 repeatedly to get the next line of input (or "" for EOF). It generates 8 repeatedly to get the next line of input (or "" for EOF). It generates
9 5-tuples with these members: 9 5-tuples with these members:
10 10
(...skipping 218 matching lines...) Expand 10 before | Expand all | Expand 10 after
229 elif toknum == DEDENT: 229 elif toknum == DEDENT:
230 indents.pop() 230 indents.pop()
231 continue 231 continue
232 elif toknum in (NEWLINE, NL): 232 elif toknum in (NEWLINE, NL):
233 startline = True 233 startline = True
234 elif startline and indents: 234 elif startline and indents:
235 toks_append(indents[-1]) 235 toks_append(indents[-1])
236 startline = False 236 startline = False
237 toks_append(tokval) 237 toks_append(tokval)
238 238
239 cookie_re = re.compile(r'^[ \t\f]*#.*coding[:=][ \t]*([-\w.]+)', re.ASCII) 239 cookie_re = re.compile(r'^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)', re.ASCII)
240 blank_re = re.compile(br'^[ \t\f]*(?:[#\r\n]|$)', re.ASCII) 240 blank_re = re.compile(br'^[ \t\f]*(?:[#\r\n]|$)', re.ASCII)
241 241
242 def _get_normal_name(orig_enc): 242 def _get_normal_name(orig_enc):
243 """Imitates get_normal_name in tokenizer.c.""" 243 """Imitates get_normal_name in tokenizer.c."""
244 # Only care about the first 12 characters. 244 # Only care about the first 12 characters.
245 enc = orig_enc[:12].lower().replace("_", "-") 245 enc = orig_enc[:12].lower().replace("_", "-")
246 if enc == "utf-8" or enc.startswith("utf-8-"): 246 if enc == "utf-8" or enc.startswith("utf-8-"):
247 return "utf-8" 247 return "utf-8"
248 if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \ 248 if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \
249 enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")): 249 enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")):
(...skipping 317 matching lines...) Expand 10 before | Expand all | Expand 10 after
567 stashed = None 567 stashed = None
568 568
569 for indent in indents[1:]: # pop remaining indent levels 569 for indent in indents[1:]: # pop remaining indent levels
570 yield (DEDENT, '', (lnum, 0), (lnum, 0), '') 570 yield (DEDENT, '', (lnum, 0), (lnum, 0), '')
571 yield (ENDMARKER, '', (lnum, 0), (lnum, 0), '') 571 yield (ENDMARKER, '', (lnum, 0), (lnum, 0), '')
572 572
573 if __name__ == '__main__': # testing 573 if __name__ == '__main__': # testing
574 import sys 574 import sys
575 if len(sys.argv) > 1: tokenize(open(sys.argv[1]).readline) 575 if len(sys.argv) > 1: tokenize(open(sys.argv[1]).readline)
576 else: tokenize(sys.stdin.readline) 576 else: tokenize(sys.stdin.readline)
OLDNEW
« no previous file with comments | « Lib/idlelib/IOBinding.py ('k') | Lib/test/test_importlib/source/test_source_encoding.py » ('j') | no next file with comments »

RSS Feeds Recent Issues | This issue
This is Rietveld 894c83f36cb7+