Rietveld Code Review Tool
Help | Bug tracker | Discussion group | Source code | Sign in
(53673)

Unified Diff: Lib/lib2to3/pgen2/tokenize.py

Issue 18873: "Encoding" detected in non-comment lines
Patch Set: Created 6 years ago
Use n/p to move between diff chunks; N/P to move between comments. Please Sign in to add in-line comments.
Jump to:
View side-by-side diff with in-line comments
Download patch
« no previous file with comments | « Lib/idlelib/IOBinding.py ('k') | Lib/lib2to3/tests/data/false_encoding.py » ('j') | no next file with comments »
Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
--- a/Lib/lib2to3/pgen2/tokenize.py Thu Sep 05 18:02:31 2013 +0300
+++ b/Lib/lib2to3/pgen2/tokenize.py Sat Sep 07 17:49:25 2013 +0300
@@ -236,7 +236,7 @@
startline = False
toks_append(tokval)
-cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
+cookie_re = re.compile(r'^[ \t\f]*#.*coding[:=][ \t]*([-\w.]+)', re.ASCII)
def _get_normal_name(orig_enc):
"""Imitates get_normal_name in tokenizer.c."""
@@ -281,11 +281,10 @@
line_string = line.decode('ascii')
except UnicodeDecodeError:
return None
-
- matches = cookie_re.findall(line_string)
- if not matches:
+ match = cookie_re.match(line_string)
+ if not match:
return None
- encoding = _get_normal_name(matches[0])
+ encoding = _get_normal_name(match.group(1))
try:
codec = lookup(encoding)
except LookupError:
« no previous file with comments | « Lib/idlelib/IOBinding.py ('k') | Lib/lib2to3/tests/data/false_encoding.py » ('j') | no next file with comments »

RSS Feeds Recent Issues | This issue
This is Rietveld 894c83f36cb7+