classification
Title: PEP 414 installation hook fails with an AssertionError
Type: behavior Stage: resolved
Components: Versions: Python 3.3
process
Status: closed Resolution: duplicate
Dependencies: Superseder: tokenizer.untokenize not invariant with line continuations
View: 9974
Assigned To: Nosy List: aronacher, vinay.sajip
Priority: normal Keywords:

Created on 2012-05-03 15:42 by vinay.sajip, last changed 2016-04-23 20:47 by berker.peksag. This issue is now closed.

Messages (2)
msg159864 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-05-03 15:42
I'm not sure if I've done something wrong, but I get an AssertionError when trying to run the tokenizer on a Python file from the Django source. The gist at https://gist.github.com/1977558 has the files concerned:

1. test_tokenize.py - the script which fails
2. tokenize_example.py - the file being tokenized and untokenized. This is from the Django source: django/extras/csrf_migration_helper.py
3. tokenizer.py - your tokenize module, I renamed it because I was working in /tmp and didn't want to import the Python 3.2 stdlib's tokenize.py
4. The test output shows that the tokenize_example module imports OK in Python 2.7.2, but running the test_tokenize script on it with Python3.2 fails with an AssertionError.

I did some more testing, there are 131 failures in the Django source tree (all look like the same AssertionError).

N.B. I posted this to your GitHub repo where you published the hook.
msg174720 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2012-11-04 00:03
Duplicate of #9974.  The tokenizer is choking on lines that end in '\'.  So even this can't be parsed:

a = 1 + \
   2
History
Date User Action Args
2016-04-23 20:47:04berker.peksagsetstatus: open -> closed
resolution: duplicate
stage: resolved
2014-12-31 16:20:19akuchlingsetnosy: - akuchling
2012-11-04 00:03:15akuchlingsetsuperseder: tokenizer.untokenize not invariant with line continuations

messages: + msg174720
nosy: + akuchling
2012-08-01 22:25:44vinay.sajipsetversions: + Python 3.3
2012-05-03 15:44:07vinay.sajipsettype: behavior
2012-05-03 15:42:20vinay.sajipcreate