I'm not sure if I've done something wrong, but I get an AssertionError when trying to run the tokenizer on a Python file from the Django source. The gist at https://gist.github.com/1977558 has the files concerned:

1. test_tokenize.py - the script which fails
2. tokenize_example.py - the file being tokenized and untokenized. This is from the Django source: django/extras/csrf_migration_helper.py
3. tokenizer.py - your tokenize module, I renamed it because I was working in /tmp and didn't want to import the Python 3.2 stdlib's tokenize.py
4. The test output shows that the tokenize_example module imports OK in Python 2.7.2, but running the test_tokenize script on it with Python3.2 fails with an AssertionError.

I did some more testing, there are 131 failures in the Django source tree (all look like the same AssertionError).

N.B. I posted this to your GitHub repo where you published the hook.

Duplicate of #9974.  The tokenizer is choking on lines that end in '\'.  So even this can't be parsed:

a = 1 + \
   2