Oh! My patch breaks "python -m". The problem is maybe no in the token 
parser but... somewhere else?
--- test.py ---
# coding: ASCII
raise Exception("line 2")
# try again!
---------------

Python 3.0 trunk unpatched:
---
$ ./python test.py
Traceback (most recent call last):
  File "test.py", line 3, in <module>

$ ./python -m test
Traceback (most recent call last):
  File "/home/haypo/prog/py3k/Lib/runpy.py", line 121, in 
_run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/haypo/prog/py3k/Lib/runpy.py", line 34, in _run_code
    exec(code, run_globals)
  File "/home/haypo/prog/py3k/test.py", line 2, in <module>
    raise Exception("line 2")
Exception: line 2
---

Python 3.0 trunk + tokenizer-coding.patch:
---
marge$ ./python test.py
Traceback (most recent call last):
  File "test.py", line 2, in <module>
    raise Exception("line 2")
Exception: line 2

Traceback (most recent call last):
  File "/home/haypo/prog/py3k/Lib/runpy.py", line 121, in 
_run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/haypo/prog/py3k/Lib/runpy.py", line 34, in _run_code
    exec(code, run_globals)
  File "/home/haypo/prog/py3k/test.py", line 1, in <module>
    # coding: ASCII
Exception: line 2
---