The following illustrates a problem with the parser 
handling the lack of trailing newlines:

>>> parser.suite('def foo():\n\tpass\n\n# comment')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<string>", line 4
    # comment
           ^
SyntaxError: invalid syntax
>>> parser.suite('def foo():\n\tpass\n\n# comment\n')
<parser.st object at 0x847f0a0>

This is similar to bug 501622, however, this only seems 
to happen when you have an indented block, followed by 
a comment line that has no trailing newline.

I traced through tokenizer.c and whittled down the issue 
into tok_get().  In the statement where it is processing 
the comment character and looking at the tabforms, in 
the first case this will end up with 'c' equal to EOF 
whereas in the second case "c" will eqaul '\n'.  When it 
equals EOF, it is unable to do the cleanup necessary to 
emit the DEDENT token (it immediately bails out with 
ENDMARKER which causes parsetok() to barf because 
the indentation level is still 1 inside tok_state).

Attached is a patch of a little hack I made that seems 
to fix the problem.  Although it seems to be a safe thing 
to do, it is definitely a hack.