RobotFileParser.parse() contains the lines

                elif line[0] == "disallow":
                    if state != 0:
                        entry.rulelines.append(RuleLine(line[1], False))
                        state = 2
                elif line[0] == "allow":
                    if state != 0:
                        entry.rulelines.append(RuleLine(line[1], True))

with no 'state = 2' in the 'allow' part.
This causes different behaviour depending on whether the file ends with
'allow' or 'disallow', or an empty line.

Those lines were taken from revision 65118.  My Python 2.5 sources are
similar.  I have not checked others.

Do you have a concrete robots.txt file I can use in a test case?

Perhaps more important than a test case, can you explain what states 0, 1 
and 2 are (maybe give them some symbolic names I can at least put in a 
comment)?  This is not my code.  Though I wrote the first version of the 
robotparser module and I served as the person who checked this version 
into the repo, as I recall this is a complete rewrite.

*sigh* there are no test cases in the current code with Allow: lines in 
test_robotparser.py.

Attached is a patch against 2.6 which adds the missing line (state = 2), a 
comment describing the three states the parser can be in and expands the 
test cases to cover this change (fail without it, pass with it).  In the 
process I snagged some broken example robots.txt files from Google's 
Googlebot help pages and turned them into test cases, both before and 
after fixing the examples.  I think this can probably go into the 
repository as a bug fix and get merged to the py3k branch.  If nobody 
complains in the next day or two I'll apply it.

committed as r 65255.