robotparser robotparser.py::RobotFileParser::can_fetch 
currently returns the result of the first applicable rule.  It 
should loop through all rules looking for anything that 
disallows access.  For example, if your first rule applies 
to 'wget' and 'python' and disallows access to /dir1/, and 
your second rule is a 'python' rule that disallows access 
to /dir2/, robotparser will falsely claim that python is 
allowed to access /dir2/.

Patch against current source attached.