Author orsenthil
Recipients acooke, benmezger, ezio.melotti, lukasz.langa, mher.movsisyan, orsenthil, r.david.murray
Date 2013-04-22.13:29:16
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1366637356.31.0.148879290969.issue17403@psf.upfronthosting.co.za>
In-reply-to
Content
My suggestion for this issue is going ahead with patch2 of Mher. It does  a simple normalization and does the right thing.

The case in the question is an empty query string and behavior or Allow and Disallow for that and patch addresses that. (I don't know why this *bug* was not detected earlier)

Robotparser implements the updated one ( www.robotstxt.org/norobots-rfc.txt) - You can check for Allow string verification in both code and tests.

That said, if updating robotparser further to more compliant with many cases which the 3rd party modules adhere, +1 to that. I suggest that be taken as a different issue and not be confused with this bug.
History
Date User Action Args
2013-04-22 13:29:16orsenthilsetrecipients: + orsenthil, ezio.melotti, acooke, r.david.murray, mher.movsisyan, lukasz.langa, benmezger
2013-04-22 13:29:16orsenthilsetmessageid: <1366637356.31.0.148879290969.issue17403@psf.upfronthosting.co.za>
2013-04-22 13:29:16orsenthillinkissue17403 messages
2013-04-22 13:29:16orsenthilcreate