Author benmezger
Recipients benmezger
Date 2013-03-12.10:58:24
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
I am trying to parse Google's robots.txt ( and it fails when checking whether I can crawl the url /catalogs/p? (which it's allowed) but it's returning false, according to my question on stackoverflow ->

Someone has answered it has to do with the line "rllib.quote(urlparse.urlparse(urllib.unquote(url))[2])" in robotparser's module, since it removes the "?" from the end of the url. 

Here is the answer I received ->
Date User Action Args
2013-03-12 10:58:24benmezgersetrecipients: + benmezger
2013-03-12 10:58:24benmezgersetmessageid: <>
2013-03-12 10:58:24benmezgerlinkissue17403 messages
2013-03-12 10:58:24benmezgercreate