I am trying to parse Google's robots.txt (http://google.com/robots.txt) and it fails when checking whether I can crawl the url /catalogs/p? (which it's allowed) but it's returning false, according to my question on stackoverflow -> http://stackoverflow.com/questions/15344253/robotparser-doesnt-seem-to-parse-correctly

Someone has answered it has to do with the line "rllib.quote(urlparse.urlparse(urllib.unquote(url))[2])" in robotparser's module, since it removes the "?" from the end of the url. 

Here is the answer I received -> http://stackoverflow.com/a/15350039/1649067

Attaching patch.

The second patch only normalizes the url. From http://www.robotstxt.org/norobots-rfc.txt it is not clear how to handle multiple rules with the same prefix.

I left a couple of comments on rietveld.

what is rietveld?

and why is this marked as "easy"?  it seems like it involves issues that aren't described well in the spec - it requires some kind of canonical way to describe urls with (and without) parameters to solve completely.

Rietveld is the review tool.  You can access it by clicking on the "review" link at the right of the patch.  You should have received an email as well when I made the review.

thanks (only subscribed to this now, so no previous email).

my guess is that google are assuming a dumb regexp so

   http://example.com/foo?

in a rule does not match

   http://example.com/foo

and also i realised that http://google.com/robots.txt doesn't contain any url with multiple parameters.  so perhaps i was wrong about needing a canonical representation (ie parameter ordering).

Well, the code is easy.  Figuring out what the code is supposed to do turns out to be hard, but we didn't know that when we marked it as easy :)  

I want to do more research before OKing a fix for this.  (There is clearly a bug, I'm just not certain what the correct fix is.)

Lucaz pointed out on IRC that the problem is that the current robotparser is implementing an outdated robots.txt standard.  He may work on fixing that.

Can you share the link of the new robots.txt standard? I may help to implement it.

I haven't a clue, that was part of the research I was going to do but haven't done yet (and probably won't for now...I'll wait to see if you or Lukaz pick it up first :).

I see he didn't nosy himself on the issue yet, though, so I've done that.  Maybe he'll respond.

robotparser implements http://www.robotstxt.org/orig.html, there's even a link to this document at http://docs.python.org/3/library/urllib.robotparser.html. As mher points out, there's a newer version of that spec formed as RFC: http://www.robotstxt.org/norobots-rfc.txt. It introduces Allow, specifies how percentage encoding should be treated and how to handle expiration.

Moreover, there is a de facto standard agreed by Google, Yahoo and Microsoft in 2008, documented by their respective blog posts:

http://googlewebmastercentral.blogspot.com/2008/06/improving-on-robots-exclusion-protocol.html

http://www.ysearchblog.com/2008/06/03/one-standard-fits-all-robots-exclusion-protocol-for-yahoo-google-and-microsoft/

http://www.bing.com/blogs/site_blogs/b/webmaster/archive/2008/06/03/robots-exclusion-protocol-joining-together-to-provide-better-documentation.aspx

For reference, there are two third-party robots.txt parsers out there implementing these extensions:

- https://pypi.python.org/pypi/reppy
- https://pypi.python.org/pypi/robotexclusionrulesparser

We need to decide how to incorporate those new features while maintaining backwards compatibility concerns.

My suggestion for this issue is going ahead with patch2 of Mher. It does  a simple normalization and does the right thing.

The case in the question is an empty query string and behavior or Allow and Disallow for that and patch addresses that. (I don't know why this *bug* was not detected earlier)

Robotparser implements the updated one ( www.robotstxt.org/norobots-rfc.txt) - You can check for Allow string verification in both code and tests.

That said, if updating robotparser further to more compliant with many cases which the 3rd party modules adhere, +1 to that. I suggest that be taken as a different issue and not be confused with this bug.

New changeset 30128355f53b by Senthil Kumaran in branch '3.3':
#17403: urllib.parse.robotparser normalizes the urls before adding to ruleline.
http://hg.python.org/cpython/rev/30128355f53b

New changeset e954d7a3bb8a by Senthil Kumaran in branch 'default':
merge from 3.3
http://hg.python.org/cpython/rev/e954d7a3bb8a

New changeset bcbad715c2ce by Senthil Kumaran in branch '2.7':
#17403: urllib.parse.robotparser normalizes the urls before adding to ruleline.
http://hg.python.org/cpython/rev/bcbad715c2ce

This is fixed in default, 3.3 and 2.7. I will merge this change to 3.2 code line before closing this. I shall raise a new request for updating robotparser with other goodies.

Perhaps it’s too late to modify the 3.2 branch now?

IMO the change made for this bug abuses the behaviour of urlunparse() removing empty query strings; see Issue 22852 where I proposed to stop it doing that.

Yes, this doesn't look like a security issue to me. Too late for 3.2. Closing this as "fixed".