http://hg.python.org/robots.txt currently disallows all robots from all paths. This means that the site doesn't show up in Google search results seeking, for instance, browsing access to the python source
https://www.google.com/search?ie=UTF-8&q=python+source+browse
https://www.google.com/search?ie=UTF-8&q=python+repo+browse
https://www.google.com/search?ie=UTF-8&q=hg+python+browse
etc...

Instead, robots.txt should allow access to the landing page, http://hg.python.org/, and the landing pages for hosted projects, e.g. http://hg.python.org/cpython/, while prohibiting access to the */rev/*, */shortlog/*, ..., directories.

This change would be very easy, cost virtually nothing, and let users find the mercurial repository viewer from search engines. Note that http://svn.python.org/ does show up in search results, as an illustration of how convenient this is.

Can you propose a robots.txt file?

I don't know too much about robots.txt but how about

Disallow: */rev/*
Disallow: */shortlog/*
Allow:

Are there any other directories we'd like to exclude?

Unfortunately, I don't think it will be that easy because I don't think robots.txt supports wildcard paths like that. Possibly, we should just whitelist a few important repositories.

Yes, I think we should whitelist rather than blacklist. The problem with letting engines index the repositories is the sheer resource cost when they fetch many heavy pages (such as annotate, etc.).

Two things: is it worth fixing this bug given the impending move to github?  Also, why is this reported here and not the pydotorg tracker?  https://github.com/python/pythondotorg/issues

Given that the last comment was 2014, I'm going to go ahead and close this issue.