➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author calvin
Recipients
Date 2000-11-02.17:40:27
SpamBayes Score
Marked as misclassified
Message-id
In-reply-to
Content 
I have written a new RobotParser module 'robotparser2.py'.

This module is

o backward compatible with the old one

o makes correct useragent matching (is buggy in
  robotparser.py)

o strips comments correctly (is buggy in robotparser.py)

o uses httplib instead of urllib.urlopen() to catch HTTP
  connect errors correctly (is buggy in robotparser.py)
  
o implements not only the draft at

http://info.webcrawler.com/mak/projects/robots/norobots.html
  but also the new one at
  http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html


Bastian Kleineidam
History
Date User Action Args
2007-08-23 15:02:26adminlinkissue402229 messages
2007-08-23 15:02:26admincreate