Author calvin
Recipients
Date 2003-09-29.13:24:08
SpamBayes Score
Marked as misclassified
Message-id
In-reply-to
Content
Logged In: YES 
user_id=9205

http://www.robotstxt.org/wc/norobots-rfc.html specifies the
401 and 403 return code consequences as restricting the
whole site (ie disallow_all = 1).

For the password input, the patch looks good to me. On the
long term, robotparser.py should switch to urllib2.py
anyway, and it should handle Transfer-Encoding: gzip.
History
Date User Action Args
2007-08-23 14:17:15adminlinkissue813986 messages
2007-08-23 14:17:15admincreate