Author nagle
Recipients nagle
Date 2016-11-21.01:23:44
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1479691426.59.0.686880429063.issue28756@psf.upfronthosting.co.za>
In-reply-to
Content
urllib.robotparser.RobotFileParser always uses the default Python user agent. This agent is now blacklisted by many sites, and it's not possible to read the robots.txt file at all.
History
Date User Action Args
2016-11-21 01:23:47naglesetrecipients: + nagle
2016-11-21 01:23:46naglesetmessageid: <1479691426.59.0.686880429063.issue28756@psf.upfronthosting.co.za>
2016-11-21 01:23:45naglelinkissue28756 messages
2016-11-21 01:23:44naglecreate