➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author nagle
Recipients nagle
Date 2016-11-21.01:23:44
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1479691426.59.0.686880429063.issue28756@psf.upfronthosting.co.za>
In-reply-to
Content 
urllib.robotparser.RobotFileParser always uses the default Python user agent. This agent is now blacklisted by many sites, and it's not possible to read the robots.txt file at all.
History
Date User Action Args
2016-11-21 01:23:47naglesetrecipients: + nagle
2016-11-21 01:23:46naglesetmessageid: <1479691426.59.0.686880429063.issue28756@psf.upfronthosting.co.za>
2016-11-21 01:23:45naglelinkissue28756 messages
2016-11-21 01:23:44naglecreate