Author terry.reedy
Recipients larsfuse, terry.reedy
Date 2018-12-14.21:08:55
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1544821735.16.0.788709270274.issue35457@psf.upfronthosting.co.za>
In-reply-to
Content
https://docs.python.org/2.7/library/robotparser.html#module-robotparser
and
https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
refers users, for file structure, to http://www.robotstxt.org/orig.html.
This says nothing about the effect of an empty file, so I don't see this as a bug.  Even if it was, I would be dubious about reversing the effect without a deprecation notice first, and definitely not in 2.7.

I would propose instead that the doc be changed to refer to the new file, with more and better examples, but add a note that robotparser interprets empty files as 'block all' rather than 'allow all'.

Try bringing this up on python-ideas.
History
Date User Action Args
2018-12-14 21:08:55terry.reedysetrecipients: + terry.reedy, larsfuse
2018-12-14 21:08:55terry.reedysetmessageid: <1544821735.16.0.788709270274.issue35457@psf.upfronthosting.co.za>
2018-12-14 21:08:55terry.reedylinkissue35457 messages
2018-12-14 21:08:55terry.reedycreate