➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author gallicrooster
Recipients berker.peksag, gallicrooster, larsfuse, terry.reedy, xtreak
Date 2020-01-02.15:45:55
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1577979955.45.0.577282770452.issue35457@roundup.psfhosted.org>
In-reply-to
Content 
Thanks @xtreak for providing some clarification on this behavior! I can write some tests to cover this behavior, assuming that we agree that an empty file means "unlimited access". This was worded as such in the old internet draft from 1996 (section 3.2.1 in https://www.robotstxt.org/norobots-rfc.txt). The current draft is more ambiguous with "If no group satisfies either condition, or no groups are present at all, no rules apply." https://tools.ietf.org/html/draft-koster-rep-00#section-2.2.1

https://www.robotstxt.org/robotstxt.html clearly states that an empty file gives full access, but I'm getting lost in figuring out which is the official spec at the moment :-)
History
Date User Action Args
2020-01-02 15:45:55gallicroostersetrecipients: + gallicrooster, terry.reedy, berker.peksag, xtreak, larsfuse
2020-01-02 15:45:55gallicroostersetmessageid: <1577979955.45.0.577282770452.issue35457@roundup.psfhosted.org>
2020-01-02 15:45:55gallicroosterlinkissue35457 messages
2020-01-02 15:45:55gallicroostercreate