Author gallicrooster
Recipients berker.peksag, gallicrooster, larsfuse, terry.reedy, xtreak
Date 2020-01-02.15:45:55
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1577979955.45.0.577282770452.issue35457@roundup.psfhosted.org>
In-reply-to
Content
Thanks @xtreak for providing some clarification on this behavior! I can write some tests to cover this behavior, assuming that we agree that an empty file means "unlimited access". This was worded as such in the old internet draft from 1996 (section 3.2.1 in https://www.robotstxt.org/norobots-rfc.txt). The current draft is more ambiguous with "If no group satisfies either condition, or no groups are present at all, no rules apply." https://tools.ietf.org/html/draft-koster-rep-00#section-2.2.1

https://www.robotstxt.org/robotstxt.html clearly states that an empty file gives full access, but I'm getting lost in figuring out which is the official spec at the moment :-)
History
Date User Action Args
2020-01-02 15:45:55gallicroostersetrecipients: + gallicrooster, terry.reedy, berker.peksag, xtreak, larsfuse
2020-01-02 15:45:55gallicroostersetmessageid: <1577979955.45.0.577282770452.issue35457@roundup.psfhosted.org>
2020-01-02 15:45:55gallicroosterlinkissue35457 messages
2020-01-02 15:45:55gallicroostercreate