Author quentin-maire
Recipients cheryl.sabella, quentin-maire
Date 2019-03-18.23:20:00
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1552951200.45.0.980846340144.issue36207@roundup.psfhosted.org>
In-reply-to
Content
I can't find a documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: http://www.eskimoz.fr/robots.txt, I believe that this is how it's implemented now in most parsers ?
History
Date User Action Args
2019-03-18 23:20:00quentin-mairesetrecipients: + quentin-maire, cheryl.sabella
2019-03-18 23:20:00quentin-mairesetmessageid: <1552951200.45.0.980846340144.issue36207@roundup.psfhosted.org>
2019-03-18 23:20:00quentin-mairelinkissue36207 messages
2019-03-18 23:20:00quentin-mairecreate