Author lukasz.langa
Recipients acooke, benmezger, ezio.melotti, lukasz.langa, mher.movsisyan, r.david.murray
Date 2013-04-22.13:16:23
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
robotparser implements, there's even a link to this document at As mher points out, there's a newer version of that spec formed as RFC: It introduces Allow, specifies how percentage encoding should be treated and how to handle expiration.

Moreover, there is a de facto standard agreed by Google, Yahoo and Microsoft in 2008, documented by their respective blog posts:

For reference, there are two third-party robots.txt parsers out there implementing these extensions:


We need to decide how to incorporate those new features while maintaining backwards compatibility concerns.
Date User Action Args
2013-04-22 13:16:24lukasz.langasetrecipients: + lukasz.langa, ezio.melotti, acooke, r.david.murray, mher.movsisyan, benmezger
2013-04-22 13:16:24lukasz.langasetmessageid: <>
2013-04-22 13:16:24lukasz.langalinkissue17403 messages
2013-04-22 13:16:23lukasz.langacreate